ZIT turns underwater plankton video into composite images that reveal animal locomotion patterns. Think of it as computational long-exposure photography for marine biology. You feed it a video of tiny creatures swimming around, and it produces a single image showing everywhere they went – crisp trails on a stable background, like light painting but for zooplankton.
The project is on PyPI and GitHub.
The Problem
You have a video of plankton swimming in a petri dish under a microscope. You want a single image that shows the motion paths. The naive approach – averaging all frames together – produces a blurry mess where the background dominates and the organisms become faint ghosts. Overlaying frames with alpha blending is slightly better but still noisy. The issue is that the organisms are tiny, the background is dominant, and naive compositing does not discriminate between “this pixel is a plankton” and “this pixel is a speck of sediment.”
Two Compositing Modes
Mode 1: Pixel Difference
The basic approach. Take a reference background frame, compare each subsequent frame pixel-by-pixel using Euclidean distance in RGB space. If a pixel differs from the background by more than an epsilon threshold, transfer it to the composite. If the difference is below a noise delta, ignore it. Simple, fast, and okay for clean videos with high contrast.
The problem: it picks up noise. Slight lighting variations, camera jitter, sediment particles drifting through the frame – all of these exceed the epsilon threshold and get composited as artifacts. For messy real-world microscope footage, this mode produces something that looks like a composite of plankton trails plus television static.
Mode 2: Entity Recognition (The Good One)
This is where OpenCV earns its keep. Instead of pixel-level comparison, we use MOG2 – Mixture of Gaussians background subtraction. MOG2 builds a statistical model of the background over time, learning which pixels are “normally” part of the static scene and which are transient foreground objects. Each frame gets a foreground mask: white where something is moving, black where the background is stable.
But the raw mask is still noisy. So we clean it up with morphological operations:
- Opening (erosion then dilation) – removes small noise speckles
- Closing (dilation then erosion) – fills small holes in detected organisms
- Additional dilation – slightly expands the detected regions to capture the full organism body
Then contour filtering. We find all contours in the cleaned mask and reject any that are too small (noise) or have extreme aspect ratios (scan lines, horizontal artifacts from the camera). What remains are the actual organisms.
The result is dramatically cleaner. The background is pristine, and you see only the plankton trails – exactly where each organism swam during the recording period.
The Parameter Sweep Grid
Different videos need different parameters. Water turbidity, lighting, camera resolution, organism size – all of these affect what threshold and minimum area values produce the cleanest composite. Tuning by trial and error is tedious.
So I built a sweep tool. sweep_grid.py generates a 5x5 grid of composites, sweeping across two axes: MinArea (the minimum contour area to count as a real organism) and Thresh (the background subtraction threshold). You look at the grid, find the cell that looks best, and use those parameters. Visual parameter search.
Trajectory Tracking
A composite shows where the animals went, but not which animal went where. So ZIT can now link detections across frames into tracks.
Every frame, each surviving contour becomes a detection: its centroid and area. Consecutive frames are matched by solving an assignment problem (SciPy’s linear_sum_assignment) on centroid distance. Any match farther than --max-jump pixels is rejected, and the leftover detection starts a new track. A track can go unseen for two sampled frames and still pick back up, so a brief occlusion doesn’t break it in half.
The first run on real footage (a cheetah in tall grass) produced about a thousand “tracks.” Almost all of them were flickering blades of grass and dust that got strung together into random zig-zags. Plankton videos have the same problem with drifting sediment. What fixed it was a single number: straightness, the distance from a track’s first point to its last divided by the total path length. A swimmer heads somewhere, so it scores near 1. Jitter goes nowhere, so it scores near 0. Keeping tracks with at least 4 sightings and a straightness of 0.7 or more cut a thousand tracks down to a few dozen.
Large animals have the opposite problem. MOG2 breaks a horse or a bird into dozens of separate blobs (legs, mane, splashes), and each blob gets its own track. --track-merge runs a morphological close sized to a fraction of the frame before tracking. That fuses the fragments back into one body without changing the composite itself.
That’s one track, followed across 23 samples at a quarter second apart, from the far bank to the foreground. The yellow squiggle in the corner is splash that happened to pass the filters.
One lesson: tracking needs a fixed camera. Even with the noise filtered out, the cheetah clip had no path worth drawing, because it’s a panning shot. The camera follows the cheetah, so the animal stays in the middle of the frame and the background is what moves.
Usage
pip install zooplankton-image-tool
zit -i plankton_video.mp4 --interval 0.5 --composite --entities --epsilon 30
--interval sets the number of seconds between frame captures (default 1). It now accepts fractions like 0.25: lower intervals give denser trails, and tracking needs them, because an animal can’t move too far between samples. --entities selects the MOG2 pipeline.
To add tracking:
zit -i clip.mp4 --interval 0.25 --composite --entities \
--trails --tracks-csv tracks.csv
--trails draws each track onto the composite in its own color, with an arrow at the end. --tracks-csv writes track_id, frame, x, y, area for every sighting, ready for speed or path-length analysis. For large animals, add --track-merge 0.1 and a bigger --max-jump.
What I Learned
The interesting technical takeaway is that background subtraction is not just for security cameras. MOG2 was designed for surveillance – detecting people walking through a scene – but it works beautifully for microscopy. The statistical background model adapts to gradual lighting changes (a problem with fixed-threshold approaches), and the morphological filtering pipeline transfers directly from “person detection” to “plankton detection” because the underlying math does not care what the foreground object is. It cares about the statistics of pixel variation over time.
The contour aspect-ratio filter was a late addition born from frustration. Some microscope cameras produce horizontal scan-line artifacts – thin horizontal stripes that MOG2 correctly identifies as “not background” but that are obviously not organisms. Filtering by aspect ratio (rejecting contours wider than they are tall by a factor of 10) eliminated these artifacts completely.
Computer vision for marine biology. Not a combination I expected to work this well. But the plankton do not care what the algorithm was designed for, and neither does the math.