ZIT turns underwater plankton video into composite images that reveal animal locomotion patterns. Think of it as computational long-exposure photography for marine biology. You feed it a video of tiny creatures swimming around, and it produces a single image showing everywhere they went: crisp trails on a stable background.
It is on PyPI.
The Problem
You have a video of plankton swimming in a petri dish under a microscope. You want a single image that shows the motion paths. The naive approach, averaging all frames together, produces a blurry mess where the background dominates and the organisms become faint ghosts. Overlaying frames with alpha blending is slightly better but still noisy. The issue is that the organisms are tiny and naive compositing does not discriminate between “this pixel is a plankton” and “this pixel is a speck of sediment.”
Two Compositing Modes
Mode 1: Pixel Difference
The basic approach. Start the composite from the first frame and compare each later frame to it pixel by pixel, using Euclidean distance in RGB space. If a pixel differs from the composite by more than an epsilon threshold, copy it in. If the pixel is within a noise delta of that frame’s average color, ignore it. Simple, fast, and okay for clean videos with high contrast.
The problem: it picks up noise. Slight lighting variations, camera jitter, sediment particles drifting through the frame: all of these exceed the epsilon threshold and get composited as artifacts. For messy real-world microscope footage, this mode produces something that looks like a composite of plankton trails plus television static.
Mode 2: Entity Recognition
This is where OpenCV earns its keep. Instead of a raw pixel difference against one image, we use MOG2, Mixture of Gaussians background subtraction. MOG2 builds a statistical model of the background over time, learning which pixels are “normally” part of the static scene and which are transient foreground objects. Each frame gets a foreground mask: white where something is moving, black where the background is stable.
But the raw mask is still noisy. So we clean it up with morphological operations:
- Opening (erosion then dilation): removes small noise speckles
- Closing (dilation then erosion): fills small holes in detected organisms
- Additional dilation: slightly expands the detected regions to capture the full organism body
Then contour filtering. We find all contours in the cleaned mask and reject any that are too small (noise) or have extreme aspect ratios (scan lines, horizontal artifacts from the camera). What remains are the actual organisms. Their pixels are pasted onto MOG2’s estimate of the empty background, later frames over earlier ones.
The result is much cleaner. The background is untouched, and you see only the plankton trails: where each organism was during the recording. In this composite of microscope footage, the fan of repeated outlines at the lower right is a single animal, drawn at each position it occupied:
The pipeline does not care what the animals are, and the other examples in this post are larger ones. This composite is from a clip of butterflies:
The Parameter Sweep Grid
Different videos need different parameters. Water turbidity, lighting, camera resolution, organism size: all of these affect what threshold and minimum area values produce the cleanest composite. Tuning by trial and error is tedious.
So I built a sweep tool. sweep_grid.py generates a 5x5 grid of composites, sweeping across two axes: MinArea (the minimum contour area to count as a real organism) and Thresh (the background subtraction threshold). You look at the grid, find the cell that looks best, and use those parameters. Visual parameter search.
Trajectory Tracking
A composite shows where the animals went, but not which animal went where. So ZIT also links detections across frames into tracks.
In each sampled frame, every surviving contour becomes a detection: its centroid and area. New detections are matched to the existing tracks by solving an assignment problem (SciPy’s linear_sum_assignment) on centroid distance. Any match farther than --max-jump pixels is rejected, and the leftover detection starts a new track. A track can go unseen for two sampled frames and still pick back up, so a brief occlusion doesn’t break it in half.
Raw tracking on real footage (a cheetah in tall grass) produces about a thousand “tracks.” Almost all of them are flickering blades of grass and dust, strung together into random zig-zags. Plankton videos have the same problem with drifting sediment. The fix is a single number: straightness, the distance from a track’s first point to its last divided by the total path length. A swimmer heads somewhere, so it scores near 1. Jitter goes nowhere, so it scores near 0. Keeping tracks with at least 4 sightings (any two points make a perfectly straight track) and a straightness of 0.7 or more cuts a thousand tracks down to a few dozen.
Large animals have the opposite problem. MOG2 breaks a horse or a bird into dozens of separate blobs (legs, mane, splashes), and each blob gets its own track. --track-merge runs a morphological closing before tracking, with a kernel sized as a fraction of the frame’s shorter side. That fuses the fragments back into one body without changing the composite itself.
That’s one track, followed across 23 samples a quarter second apart, from the far bank to the foreground. The yellow squiggle in the corner is splash that happened to pass the filters.
One lesson: tracking needs a fixed camera. Even with the noise filtered out, the cheetah clip had no path worth drawing, because it’s a panning shot. The camera follows the cheetah, so the animal stays in the middle of the frame and the background is what moves.
Usage
pip install zooplankton-image-tool
zit -i plankton_video.mp4 --interval 0.5 --composite --entities --epsilon 30
--interval sets the number of seconds between frame captures (default 1). It accepts fractions like 0.25: lower intervals give denser trails, and tracking needs them, because an animal must not move more than --max-jump pixels between samples. --entities selects the MOG2 pipeline; --epsilon and --noise set its Thresh and MinArea, the two axes of the sweep grid.
To add tracking:
zit -i clip.mp4 --interval 0.25 --composite --entities \
--trails --tracks-csv tracks.csv
--trails draws each track onto the composite in its own color, with an arrow at the end. --tracks-csv writes track_id, frame, x, y, area for every sighting, ready for speed or path-length analysis. For large animals, add --track-merge 0.1 and a bigger --max-jump.
Takeaways
Background subtraction is not just for security cameras. MOG2 was designed for surveillance, to detect people walking through a scene, but it works well for microscopy. The statistical background model adapts to gradual lighting changes (a problem with fixed-threshold approaches), and the morphological filtering pipeline transfers directly from “person detection” to “plankton detection” because the underlying math does not care what the foreground object is. It cares about the statistics of pixel variation over time.
The contour aspect-ratio filter was born of frustration. Some microscope cameras produce horizontal scan-line artifacts: thin horizontal stripes that MOG2 correctly identifies as “not background” but that are obviously not organisms. Filtering by aspect ratio (rejecting contours wider than they are tall by a factor of 20 or more) eliminates these artifacts.