ZIT turns underwater plankton video into composite images that reveal animal locomotion patterns. Think of it as computational long-exposure photography for marine biology. You feed it a video of tiny creatures swimming around, and it produces a single image showing everywhere they went – crisp trails on a stable background, like light painting but for zooplankton.
The project is on PyPI and GitHub.
The Problem
You have a video of plankton swimming in a petri dish under a microscope. You want a single image that shows the motion paths. The naive approach – averaging all frames together – produces a blurry mess where the background dominates and the organisms become faint ghosts. Overlaying frames with alpha blending is slightly better but still noisy. The issue is that the organisms are tiny, the background is dominant, and naive compositing does not discriminate between “this pixel is a plankton” and “this pixel is a speck of sediment.”
Two Compositing Modes
Mode 1: Pixel Difference
The basic approach. Take a reference background frame, compare each subsequent frame pixel-by-pixel using Euclidean distance in RGB space. If a pixel differs from the background by more than an epsilon threshold, transfer it to the composite. If the difference is below a noise delta, ignore it. Simple, fast, and okay for clean videos with high contrast.
The problem: it picks up noise. Slight lighting variations, camera jitter, sediment particles drifting through the frame – all of these exceed the epsilon threshold and get composited as artifacts. For messy real-world microscope footage, this mode produces something that looks like a composite of plankton trails plus television static.
Mode 2: Entity Recognition (The Good One)
This is where OpenCV earns its keep. Instead of pixel-level comparison, we use MOG2 – Mixture of Gaussians background subtraction. MOG2 builds a statistical model of the background over time, learning which pixels are “normally” part of the static scene and which are transient foreground objects. Each frame gets a foreground mask: white where something is moving, black where the background is stable.
But the raw mask is still noisy. So we clean it up with morphological operations:
- Opening (erosion then dilation) – removes small noise speckles
- Closing (dilation then erosion) – fills small holes in detected organisms
- Additional dilation – slightly expands the detected regions to capture the full organism body
Then contour filtering. We find all contours in the cleaned mask and reject any that are too small (noise) or have extreme aspect ratios (scan lines, horizontal artifacts from the camera). What remains are the actual organisms.
The result is dramatically cleaner. The background is pristine, and you see only the plankton trails – exactly where each organism swam during the recording period.
The Parameter Sweep Grid
Different videos need different parameters. Water turbidity, lighting, camera resolution, organism size – all of these affect what threshold and minimum area values produce the cleanest composite. Tuning by trial and error is tedious.
So I built a sweep tool. sweep_grid.py generates a 5x5 grid of composites, sweeping across two axes: MinArea (the minimum contour area to count as a real organism) and Thresh (the background subtraction threshold). You look at the grid, find the cell that looks best, and use those parameters. Visual parameter search.
Usage
pip install zooplankton-imaging-tool
zit --video plankton_video.mp4 --interval 1.0 --epsilon 30 --mode entity
The --interval flag controls how many seconds between frame captures (default 1.0). Lower intervals give you denser trails but slower processing. The --mode entity flag selects the MOG2 pipeline.
What I Learned
The interesting technical takeaway is that background subtraction is not just for security cameras. MOG2 was designed for surveillance – detecting people walking through a scene – but it works beautifully for microscopy. The statistical background model adapts to gradual lighting changes (a problem with fixed-threshold approaches), and the morphological filtering pipeline transfers directly from “person detection” to “plankton detection” because the underlying math does not care what the foreground object is. It cares about the statistics of pixel variation over time.
The contour aspect-ratio filter was a late addition born from frustration. Some microscope cameras produce horizontal scan-line artifacts – thin horizontal stripes that MOG2 correctly identifies as “not background” but that are obviously not organisms. Filtering by aspect ratio (rejecting contours wider than they are tall by a factor of 10) eliminated these artifacts completely.
Computer vision for marine biology. Not a combination I expected to work this well. But the plankton do not care what the algorithm was designed for, and neither does the math.