Julian Henry's Blog

touch_keeper: Mass-Personalized New Year's Texts

2026-05-04T00:00:00+00:00

New Year’s Eve. 11:58 PM. You want to text “Happy New Year” to everyone in your contacts. All 200+ of them. Personalized by name. Before midnight.

You could type fast. Or you could automate it.

touch_keeper is a Python CLI tool that parses your phone contacts (VCF file), sends a personalized SMS to each contact via the Twilio API, runs a webhook server that auto-replies when people text back, and then analyzes the response data with matplotlib visualizations.

The project is on GitHub. Apache 2.0 licensed.

200+ personalized SMS messages radiating out at midnight.

The Three Stages

1. Send

touch-keeper send --contacts contacts.vcf --sender "Jules"

Parses the VCF file, extracts names and phone numbers, normalizes numbers to E.164 format (handling the chaos of inconsistent phone number formatting – parentheses, dashes, spaces, country codes, no country codes), and fires off a personalized SMS to each contact via the Twilio API:

Happy New Year, Maria!!! (~‾⌣‾)~ Cheers, Jules

Messages are spaced with a configurable delay (default 1 second) to avoid hitting Twilio’s rate limits. A dry-run mode previews all messages without sending.

2. Serve

touch-keeper serve

Launches a Flask webhook server at /sms. When Twilio receives a reply from one of your contacts, it forwards the message to your webhook. The server handles replies differently based on whether the person has replied before:

First-time replier: Gets a warm, personal-sounding response: “May the new decade find you great happiness and prosperity.”
Repeat replier: Gets a random celebration emoji from a curated set – party poppers, champagne glasses, fireworks, confetti balls.

All incoming messages are logged to response.log with the sender’s phone number and timestamp.

The auto-reply is the part that creates the most entertainment. People think they are texting a person. They are texting a Flask server. The responses are just plausible enough to sustain a back-and-forth for 2-3 messages before suspicion sets in.

3. Analyze

touch-keeper analyze --contacts contacts.vcf --log response.log

Reads the response log, maps phone numbers back to contact names using the VCF file, and generates:

A text summary of all replies per contact
A bar chart histogram of response frequency (sorted descending – who replied the most?)
A time-series scatter plot of when replies came in, with quartile lines showing the distribution of reply times

The Response Log: A Time Capsule

The included response.log contains 173 real replies from what appears to be a New Year’s Eve 2019/2020 deployment. I am going to quote some of these because they are a genuine cross-section of human reaction to receiving an automated-but-personalized text at midnight:

The grateful:

“Love you juju”

The suspicious:

“Is this some auto response shit”

The testing:

“Did you get a new phone??”

The wholesome:

“Happy New Year!! Miss you man”

The pragmatic:

“Coffee or a drink… or a blunt whatever works”

The recursive:

Someone figured out the auto-reply and kept texting to see how many different emojis they could get.

173 replies out of ~200 contacts is a remarkable response rate. Turns out, people appreciate being remembered at midnight even if the remembering was automated. The personalization (including their name) makes each message feel intentional, and the auto-reply sustains the illusion just long enough for the warmth to land.

Technical Notes

The codebase is clean modern Python. Frozen dataclasses for immutable data models (Contact, Settings). Type annotations throughout. Strict mypy. ruff for linting. pytest for testing. src/ layout convention. pyproject.toml with Hatchling build system.

The VCF parser handles the various vCard format quirks – multiple phone numbers per contact (picks the first mobile number), names stored as “Last;First” or “First Last”, phone numbers with tel: URI prefixes. Phone number normalization to E.164 strips all formatting characters and prepends the country code if missing.

The analysis module uses pandas for data manipulation and matplotlib for the visualizations. The quartile lines on the time-series plot are a nice touch – they show that most replies cluster in the first 30 minutes after midnight, with a long tail of late-night stragglers responding hours later.

Why Twilio?

Twilio is the path of least resistance for programmatic SMS. The Python SDK (twilio>=9.0) wraps the REST API cleanly, and the webhook integration with Flask is trivial – Twilio sends a POST request to your /sms endpoint with the sender’s number and message body. You respond with TwiML (Twilio Markup Language), which is XML that specifies the reply message. It is not elegant, but it works, and you can have the whole thing running in 20 lines of Flask.

The cost is roughly $0.0075 per SMS in the US. 200 contacts costs about $1.50 in Twilio credits. A dollar fifty to text your entire phone book a personalized New Year’s message. That is cheaper than a greeting card.

The Philosophical Bit

There is something slightly absurd about automating personal connection. The whole point of a New Year’s text is that someone thought of you. If a script thought of you, does it count?

I think it does, actually. The automation handles the logistics – the typing, the timing, the 200 repetitions of the same sentiment. But the decision to send the text was human. The contact list was curated by a human over years of real relationships. The message template was written by a human who wanted those specific people to feel remembered. The automation scales the intention without diluting it.

173 people replied. Several led to actual meetups in the following weeks. One person I had not spoken to in three years texted back and we reconnected. The script did not create the connection. It removed the friction that was preventing it.

Keep in touch.

Hacker de Teclado: Acoustic Side-Channel Attack

2026-04-27T00:00:00+00:00

View on GitHub

Every key on your keyboard makes a slightly different sound when pressed. The spacebar sounds different from the Enter key. The ‘A’ key sounds different from the ‘S’ key – subtly, but measurably. If you record the audio of someone typing and feed it through a trained neural network, you can reconstruct what they typed.

This project is a from-scratch reimplementation of Harrison, Toreini, and Mehrnezhad’s 2023 paper “A Practical Deep Learning-Based Acoustic Side Channel Attack on Keyboards” (arXiv:2308.01074). Written in Portuguese-flavored Python, built on PyTorch and librosa.

The project is on GitHub. MIT licensed.

The Pipeline

graph TD
    A[Raw Audio Recording] -->|librosa onset detection| B[Individual Keystroke Chunks]
    B -->|Mel Spectrogram| C[64x64 Spectral Images]
    C -->|Time-Shift Augmentation| D[Augmented Dataset]
    D -->|SpecAugment Masking| E[Masked Spectrograms]
    E -->|CoAtNet| F[36-Class Prediction: a-z, 0-9]

Three stages: collect data, train a classifier, run the attack.

Stage 1: Data Collection

The interactive recorder (quickstart.py) walks you through pressing each key on your keyboard. For each of the 36 alphanumeric keys (a-z, 0-9), it prompts you, records a short clip via your microphone using sounddevice, auto-detects the keystroke onset in the audio, and saves the isolated keystroke as a .wav file in a per-key directory.

For quick experimentation without a physical keyboard, a synthetic data generator (gerar_exemplo_zorro.py – “generate fox example”) creates fake keystroke audio using decaying sine waves at distinct frequencies. Each key gets a unique base frequency, so the spectrograms are distinguishable even though the sounds are artificial. The full pipeline (train, predict) works on synthetic data, letting you test end-to-end in under a minute.

The demo phrase is “o zorro e gris” – Portuguese for “the fox is grey.”

Stage 2: Training

The training pipeline faithfully reproduces the paper’s methodology.

Onset Detection. librosa detects keystroke onsets in the audio using energy-based peak detection with a 300ms minimum distance filter (to prevent detecting the same keypress twice).

Mel Spectrogram Extraction. Each keystroke chunk is converted into a 64-band Mel spectrogram with 1024-point FFT and 225 hop length, producing a 64x64 spectral image. This is the feature representation: a 2D image where the x-axis is time, the y-axis is frequency (mel-scaled), and pixel intensity represents energy at that time-frequency bin.

Data Augmentation. Two techniques from the paper:

Time-shift augmentation – each keystroke is speed-distorted by +/- 40% using librosa.effects.time_stretch, producing 2 augmented copies per sample. This teaches the model to be invariant to typing speed.
SpecAugment – at training time, random rectangular masks are applied to the spectrogram (2 frequency masks and 2 time masks, each spanning 10% of the axis width). This is the same augmentation technique used in speech recognition (Park et al. 2019), and it prevents the model from overfitting to specific time-frequency patterns.

The Model: CoAtNet. The classifier is a CoAtNet (Convolutional + Attention Network) – a hybrid architecture that combines the local feature extraction of convolutions with the global context modeling of self-attention:

Stem. Conv2D (1 -> 32 channels, stride 2) with BatchNorm and GELU activation. Halves spatial dimensions.
MBConv Phase. Two Mobile Inverted Bottleneck Convolution blocks, expanding to 64 then 128 channels. These are depth-wise separable convolutions with squeeze-and-excitation – the same building blocks used in EfficientNet. They capture local spectral patterns: the frequency distribution of a single keystroke.
Transformer Phase. Two Transformer Encoder layers with 4-head self-attention (d_model=128, feedforward dimension 512). These capture global dependencies across the entire spectrogram: the temporal shape of the keystroke’s decay envelope, the relationship between the initial attack and the subsequent resonance.
Classification Head. Adaptive 2D Average Pooling -> Fully Connected Linear layer -> 36-class output.

The MBConv blocks say “what does this keystroke look like locally?” The Transformer blocks say “what does the overall shape tell us?” Together, they classify each 64x64 spectrogram as one of 36 alphanumeric characters.

Training Hyperparameters. Adam optimizer with max LR 5e-4 and linear annealing over 1100 epochs. Batch size 16. 80/20 train/val split. Early stopping with patience of 50 epochs. Best model checkpointed to keystroke_model_best.pth.

The model supports Apple MPS (Metal), CUDA, and CPU fallback. On an M-series Mac, training completes in minutes.

Stage 3: The Attack

python hacker_de_teclado.py --prever recording.wav

Given a recording of someone typing, the system:

Detects each keystroke onset using the same librosa pipeline
Extracts the 64x64 Mel spectrogram for each detected keystroke
Runs each spectrogram through the trained CoAtNet
Outputs the predicted text: o z o r r o e g r i s

Why This Works

It seems implausible that a microphone across the room could distinguish ‘A’ from ‘S’. But the acoustic differences are real and arise from physical properties:

Key position. Keys in different positions on the keyboard produce different resonance patterns because the mechanical structure beneath them varies. A key near the edge of the keyboard plate has different vibrational modes than a key near the center.

Finger angle. Different keys are typically struck by different fingers at different angles. The ring finger hitting ‘A’ produces a different impact profile than the index finger hitting ‘J’.

Travel distance. Adjacent keys have slightly different travel distances and spring tensions due to manufacturing tolerances, wear patterns, and the geometry of the underlying switch mechanism.

These differences are tiny – inaudible to a human listener comparing two keystrokes – but they are consistent and repeatable. A Mel spectrogram captures them as subtle variations in the frequency distribution of the keystroke’s initial attack and subsequent decay. The CoAtNet learns to detect these patterns.

The Harrison et al. paper reports 95% accuracy on a MacBook Pro keyboard using a smartphone microphone placed nearby. The accuracy degrades with distance, ambient noise, and different keyboard models (the model trained on one keyboard does not transfer well to another, because the acoustic properties are hardware-specific).

The Security Implications

This is a real attack vector. If someone can record the audio of your typing – via a compromised microphone, a nearby phone, or even a video call where keyboard sounds leak through – they can potentially reconstruct your keystrokes. Passwords, emails, code, messages.

Defenses include:

Acoustic noise injection – playing white noise near the keyboard to mask keystroke sounds
Silent keyboards – membrane keyboards produce weaker acoustic signatures than mechanical ones
Software-based keystroke randomization – introducing random delays between keystrokes to disrupt the temporal patterns
Awareness – muting your microphone during video calls when typing sensitive information

The project is educational. It demonstrates that the attack is accessible (PyTorch + librosa + a microphone), reproducible (the synthetic data generator lets anyone test the pipeline), and frighteningly effective on controlled setups. The paper it reimplements is publicly available. The code is MIT licensed. The fox is grey.

The Swiss-Mexican Auction: Combinatorial Bidding

2026-04-20T00:00:00+00:00

View on GitHub

What if eBay let you bid on bundles?

You want a camera body, a specific lens, and a tripod – but only if you can get all three. Buying the lens without the body is useless. Buying the body without the lens is expensive paperweight acquisition. You want the bundle, all or nothing.

This is the exposure problem in auction theory, and solving it optimally is NP-complete. The Swiss-Mexican Auction is a dual-layer framework that says: forget optimality. Get close enough, fast enough, at the scale of a real marketplace.

The project is on GitHub.

Interactive conflict graph: bidder nodes orbit the auctioneer. Edges represent item conflicts between bundles.

The Problem: Satisfiability and Expenditure

Call it the SEP: Satisfiability and Expenditure Problem. You have a marketplace with $N$ items and $M$ bidders. Each bidder $i$ wants a specific bundle $O_i \subseteq {1, …, N}$ (all or nothing) and has a hard budget ceiling $B_i$. The auctioneer wants to maximize total revenue while respecting three constraints:

Budget: No bidder pays more than $B_i$
Indivisibility: Either bidder $i$ gets the entire bundle $O_i$ or nothing
Exclusivity: Each item can be allocated to at most one bidder

This is simultaneously a Set Packing problem (maximize the number of non-overlapping bundles) and a Multi-Dimensional Knapsack problem (maximize revenue subject to capacity constraints). Both are NP-hard individually. Combined, the ILP formulation is:

\[\max \sum_{i=1}^{M} B_i \cdot x_i\]

subject to:

\[\sum_{i: j \in O_i} x_i \leq 1 \quad \forall j \in \{1, ..., N\}\] \[x_i \in \{0, 1\} \quad \forall i\]

At eBay’s scale – 1.7 billion items, 134 million buyers – exact ILP solutions are computationally impossible. The Swiss-Mexican Auction does not even try.

The Two Layers

The Swiss Layer (Constraints)

The “Swiss” layer is the rigid, formal mathematical scaffolding. It defines the ILP, establishes the feasibility space, and provides theoretical upper bounds via LP relaxation (relax $x_i \in {0,1}$ to $x_i \in [0,1]$ and solve the continuous problem with a linear solver). The LP relaxation gives you a ceiling: “no feasible allocation can exceed this revenue.” You cannot achieve it, but you can measure how close your heuristic gets.

The Mexican Layer (Execution)

The “Mexican” layer is the heuristic. Fast, fluid, good enough. The algorithm:

Compute each bidder’s value density: $\rho_i = B_i / O_i $ (budget per item in the bundle)
Sort all bidders by $\rho_i$ in descending order
Iterate: for each bidder, check if all items in their bundle $O_i$ are still available. If yes, allocate. If any item is taken, skip.

That is it. $O(N \log N)$ time (dominated by the sort). The greedy heuristic favors bidders who pay the most per item – the efficient bidders – and allocates first-come-first-served among non-conflicting bundles.

Results: 50,000 Items, 20,000 Bidders, 0.26 Seconds

The simulation generates a realistic marketplace:

50,000 items with log-normal price distributions
20,000 bidders with bundle sizes ranging from 1 to 15 items
Budgets drawn from a distribution centered around the sum of desired item prices (with noise)

The greedy algorithm clears this market in 0.26 seconds. It satisfies 36% of bidders (7,200 out of 20,000) with zero item conflicts, zero budget violations, and extracts ~$945,000 in revenue.

The LP-Guided Hybrid

The “Swiss Fallback” improves on pure greedy. It solves the LP relaxation first (using SciPy’s HiGHS solver), extracts the fractional solution weights, and uses them to re-rank bidders before running the greedy pass. Bidders that the LP heavily weights get priority in the greedy allocation.

On a smaller test market (500 items, 1,000 bidders), the hybrid closes 92.8% of the gap between pure greedy and the theoretical LP upper bound, achieving an 8.8% revenue improvement. The LP solve adds computational cost, but for markets where the revenue stakes justify it, the hybrid is the clear winner.

Two Emergent Phenomena

The Anti-Whale Effect

The greedy algorithm naturally favors small, targeted bidders over large-bundle “whales.” A bidder wanting 2 items with a $500 budget has $\rho = 250$ per item. A bidder wanting 15 items with a $2,000 budget has $\rho = 133$ per item. The small bidder ranks higher. By the time we reach the whale, several of their desired items are already allocated to smaller bidders, and the whale is skipped.

This produces a more democratic marketplace. The algorithm does not discriminate by total wealth – it discriminates by efficiency. A buyer willing to pay a premium for a small, specific bundle is prioritized over a deep-pocketed buyer making a speculative grab at a large bundle. This is not a designed feature. It is an emergent property of sorting by value density.

The Tequila-Snow Phase Transition

Market liquidity collapses non-linearly as desired bundle sizes grow. This is the “Tequila-Snow” phase transition, named by analogy to statistical mechanics.

When average bundle size is 1-2 items, the market is liquid – most bidders can be satisfied because their bundles rarely conflict. When average bundle size exceeds 8-10 items, the market freezes – almost no one gets their bundle because the probability of at least one item conflict approaches 1.

The critical insight: there is a Viscous Goldilocks Zone around bundle size 3-4 where the product of satisfaction rate and revenue per bidder is maximized. Smaller bundles have high satisfaction but low per-bidder revenue. Larger bundles have high per-bidder revenue but near-zero satisfaction. The sweet spot is in the middle. This has practical implications for marketplace design: if you are building an auction platform that supports bundling, you should nudge bidders toward bundles of 3-4 items for maximum overall market efficiency.

xychart-beta
    title "Tequila-Snow Phase Transition"
    x-axis "Average Bundle Size" [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15]
    y-axis "Satisfaction Rate (%)" 0 --> 100
    line [95, 72, 54, 41, 31, 24, 18, 14, 11, 8, 2]

Future Horizons

The paper sketches several extensions:

Quantum Annealing. The ILP can be reformulated as a QUBO (Quadratic Unconstrained Binary Optimization) problem, which maps naturally onto an Ising model. D-Wave’s quantum annealers natively solve QUBO problems. For a 50,000-item market, the Ising model would have 20,000 qubits (one per bidder) with pairwise couplings encoding item conflicts. Current quantum hardware cannot handle this scale, but the formulation is ready for when it can.

Neural Mechanism Design. Train a Graph Neural Network on the conflict graph (bidders as nodes, item conflicts as edges) to learn allocation strategies that generalize across market structures. The GNN could learn market-type-specific heuristics that outperform the generic value-density sort.

Zero-Knowledge Proofs. Bidders currently reveal their budgets to the auctioneer. With zk-SNARKs, bidders could prove “my budget exceeds the price of my bundle” without revealing the actual budget. Cryptographic privacy for auction participants.

The Swiss-Mexican Auction is not optimal. By construction, it cannot be – the optimal solution is NP-complete. But it is fast, it is principled, it closes 93% of the optimality gap, and it clears a 50,000-item market in a quarter of a second. Sometimes good enough, fast enough, is the right answer.

git-goblin: 195+ Shortcuts and a Pure Bash TUI

2026-04-13T00:00:00+00:00

View on GitHub

git-goblin is a shell productivity toolkit that replaces verbose CLI commands with short, mnemonic shortcuts. 195+ aliases and functions covering Git, Docker, Kubernetes, AWS, GCP, Terraform, Helm, Ansible, and general shell utilities. One-command setup. Source it into your bash or zsh, and your terminal becomes a different animal.

The project is on GitHub. MIT licensed.

The Greatest Hits

Before we get to the interesting stuff, here is the practical pitch. These are commands I use every single day:

gg "fix the auth bug"     # git add -A && git commit -m "..." && git push
gs                        # git status
gd                        # git diff
gb                        # git branch
gco feature               # git checkout feature
gpr                       # create GitHub PR via gh cli
gsquash 3                 # squash last 3 commits
gwip                      # commit everything as WIP
gunwip                    # undo the last WIP commit
presto                    # nuclear option: wipe git history (with confirmation)

Docker:

dps                       # docker ps
dcu                       # docker compose up -d
dcd                       # docker compose down
dex container bash        # docker exec -it container bash
drmi                      # docker rmi (remove image)

Kubernetes:

kgp                       # kubectl get pods
kgs                       # kubectl get services
kl pod-name               # kubectl logs pod-name
kex pod-name bash          # kubectl exec -it pod-name -- bash
ksc context                # kubectl config use-context

AWS, GCP, Terraform, Helm, Ansible – all have similar shortcut families. The full list is 195+ commands. You do not memorize them all. You memorize the ones you use, and for the rest, there is the G command.

The `G` Command

This is the centerpiece of the project, and the reason it is more than a dotfiles repo.

G is a “hot-doc autofiller” that operates in three modes:

Mode 1: Best-Guess

G 'git status'

You type any raw CLI command – or even a vague description – and G fuzzy-matches it to the closest git-goblin shortcut. It displays the documentation and copies the shortcut to your clipboard. You typed git status? G tells you the shortcut is gs, shows you what it does, and copies gs to your clipboard. Done.

The matching uses a scoring algorithm that weights exact substring matches highest, then token overlap, then edit distance. It is not perfect. It does not need to be. It needs to be faster than grepping through a 200-line alias file, and it is.

Mode 2: Add

G -a 'mycommand' 'what it does'

Register your own custom shortcuts. They persist across sessions in ~/.git-goblin-custom and are searchable through G just like the built-in commands. Your team has a weird deploy script? Add it. Your CI pipeline has a 40-character incantation? Add it.

Mode 3: Interactive TUI

No arguments. G launches a full-screen fuzzy-find interface. Type to filter. Arrow keys to navigate. Enter to select and copy. It looks like fzf, but it is built entirely in pure Bash.

No dependencies. No fzf. No ncurses. No Python. The TUI uses raw terminal escape sequences (\033[), read -rsn1 for character-by-character input handling, and ANSI color codes for syntax highlighting. The entire interactive mode is a bash function that manages cursor position, screen clearing, and input parsing by hand.

I built it this way because I wanted G to work on any machine with bash. No installation step beyond sourcing the file. No brew install fzf prerequisite. You ssh into a bare-bones production server, source git-goblin, and the TUI works.

Setup

./setup-git-goblin

One command. It auto-detects your shell (bash or zsh), backs up your RC file, sources the function files, and makes scripts executable. Takes about 3 seconds.

The Python Utilities

git-goblin also ships with a handful of Python tools:

kash – A file-based function cache decorator. Decorate a Python function with @kash, and its return value gets cached to a .kash JSON file on disk, keyed by the arguments. Next time you call the function with the same arguments, it reads from the file instead of recomputing. It is functools.lru_cache but persistent across process restarts. Useful for expensive API calls during development.

from kash import kash

@kash
def expensive_api_call(query):
    return requests.get(f"https://api.example.com/{query}").json()

autosave-git-recursively.py – Walks a directory tree and auto-commits/pushes every git repository it finds. I run this as a cron job on my dev machine. It is an autosave for all my projects.

repoinspector.py – Fetches all of a GitHub user’s public repositories and aggregates their open issues into a single summary. Useful for triaging across many repos without clicking through GitHub’s web UI.

Why Not Just Use Oh My Zsh?

Oh My Zsh has git aliases. They are fine. git-goblin differs in three ways:

Scope. OMZ aliases are git-only. git-goblin covers Git, Docker, Kubernetes, AWS, GCP, Terraform, Helm, Ansible, and shell utilities.
The G command. OMZ does not have a command finder. You memorize the aliases or you grep through the source. G is the difference between a reference card and a searchable database.
Portability. git-goblin works on bash and zsh with no framework dependency. OMZ is a zsh framework that you install. git-goblin is a file that you source.

The built-in gg-help command provides a searchable, section-filterable reference table:

gg-help              # show everything
gg-help docker       # show only Docker shortcuts
gg-help kubernetes   # show only Kubernetes shortcuts

195 shortcuts. A fuzzy-find TUI in pure Bash. Zero dependencies. One goblin.

README Rosetta: Open Source in Every Language

2026-04-06T00:00:00+00:00

View on GitHub

Most open source projects have English-only documentation. This is a barrier. A significant percentage of the global developer population reads English as a second language, and for many, the friction of parsing a technical README in a non-native language is enough to keep them from adopting a tool they would otherwise love.

README Rosetta automates the translation of README files and Sphinx documentation into 70+ languages using locally-running LLMs via Ollama. No API keys. No cloud costs. No data leaving your machine. One command:

readme-rosetta --langs es fr de ja zh ar hi

The project is on PyPI (v0.1.7) and GitHub. MIT licensed.

The Problem with LLM Translation of Structured Text

Raw LLMs are notoriously unreliable for translating Markdown. I learned this the hard way. Here is a non-exhaustive list of things that go wrong when you naively ask an LLM to translate a README:

Code block corruption. The LLM helpfully “translates” your Python code. import requests becomes importar solicitudes. My God.
Hallucinated links. The original has [docs](https://example.com/docs). The translation has [documentos](https://ejemplo.com/documentos). That URL does not exist.
Conversational preambles. Instead of returning the translation, the model starts with “Sure! Here’s the translation of your README into Spanish:” and then provides the translation. Now your translated README begins with an English sentence from the LLM.
Line count explosion. A single-sentence bullet point becomes a three-sentence paragraph because the LLM decided to elaborate.
Placeholder mangling. If you use {variable} template syntax, the LLM either translates the variable name or drops the braces.

README Rosetta addresses all of this with a multi-layered defense system.

Code Block Protection

Before sending any text to the LLM, README Rosetta replaces all fenced code blocks with numbered placeholders:

```python
def hello():
    print("world")
```

becomes ROSETTA_CB_0. The LLM never sees the code. After translation, the placeholders are swapped back in. The code is untouched. This is the single most important feature in the entire tool, and it is embarrassingly simple.

For Sphinx documentation, the same approach extends to reStructuredText syntax: directives, roles, and inline literals are replaced with ROSETTA_RST_N placeholders. The LLM translates the prose. The markup survives.

Hallucination Detection and Retry Logic

After each translation, README Rosetta validates the output:

Placeholder count check. If the original had 5 ROSETTA_CB_N placeholders and the translation has 4, something was dropped. Retry.
Link verification. Compare URLs in the original and translation. If the translation contains URLs not present in the original, the LLM hallucinated them. Retry.
Conversational response detection. If the translation starts with patterns like “Here is,” “Sure!”, “Of course,” or “I’d be happy to,” the LLM prefixed its response with preamble. Strip it or retry.
Line count sanity. If a single input line produced 5+ output lines, the LLM elaborated instead of translating. Retry.

Each retry uses an increasingly strict system prompt. The first attempt is polite: “Please translate the following Markdown content.” The retry is firm: “Translate ONLY the text. Do NOT add any explanations, preambles, or commentary. Do NOT modify URLs, code blocks, or formatting.”

Up to 2 retries per chunk. If all fail, the original text is preserved with a comment noting the failed translation.

Translation Caching

Translations are cached in a .rosetta_cache.json file using MD5 hashes of the source content. If you re-run the tool after editing only one section of your README, only that section gets re-translated. The rest is served from cache. This matters when you are translating into 20+ languages – you do not want to re-translate 500 chunks because you fixed a typo in one paragraph.

Output Modes

Split mode (default): Generates separate files – README.es.md, README.fr.md, README.ja.md, etc. Each file is self-contained and includes a navigation table at the top (the “Rosetta stone”) with links to all translated versions.

Unified mode (--no-split): Appends all translations into a single README.md, separated by HTML comment markers. Useful for projects that want everything in one file.

The navigation table itself is auto-generated:

| [English](README.md) | [Español](README.es.md) | [Français](README.fr.md) | [日本語](README.ja.md) | [中文](README.zh.md) | ...

Sphinx Integration

For larger projects with Sphinx documentation, README Rosetta can set up the entire i18n pipeline:

readme-rosetta --sphinx --langs es fr de ja --docs-dir ./docs

This runs sphinx-build gettext to generate .pot files, creates locale directories, translates the .po files (with rST syntax protection), and builds localized HTML. The output is a complete set of translated Sphinx docs ready for hosting.

GitHub Actions

A workflow template is included that installs Ollama on a GitHub Actions runner, pulls the model, and auto-translates on every push to main. Internationalization that happens automatically in CI rather than being an aspirational backlog item that never gets done.

Usage

pip install readme-rosetta

# Translate to 5 languages
readme-rosetta --langs es fr de ja zh

# Use a specific model
readme-rosetta --model llama3.2 --langs ar hi ko

# Dry run (preview without writing)
readme-rosetta --dry-run --langs es

# Sphinx docs
readme-rosetta --sphinx --langs es fr de ja --docs-dir ./docs

Configuration can also live in pyproject.toml:

[tool.readme-rosetta]
model = "llama3.2"
languages = ["es", "fr", "de", "ja", "zh"]

The Technical Core

The source is 7 files. translator.py handles the Ollama integration, caching, and retry logic. markdown_handler.py handles code block protection/restoration and the navigation table. sphinx_handler.py handles the Sphinx i18n pipeline. lang_codes.py is a mapping of 70+ ISO language codes to language names. cli.py orchestrates everything.

The tool defaults to llama3.2 via Ollama because it is small enough to run on a laptop, fast enough for batch translation, and accurate enough for technical prose. You can swap in any Ollama-compatible model.

The hallucination detection is regex-heavy by design. I considered using a second LLM call to validate the first LLM’s output, but that doubles the cost and introduces a new failure mode (what if the validator hallucinates?). Pattern matching is deterministic, fast, and does not argue with you.

Making documentation accessible should not be hard. It should not require a translation team or a localization budget. It should be one command. That is what README Rosetta does.

Mr. Worldwide: Hello in 80 Languages as a GIF

2026-03-30T00:00:00+00:00

View on GitHub

Mr. Worldwide generates animated GIFs of a single word – “Hello,” “Love,” whatever you want – translated into 80+ languages, with each translation rendered as a frame. Each frame can optionally display the translated word over a culturally relevant photograph from that country. The result is a looping, globe-trotting GIF that says your word in English, then Spanish, then Japanese, then Amharic, then Tibetan, then Yoruba, and on and on through 80+ languages.

The project is on GitHub.

The Hard Problem: Rendering Every Script on Earth

Generating a GIF is easy. Pillow does that in five lines. The hard problem is rendering text correctly in 80+ writing systems.

“Hello” in English uses Latin characters. “こんにちは” uses CJK ideographs. “مرحبا” uses Arabic script (right-to-left). “สวัสดี” uses Thai. “ᓱᓇᑦᓯᐊᖅ” uses Canadian Aboriginal Syllabics. “བཀྲ་ཤིས་བདེ་ལེགས” uses Tibetan. Each of these scripts requires a different font, and many system fonts support only a handful of scripts.

Mr. Worldwide bundles 25+ Google Noto fonts to cover the full Unicode range. The font selection is automatic: for each character in the translated word, the tool checks which Unicode code point range it falls into and selects the corresponding Noto font. Arabic characters get NotoSansArabic, Devanagari gets NotoSansDevanagari, CJK gets NotoSansCJK, and so on.

The font sizing is also automatic. Each translation has a different string length and character width. “Hello” in English is 5 compact Latin characters. “Привет” in Russian is 6 Cyrillic characters of similar width. “こんにちは” in Japanese is 5 wide CJK characters. Mr. Worldwide measures the rendered text width for each translation and scales the font size to fit within the frame. No text gets cut off. No frames have tiny unreadable text in one corner.

Smart Contrast Coloring

When overlaying text on a country photograph (say, “Bonjour” over a picture of the Eiffel Tower), the text needs to be legible. White text on a bright sky is invisible. Black text on a dark scene is invisible.

Mr. Worldwide solves this with k-means color clustering. For each background image:

Cluster the image pixels into dominant colors using scipy.cluster.vq.kmeans
Identify the most prominent background hues
Select a text color that maximizes both contrast (distance from the dominant colors) and vibrancy (saturation)

The result is text that pops against any background – bright cyan over a dark forest, deep navy over a sunlit beach, vivid magenta over a gray cityscape.

Flag-Color Text

An alternative to smart contrast: paint each character using the colors of the corresponding country’s flag. “Hola” rendered in the red, yellow, and red of the Spanish flag. “Bonjour” in blue, white, and red. “Hallo” in the black, red, and gold of Germany.

The flag colors are extracted from SVG files stored in a banderas/ directory (banderas is Spanish for “flags”). The tool parses the fill attributes from each SVG and maps them to character indices. Character 1 gets color 1, character 2 gets color 2, and so on, cycling through the palette.

The Full CLI

python mr_worldwide.py --word Hello --use-icons --smart-colors --size 800x600

python mr_worldwide.py --word Love --flag-colors --delay 150 --output love_flags.gif

python mr_worldwide.py --text-array "Hola,Bonjour,Ciao,Hallo" --rainbow --sine-delay

Options include:

--use-icons – overlay text on country photographs
--smart-colors – k-means contrast text coloring
--flag-colors – paint text with country flag colors
--rainbow – hue-shift the text color across frames
--sine-delay – a sinusoidal timing effect that “dwells” on each frame in sequence, creating a wave-like viewing rhythm
--langs – filter to specific ISO language codes (e.g., --langs es fr de ja)
--size – output dimensions
--delay – milliseconds per frame

The example gallery includes 14 pre-built scripts covering every permutation of these options, each with a pre-generated demo GIF.

Design Decisions

Why local translations instead of an API? The 80+ translations are hardcoded in a translations.json file rather than fetched from Google Translate at runtime. This was deliberate. Machine translation APIs are rate-limited, cost money at scale, and produce inconsistent results for single-word translations (context-free translation of “Love” can yield wildly different results depending on the API’s mood). By curating the translations manually, every output is verified and stable.

Deduplication. Many languages share the same word. “Hello” is “Halo” in both Indonesian and Malay. Rather than showing two identical frames, Mr. Worldwide deduplicates translations before rendering. The GIF only shows unique strings.

Geographic ordering. The translations are sorted by region (Europe, Asia, Africa, Americas, Oceania) with priority languages (English, Spanish, Italian, French) first. This gives the GIF a natural geographic flow rather than a random jumble.

Background image sourcing. The country photographs are fetched from the Pexels API and Wikimedia Commons, organized into per-country directories. The tool selects a random image from each country’s directory, so regenerating the GIF produces visual variety even with the same parameters.

A fun intersection of internationalization, image processing, and creative coding. The kind of project that is both technically interesting (font detection across Unicode ranges, k-means color analysis) and produces something you can actually send to your friends.

SuperWand: Magic Wand for Image Retheming

2026-03-23T00:00:00+00:00

View on GitHub

SuperWand takes an image, identifies its dominant color regions using KMeans clustering, and replaces those colors with any of 18 curated aesthetic themes – Vaporwave, Cyberpunk, Tropical, Arctic, and so on. It is a magic wand for recoloring. Point it at a posterized Charizard, pick “Midnight,” and out comes a moonlit dragon.

The project is on PyPI (v0.2.10) and GitHub. Licensed under Apache 2.0.

How It Works

The pipeline has three stages.

1. Region Identification

Given an image, SuperWand uses scikit-learn’s KMeans to cluster all pixels into k regions (default 4) by color similarity. For large images, it downsamples before clustering (because running KMeans on a 4000x3000 image with 12 million pixels is a good way to watch your laptop overheat) and then applies the cluster assignments at full resolution.

Each pixel gets a label: region 0, region 1, region 2, region 3. These regions correspond roughly to the dominant color areas – the sky, the ground, the subject, the shadows.

2. Theme Injection

Each region gets mapped to a color from the chosen theme. The 18 themes are hand-curated palettes:

Theme	Vibe
Tropical	Warm greens, coral, turquoise
Cyberpunk	Neon pink, electric blue, dark purple
Vaporwave	Pastel pink, lavender, mint
Arctic	Ice blue, white, steel gray
Retro80s	Hot pink, electric cyan, chrome yellow
Volcano	Deep red, orange, obsidian black
…	(12 more)

The mapping is straightforward: the brightest region gets the lightest theme color, the darkest region gets the darkest theme color, and intermediates are matched accordingly. The result is a recolored image that preserves the original structure (edges, shapes, textures) but wears an entirely different color palette.

3. Gradient Application

Flat color replacement looks… flat. So SuperWand supports five gradient styles that can be applied per region:

Bottom-up / Top-down – vertical gradient across the region
Left-right / Right-left – horizontal gradient
Radial – gradient radiating from the region’s centroid outward

Each gradient has a configurable polarity/bias parameter that controls where the midpoint sits. The gradient is computed as a NumPy array and multiplied element-wise against the region’s pixels. The result is smooth color transitions within each region instead of uniform blocks.

SuperWand Studio

The CLI is fine for scripting, but the real fun is the Studio – a Flask-based web UI served locally at http://127.0.0.1:5001. Upload an image, and you get an interactive workspace:

Pick a theme or set custom per-region colors
Adjust the number of KMeans clusters (more clusters = finer region detection)
Apply different gradient styles to different regions
Toggle morphological flood-fill smoothing (uses SciPy’s binary dilation/closing to smooth jagged region boundaries)
See the result instantly as a live preview

The Studio also handles CSS retheming. Upload a stylesheet, and SuperWand parses all hex color codes, clusters them with KMeans, and maps the clusters to your chosen theme. The before/after is surprisingly dramatic:

CLI Usage

pip install superwand

superwand zebra.png -theme Urban

superwand charizard.png -theme Vaporwave -k 6 -gradient radial

The -k flag controls the number of color regions. More regions means finer-grained recoloring. For simple posterized art (like the Charizard), 4 regions works well. For photographs with subtle gradients, 6-8 regions capture more detail.

The Interesting Parts

The NumPy optimization was worth the effort. The naive approach – iterating over each pixel in a Python loop to check its cluster assignment and replace its color – is painfully slow for large images. The vectorized approach uses boolean mask indexing: image[labels == k] = theme_color[k] replaces an entire region in one operation. The gradient computation is similarly vectorized: generate the gradient array for the bounding box, mask it to the region shape, and multiply. No Python loops touch individual pixels.

The morphological flood filling (optional, via SciPy) addresses a visual artifact of KMeans clustering: jagged region boundaries. KMeans assigns each pixel independently, so the boundary between two regions can be noisy – a few pixels of region 1 embedded in region 2 because their color was ambiguous. Binary dilation followed by closing smooths these boundaries, producing regions with cleaner edges. It is a post-processing step borrowed from medical image segmentation, applied here to make Charizards look better.

The CSS retheming feature came from a practical itch. I was reskinning a web project and manually hunting for every hex code in the stylesheets. SuperWand automates this: it parses hex patterns from the CSS file, clusters them (because many similar shades should map to the same theme color), replaces each cluster with the nearest theme color, and writes the new stylesheet. It is the kind of thing that saves you two hours of find-and-replace.

Eighteen themes. Five gradient styles. One wand.

ZIT: Zooplankton Image Tool

2026-03-16T00:00:00+00:00

View on GitHub

ZIT turns underwater plankton video into composite images that reveal animal locomotion patterns. Think of it as computational long-exposure photography for marine biology. You feed it a video of tiny creatures swimming around, and it produces a single image showing everywhere they went – crisp trails on a stable background, like light painting but for zooplankton.

The project is on PyPI and GitHub.

The Problem

You have a video of plankton swimming in a petri dish under a microscope. You want a single image that shows the motion paths. The naive approach – averaging all frames together – produces a blurry mess where the background dominates and the organisms become faint ghosts. Overlaying frames with alpha blending is slightly better but still noisy. The issue is that the organisms are tiny, the background is dominant, and naive compositing does not discriminate between “this pixel is a plankton” and “this pixel is a speck of sediment.”

Two Compositing Modes

Mode 1: Pixel Difference

The basic approach. Take a reference background frame, compare each subsequent frame pixel-by-pixel using Euclidean distance in RGB space. If a pixel differs from the background by more than an epsilon threshold, transfer it to the composite. If the difference is below a noise delta, ignore it. Simple, fast, and okay for clean videos with high contrast.

The problem: it picks up noise. Slight lighting variations, camera jitter, sediment particles drifting through the frame – all of these exceed the epsilon threshold and get composited as artifacts. For messy real-world microscope footage, this mode produces something that looks like a composite of plankton trails plus television static.

Mode 2: Entity Recognition (The Good One)

This is where OpenCV earns its keep. Instead of pixel-level comparison, we use MOG2 – Mixture of Gaussians background subtraction. MOG2 builds a statistical model of the background over time, learning which pixels are “normally” part of the static scene and which are transient foreground objects. Each frame gets a foreground mask: white where something is moving, black where the background is stable.

But the raw mask is still noisy. So we clean it up with morphological operations:

Opening (erosion then dilation) – removes small noise speckles
Closing (dilation then erosion) – fills small holes in detected organisms
Additional dilation – slightly expands the detected regions to capture the full organism body

Then contour filtering. We find all contours in the cleaned mask and reject any that are too small (noise) or have extreme aspect ratios (scan lines, horizontal artifacts from the camera). What remains are the actual organisms.

The result is dramatically cleaner. The background is pristine, and you see only the plankton trails – exactly where each organism swam during the recording period.

The Parameter Sweep Grid

Different videos need different parameters. Water turbidity, lighting, camera resolution, organism size – all of these affect what threshold and minimum area values produce the cleanest composite. Tuning by trial and error is tedious.

So I built a sweep tool. sweep_grid.py generates a 5x5 grid of composites, sweeping across two axes: MinArea (the minimum contour area to count as a real organism) and Thresh (the background subtraction threshold). You look at the grid, find the cell that looks best, and use those parameters. Visual parameter search.

Usage

pip install zooplankton-imaging-tool

zit --video plankton_video.mp4 --interval 1.0 --epsilon 30 --mode entity

The --interval flag controls how many seconds between frame captures (default 1.0). Lower intervals give you denser trails but slower processing. The --mode entity flag selects the MOG2 pipeline.

What I Learned

The interesting technical takeaway is that background subtraction is not just for security cameras. MOG2 was designed for surveillance – detecting people walking through a scene – but it works beautifully for microscopy. The statistical background model adapts to gradual lighting changes (a problem with fixed-threshold approaches), and the morphological filtering pipeline transfers directly from “person detection” to “plankton detection” because the underlying math does not care what the foreground object is. It cares about the statistics of pixel variation over time.

The contour aspect-ratio filter was a late addition born from frustration. Some microscope cameras produce horizontal scan-line artifacts – thin horizontal stripes that MOG2 correctly identifies as “not background” but that are obviously not organisms. Filtering by aspect ratio (rejecting contours wider than they are tall by a factor of 10) eliminated these artifacts completely.

Computer vision for marine biology. Not a combination I expected to work this well. But the plankton do not care what the algorithm was designed for, and neither does the math.

Language Notes Repos

2026-03-09T00:00:00+00:00

It’s my personal passion to learn new languages, and it matters to me because it keeps me curious, humble, and connected to people I would otherwise never really meet. I like to keep my new words with me on the go. You can find them here for your perusing, and I’m curious how others do it too. Are Anki cards the most popular? Should I switch?

If you’re working through a language and want to compare approaches, feel free to browse or fork. These are living scratchpads, not polished textbooks, but that’s part of the fun.

한글 낙서: Hangul Graffiti

2026-03-09T00:00:00+00:00

There is something disarming about a language that bows before it speaks.

Korean does not merely encode information. It encodes relationships. Every verb ending is a social contract – a declaration of how you see the person standing in front of you. In English, “please sit down” works for your boss and your dog. In Korean, you’d better know the difference between 앉으세요 and 앉아, or you will insult one and confuse the other.

I have been collecting notes on Korean for a few years now. What follows is a distillation of those notes, organized not as a textbook would but as a learner actually encounters the language: in gyms, in novels, in text messages, and in the strange space between what is said and what is meant.

The Architecture of Hangul

King Sejong the Great (세종대왕) invented Hangul in 1443, and the story is almost too good to be true. The consonant shapes are modeled after the physical position of the tongue and mouth when you pronounce them:

ㄱ (g/k): the back of the tongue rising toward the soft palate
ㄴ (n): the tongue touching the upper gum ridge
ㅁ (m): the shape of closed lips
ㅅ (s): the shape of a tooth
ㅇ (ng/silent): the shape of the throat

Vowels are built from three elements: a dot (representing the sun/heaven), a horizontal line (the earth), and a vertical line (a person standing). From these three primitives, the entire vowel system unfolds:

Vowel	Sound	Construction
ㅏ	a	vertical + right dot (bright)
ㅓ	eo	vertical + left dot (dark)
ㅗ	o	horizontal + top dot (bright)
ㅜ	u	horizontal + bottom dot (dark)
ㅡ	eu	horizontal line alone
ㅣ	i	vertical line alone

This is a writing system designed by committee – but the good kind, the kind where the committee included phonologists. The result is that Korean is arguably the most rationally designed script in active use anywhere on Earth.

Pronunciation: The Rules They Don’t Teach First

Two rules I picked up early that cleared up a lot of confusion:

1. ㅅ before ㅣ or ㅑ/ㅕ/ㅛ/ㅠ becomes “sh”

The consonant ㅅ is normally an “s,” but place it before any “i” or “y” vowel and it palatalizes to “sh.” This is why 시 sounds like “shi” and 신문 (newspaper) is “shin-mun,” not “sin-mun.” The word 시작 (beginning) is “shi-jak,” and 식당 (restaurant) is “shik-dang.”

2. The silent 받침

When ㄹ sits in the 받침 (bottom consonant position), it behaves differently than you’d expect. Korean syllable blocks stack consonants and vowels into squares, and the bottom slot – the 받침 – follows its own rules of liaison and assimilation. The consonant at the bottom of one syllable bleeds into the top of the next, creating pronunciation chains that make spoken Korean sound nothing like its spelling suggests.

For example:

독립 (independence) is pronounced “dong-nip” not “dok-lip”
한국어 (Korean language) is pronounced “han-gu-geo” – the ㄱ 받침 links to the next syllable’s vowel

Grammar: Hierarchy in the Verb

Korean has seven speech levels, though modern usage mostly collapses these into four. The critical insight is that the verb ending changes based on your relationship to the listener – not the subject, the listener:

Level	Ending	When to Use	Example (to go)
Formal polite (합쇼체)	-ㅂ니다 / -습니다	Business, news, strangers	갑니다
Informal polite (해요체)	-아요 / -어요	Default safe choice	가요
Casual (해체)	-아 / -어	Close friends, younger people	가
Formal plain (해라체)	-ㄴ다 / -는다	Writing, narration, diaries	간다

The same verb 가다 (to go) is four different social acts depending on the ending. Get it wrong and you’ve made a statement about the relationship, not about where you’re going.

Conjugation in Practice

Unlike European languages with their tables of person and number, Korean verbs don’t conjugate for who is doing the action. They conjugate for how you feel about the person you’re talking to. The subject is often dropped entirely. Context carries it.

Here is 먹다 (to eat) across several constructions:

Form	Korean	Literal
Formal polite	먹습니다	(one) eats [sir/ma’am]
Informal polite	먹어요	(one) eats [politely]
Casual	먹어	eat / eats
Past (polite)	먹었어요	ate
Future (polite)	먹을 거예요	will eat
Negative	안 먹어요	doesn’t eat
Want to	먹고 싶어요	wants to eat
Can	먹을 수 있어요	can eat
Progressive	먹고 있어요	is eating

Notice how the stem 먹 stays constant while the endings do all the work. This is the agglutinative nature of Korean: you stack suffixes like LEGO bricks.

The Particles: Small Words, Heavy Lifting

Korean particles are postpositions – they attach after the noun, not before it. And they do everything.

Particle	Function	Example
은/는	Topic marker	저는 학생이에요 (As for me, I’m a student)
이/가	Subject marker	비가 와요 (Rain is coming)
을/를	Object marker	커피를 마셔요 (I drink coffee)
에	Location / time	학교에 가요 (I go to school)
에서	Location of action	집에서 공부해요 (I study at home)
의	Possession	나의 책 (my book)
도	Also/too	저도 가요 (I’m going too)
한테/에게	To (a person)	친구한테 줘요 (I give it to a friend)

The distinction between 은/는 (topic) and 이/가 (subject) is one of the deepest rabbit holes in Korean linguistics. Roughly: 은/는 sets the frame (“as for X…”), while 이/가 identifies (“it is X that…”). The sentence 제가 학생이에요 emphasizes that I am the student (maybe someone asked “who’s the student?”), while 저는 학생이에요 simply states the fact about me.

At the Gym: 헬스장에서

Some of the most useful Korean I’ve picked up comes from the gym. The phrasebook doesn’t prepare you for wanting to ask someone if they’re done with the squat rack:

Korean	English	Context
몇 세트 남았어요?	How many sets do you have left?	Politely waiting for equipment
좀 보조 해주실 수 있나요?	Can you spot me?	Asking for help on bench press
아직 쓰시고 있나요?	Are you still using this?	Gesturing at a machine
하루에 몇 끼 먹나요?	How many meals do you eat a day?	Gym small talk

And the vocabulary that comes with the culture:

헬창 – literally an abbreviation meaning something like “health fiend.” Used as a playful, almost affectionate compliment among gym-goers, though it sounds crude to outsiders. Think “gym rat” but with more edge.
빵빵하다 – describes big, pumped muscles. 근육이 빵빵! (“Muscles are poppin’!”)
오운완 (abbreviation of 오늘도 운동 완료했다) – “Finished my workout for today.” The hashtag of Korean fitness Instagram. Sometimes abbreviated further to just ㅇㅇㅇ, because even abbreviations get abbreviated.

The counter system reveals itself here too: 끼 is the counter for meals (하루에 몇 끼?), 세트 borrows the English “set,” and 체지방 (body fat) and 체중 (body weight) share the hanja character 체 (body, 體).

Reading Korean Literature: 선화 by 김이

The jump from textbook Korean to literary Korean is a canyon. I tried reading 선화 by 김이, published by 은행나무, and the opening pages alone were a vocabulary tsunami. But the prose was beautiful:

나는 타인의 흉터를 빤히 쳐다보는 버릇이 있었다.

“I had a habit of staring intently at other people’s scars.”

누구든 상처가 있다. 상처에서 흐르던 피가 굳고 딱지가 내려앉고, 딱지가 떨어진 자리에 솟은 새살이 바로 상처를 반추하게 하는 흉터였다.

“Everyone carries wounds. The blood that flowed from wounds dries, scabs settle, and the new flesh that rises where scabs have fallen – that is the scar that makes you ruminate on the wound.”

세상에 나만 흉터가 있는 게 아니었으니까.

“Because I wasn’t the only one in the world with scars.”

The vocabulary of wounds and healing in Korean is evocative:

Korean	English	Notes
흉터	scar	흉 (ugly) + 터 (site) – the site of ugliness
상처	wound	from Hanja 傷處 – place of injury
딱지	scab	also means “ticket” or “tag” – the body’s parking ticket
새살	new flesh	새 (new) + 살 (flesh) – beautifully literal
반추하다	to ruminate	from 反芻 – what cows do, applied to thought
아물다	to heal (a wound)	no hanja, pure Korean
빤히 쳐다보다	to stare intently	빤히 (fixedly) + 쳐다보다 (to gaze up at)

수고하세요: The Farewell That Means “Work Hard”

I wrote about this in Shtetl Length, but it deserves elaboration here.

수고하세요 is a casual farewell rooted in Korean work culture. It literally means something like “please labor/exert yourself,” but in practice it functions as “good work, see you later” or “keep it up.” The upper politeness register manifests as 수고하셨어요 or 수고하셨습니다, used when:

A colleague has finished a full day of work: “Great job today + goodbye”
Someone (like a cashier) has done effort on your behalf: “Thank you for your effort”

As the customer, you could say 수고하셨습니다 as a gesture of gratitude – acknowledging the work the other person has done.

The grammar is revealing:

수고 (苦勞): labor, exertion, toil
하세요: polite imperative of 하다 (to do) – “please do”
하셨어요: past tense honorific – “you did (honorably)”
하셨습니다: past tense formal honorific – the most deferential form

So the farewell literally commands someone to work hard, and the thank-you literally praises them for having worked hard. Working hard is as common a social invocation in Korean culture as invoking God in Latin cultures, or saying “take care” in English. Except 수고하세요 is more specific – it does not wish you wellness, it wishes you productive suffering.

The Korean Keyboard: 두벌식

Learning to type in Korean is its own adventure. The standard Korean keyboard layout (두벌식, “two-set”) splits consonants to the left hand and vowels to the right:

ㅂ ㅈ ㄷ ㄱ ㅅ  ㅛ ㅕ ㅑ ㅐ ㅔ
 ㅁ ㄴ ㅇ ㄹ ㅎ  ㅗ ㅓ ㅏ ㅣ
  ㅋ ㅌ ㅊ ㅍ   ㅠ ㅜ ㅡ

Hold Shift for tense consonants (ㅃ ㅉ ㄸ ㄲ ㅆ) and compound vowels (ㅒ ㅖ). The layout is phonetically organized: consonants on the left, vowels on the right. Your hands alternate with almost every keystroke, which makes Korean typing surprisingly rhythmic once you internalize it.

My early keyboard practice files are pure gibberish – mashing keys to build muscle memory. The word 산화 (oxidation) sits at the bottom of one such file, a lone recognizable word in a sea of random jamo. Progress, I suppose, is measured in the ratio of intelligible words to noise.

What Korean Teaches You About Language

Every language you learn restructures how you think. Spanish taught me that objects can have gender. Portuguese taught me that the subjunctive is not optional. French taught me that spelling and pronunciation exist in separate universes. But Korean taught me something more fundamental: that grammar can encode social relationships, that the verb is not just an action but a posture.

The language forces you to decide, before you open your mouth, who you are in relation to the person you’re speaking to. There is no neutral register. Every sentence is a tiny act of social positioning. And once you internalize this, you start noticing how English accomplishes the same thing through different mechanisms – tone, word choice, the presence or absence of “please” – all the implicit hierarchy that Korean makes explicit.

한국어를 배우는 것은 끝이 없는 여행입니다. 하지만, 그 여행이 제일 재미있는 부분이에요.

Learning Korean is a journey without end. But the journey is the best part.

Ráfaga: Log-Mean Reversion of VIX

2026-03-09T00:00:00+00:00

View on GitHub

Ráfaga – Spanish for “gust” – began as a gust of hubris, a conviction that I could model the CBOE Volatility Index with a handful of characteristic functions, some complex integration, and a prayer. The source material: Qunfang Bao’s 2013 PhD thesis, Mean-Reverting Logarithmic Modeling of VIX, a paper I found sitting in the MPRA archives like a loaded weapon waiting for someone dumb enough to pick it up.

I picked it up.

The idea is seductive in its clarity. The logarithm of the VIX follows an Ornstein-Uhlenbeck process. Mean-reverting. Elegant. You bolt on Poisson jumps with exponential size distributions to capture the fat tails, and suddenly you have a model that prices VIX futures and options with ~97% accuracy. The theory is beautiful. The implementation is where Python tried to kill me.

But before we get to the carnage, let me build the math from scratch. If you survived a semester of calculus and remember what an integral is, you can follow this. I will take you from a rubber band to a jump-diffusion characteristic function, one layer at a time.

Part I: The Rubber Band – What Mean Reversion Means

Ordinary Differential Equations and Springs

Imagine you have a number $Y$ that represents, say, the log of a stock’s volatility. Right now $Y$ is at some value, and it has a “home” it wants to return to, which we call $\theta$. The simplest model of this homing behavior is an ordinary differential equation:

\[\frac{dY}{dt} = \kappa(\theta - Y)\]

This says: the rate of change of $Y$ is proportional to how far $Y$ is from home. If $Y$ is above $\theta$, the quantity $(\theta - Y)$ is negative, so $Y$ decreases. If $Y$ is below $\theta$, the quantity is positive, so $Y$ increases. The constant $\kappa > 0$ controls the speed. Big $\kappa$, fast return. Small $\kappa$, slow drift.

You solved this in your first ODE course. Separate variables:

\[\frac{dY}{\theta - Y} = \kappa\,dt\]

Integrate both sides:

\[-\ln\lvert\theta - Y\rvert = \kappa t + C\]

Exponentiate, apply the initial condition $Y(0) = Y_0$, and you get:

\[Y(t) = \theta + (Y_0 - \theta)\,e^{-\kappa t}\]

This is exponential decay toward $\theta$. A rubber band. A spring. Nothing stochastic yet, just a deterministic pull toward home. The “half-life” of a displacement from $\theta$ is $t_{1/2} = \frac{\ln 2}{\kappa}$. If $\kappa = 5$ (annualized), the half-life is about 50 trading days. If the VIX spikes from 15 to 40, you expect it to be halfway back to its long-term mean in roughly two months. This is not a guess. This is what 20 years of daily VIX data confirms when you fit an AR(1) regression to the log-VIX time series.

Adding Noise: The Stochastic Differential Equation

The deterministic model is too clean. The VIX does not slide back to $\theta$ along a smooth exponential curve. It wobbles. It jitters. It sometimes mean-reverts in a day and sometimes takes six months. We need randomness.

In calculus you learned about the Riemann integral. In stochastic calculus, there is an analogous object called a Wiener process (or Brownian motion), denoted $W_t$. The key properties:

$W_0 = 0$
Increments $W_{t+\Delta t} - W_t$ are normally distributed with mean 0 and variance $\Delta t$
Non-overlapping increments are independent

Think of $W_t$ as the cumulative sum of infinitely many infinitesimal coin flips. At any finite time $t$, $W_t \sim \mathcal{N}(0, t)$. The paths are continuous but nowhere differentiable – a fractal jitter.

We bolt this onto our rubber band:

\[dY_t = \kappa(\theta - Y_t)\,dt + \sigma\,dW_t\]

This is the Ornstein-Uhlenbeck (OU) process, and it is the foundation of everything that follows. The first term is the deterministic pull toward $\theta$. The second term is random noise scaled by $\sigma$, the “volatility of volatility” (or vol-of-vol). Every instant, the process gets tugged toward home and simultaneously kicked by a random perturbation.

What does the solution look like? Apply the integrating factor $e^{\kappa t}$ (just like you would for a first-order linear ODE) and integrate:

\[Y_t = \theta + (Y_0 - \theta)e^{-\kappa t} + \sigma\int_0^t e^{-\kappa(t-s)}\,dW_s\]

The first two terms are the deterministic solution you already know. The third term is a stochastic integral – a weighted sum of all the random shocks from time 0 to $t$, where recent shocks (small $t - s$) are weighted heavily and old shocks (large $t - s$) are exponentially forgotten. This is the mean-reverting memory of the process.

Since the stochastic integral is a sum of normals (with deterministic weights), $Y_t$ is itself normally distributed:

\[Y_t \sim \mathcal{N}\left(\theta + (Y_0 - \theta)e^{-\kappa t},\;\frac{\sigma^2}{2\kappa}(1 - e^{-2\kappa t})\right)\]

The mean decays exponentially toward $\theta$. The variance saturates at $\frac{\sigma^2}{2\kappa}$ as $t \to \infty$. This is the stationary distribution. No matter where the process starts, it eventually forgets its initial condition and fluctuates around $\theta$ with a fixed spread determined by the ratio $\sigma^2 / \kappa$. High noise and low reversion means wide fluctuations. High reversion and low noise means the process hugs $\theta$ tightly.

This is the MRLR model: Mean-Reverting Logarithmic. We set $Y_t = \ln(\text{VIX}_t)$, so the VIX itself is log-normally distributed around $e^\theta$. The model has three parameters: $\kappa$ (reversion speed), $\theta$ (long-term log-VIX level), and $\sigma$ (vol-of-vol). Estimate them from data, and you can price VIX futures and options.

But Why Mean-Reverting? Why Not a Random Walk?

This is worth pausing on, because the choice of mean reversion is not cosmetic. It is the single most consequential modeling decision in the entire project, and if you get it wrong, everything downstream is garbage.

Stock prices are typically modeled as geometric Brownian motion – a random walk with drift. The idea is that Apple’s stock price today does not “want” to return to any particular level. There is no gravitational home. If AAPL is at 200, it is just as likely to wander to 250 as to drift back to 150. The efficient market hypothesis says the current price already reflects all information, so there is no force pulling it anywhere. You model it as a random walk and that is a reasonable first approximation.

The VIX is not a stock price. The VIX is a measure of fear.

Fear has a baseline. Humans cannot sustain panic indefinitely. Markets cannot sustain 80-vol indefinitely. When the VIX is at 40, traders are terrified, hedging is expensive, and the entire options market is priced for Armageddon. But Armageddon, as a rule, does not last. The panic subsides. The hedges get unwound. Implied volatility collapses back toward its long-term average. Conversely, when the VIX is at 10, complacency reigns. Everyone is short vol. Everyone is selling puts. And then some exogenous shock – a pandemic, a war, a bank failure, a leveraged fund blowing up – arrives, and the VIX spikes back up. The floor is soft but real.

This is mean reversion. The VIX has a home. It wanders away from home – sometimes violently – but it always comes back. Twenty-two years of daily data confirm this: fit an AR(1) regression to log-VIX, and the coefficient $\beta$ is significantly less than 1. If $\beta = 1$, you have a random walk. If $\beta < 1$, you have mean reversion. For the VIX, $\beta \approx 0.98$ at a daily frequency, which implies an annualized $\kappa$ of roughly 5. The VIX is not a random walk. It is a rubber band anchored to a long-term mean of about $e^{2.8} \approx 16$.

If you modeled the VIX as a random walk, your futures prices would be wrong (they would not converge to a long-term level), your option prices would be wrong (you would misprice the term structure), and your hedging ratios would be wrong (you would ignore the gravitational pull). Mean reversion is not an assumption. It is a fact about what the VIX is – a fear gauge with a home address.

The Ornstein-Uhlenbeck process is the simplest continuous-time model that captures this fact. It is exactly a random walk plus gravity. That is why Bao chose it, and that is why it works.

Part II: Pricing – From Process to Product

VIX Futures

A VIX futures contract is an agreement to exchange cash based on the VIX at some future date $T$. Under risk-neutral pricing, the futures price $F(t, T)$ is the expected value of $\text{VIX}_T$ given today’s information:

\[F(t, T) = \mathbb{E}^{\mathbb{Q}}[\text{VIX}_T \,\vert\, \mathcal{F}_t]\]

Since $Y_T = \ln(\text{VIX}_T)$ is normal under MRLR, $\text{VIX}_T = e^{Y_T}$ is log-normal, and the expected value of a log-normal random variable $e^X$ where $X \sim \mathcal{N}(\mu, \sigma^2)$ is $e^{\mu + \sigma^2/2}$. Plug in our expressions for the mean and variance of $Y_T$:

\[F(t, T) = \exp\!\left(e^{-\kappa\tau}\ln(\text{VIX}_t) + \theta(1 - e^{-\kappa\tau}) + \frac{\sigma^2}{4\kappa}(1 - e^{-2\kappa\tau})\right)\]

where $\tau = T - t$. Written more compactly:

\[F(t, T) = \text{VIX}_t^{\,\phi} \cdot M\]

where $\phi = e^{-\kappa\tau}$ is the mean-reversion factor and $M$ absorbs the rest. As $\tau \to \infty$, $\phi \to 0$, and the futures price converges to a constant determined solely by $\theta$ and $\sigma/\kappa$. The current VIX level becomes irrelevant for long-dated futures. This is the term structure of VIX futures, and it is why VIX futures are almost always in contango (upward-sloping) – the long-term expected VIX sits above the current spot when VIX is low.

VIX Options (Black-Scholes Variant)

A European call option on the VIX with strike $K$ and expiry $T$ pays $\max(\text{VIX}_T - K, 0)$ at expiry. Under risk-neutral pricing:

\[C(t, T, K) = e^{-r\tau}\,\mathbb{E}^{\mathbb{Q}}[\max(\text{VIX}_T - K, 0)]\]

Since $\text{VIX}_T$ is log-normal under MRLR, this expectation has a closed-form solution identical in structure to the Black-Scholes formula. Define:

\[\sigma_{\text{eff}}^2 = \frac{\sigma^2}{2\kappa}(1 - e^{-2\kappa\tau})\]

This is the total variance of $\ln(\text{VIX}_T)$ over the interval $[t, T]$. Then:

\[d_1 = \frac{\ln(F/K) + \frac{1}{2}\sigma_{\text{eff}}^2}{\sigma_{\text{eff}}}, \qquad d_2 = d_1 - \sigma_{\text{eff}}\] \[C(t, T, K) = e^{-r\tau}\left(F\,\Phi(d_1) - K\,\Phi(d_2)\right)\]

where $\Phi$ is the standard normal CDF. This is it. The MRLR option price. It looks like Black-Scholes because it is Black-Scholes – just with a mean-reverting forward and an effective volatility that accounts for the OU dynamics instead of geometric Brownian motion.

The problem: this works fine for at-the-money options but fails on the wings. The log-normal distribution from pure diffusion does not have fat enough tails to explain the prices of deep out-of-the-money VIX calls. The market prices those calls as though the VIX can suddenly spike 10 points in a day – because it can. The MRLR model, with its smooth Gaussian increments, says this is vanishingly unlikely. The market disagrees.

We need jumps. Jump like Jordan – the VIX does not get to 80 by dribbling.

Part III: The Jumps – Making the Model Honest

What Is a Jump Process?

In the OU process, the path of $Y_t$ is continuous. It wiggles, but it never teleports. In reality, the VIX can leap from 15 to 30 overnight (February 5, 2018, anyone?). A continuous process cannot produce this behavior without an absurdly large $\sigma$, which would then overestimate volatility the rest of the time.

Enter the Poisson process $N_t$. This is a counting process: $N_t$ counts the number of “events” (jumps) that have occurred by time $t$. The jumps arrive randomly at rate $\lambda$ – on average, $\lambda$ jumps per unit time. The probability of exactly $k$ jumps in an interval of length $\Delta t$ is:

\[P(N_{t+\Delta t} - N_t = k) = \frac{(\lambda \Delta t)^k}{k!}\,e^{-\lambda \Delta t}\]

For small $\Delta t$, there is at most one jump (probability $\approx \lambda \Delta t$) or no jump (probability $\approx 1 - \lambda \Delta t$). The Poisson process is the simplest model of rare, discrete events.

Now, each time a jump arrives, the VIX does not just nudge – it leaps by a random amount $J$. In Bao’s model, the jump sizes are exponentially distributed with parameter $\eta > 0$:

\[J \sim \text{Exp}(\eta), \qquad f_J(j) = \eta\,e^{-\eta j}, \quad j \geq 0\]

The mean jump size is $1/\eta$. The jumps are always positive – the VIX jumps up. This is physically correct: volatility spikes are sudden upward moves driven by panic. Volatility does not “spike” downward; it decays downward, which the mean-reversion term already handles. The asymmetry between jumps (sudden, upward) and decay (gradual, gravitational) is the fundamental dynamic of the VIX, and the MRLRJ model captures both.

The MRLRJ Stochastic Differential Equation

Bolt the Poisson jumps onto the OU process:

\[dY_t = \kappa(\theta - Y_t)\,dt + \sigma\,dW_t + J_t\,dN_t\]

Three terms:

$\kappa(\theta - Y_t)\,dt$ – the deterministic pull toward home
$\sigma\,dW_t$ – continuous random noise
$J_t\,dN_t$ – discrete random jumps: at each Poisson event (on average $\lambda$ times per year), $Y_t$ increases by a random amount $J_t \sim \text{Exp}(\eta)$

This is the MRLRJ model: Mean-Reverting Logarithmic with Random Jumps. Five parameters now: $\kappa$, $\theta$, $\sigma$, $\lambda$ (jump frequency), and $\eta$ (inverse mean jump size).

Why Jumps Break Closed-Form Pricing

Under MRLR, $Y_T$ was normally distributed, so $\text{VIX}_T$ was log-normal, and we used the Black-Scholes formula. Under MRLRJ, $Y_T$ is not normally distributed. The jump component adds a mixture of shifted exponentials to the Gaussian base, creating a distribution with a heavier right tail – exactly the fat tail the market is pricing.

There is no closed-form expression for the CDF of this distribution. You cannot write down $d_1$ and $d_2$ and call it a day. But there is another way in.

The Characteristic Function

Here is where we need one idea from analysis that you may not have seen in a standard calculus sequence, but which is straightforward once stated.

The characteristic function of a random variable $X$ is:

\[\psi_X(u) = \mathbb{E}[e^{iuX}] = \int_{-\infty}^{\infty} e^{iux}\,f_X(x)\,dx\]

This is the Fourier transform of the probability density $f_X$. The function $\psi_X(u)$ encodes all distributional information about $X$. If you know $\psi_X$, you can recover $f_X$ by inverse Fourier transform. You can compute any moment by differentiating $\psi_X$ at $u = 0$. And crucially, you can compute option prices.

Why use characteristic functions instead of the density? Because for many processes – including jump-diffusions – the characteristic function has a closed-form expression even when the density does not.

For the MRLRJ model, after considerable algebra (Bao’s paper works through it in detail), the characteristic function of $Y_T = \ln(\text{VIX}_T)$ given today’s state is:

\[\psi(u) = \exp\!\Big(iu\phi\ln(\text{VIX}_t) + iu\theta(1-\phi) - \frac{u^2\sigma^2}{4\kappa}(1 - e^{-2\kappa\tau}) + \frac{\lambda}{\kappa}\ln\frac{\eta - iu\phi}{\eta - iu}\Big)\]

where $\phi = e^{-\kappa\tau}$ as before. Let me walk through each term so you see where it comes from:

Term 1: $iu\phi\ln(\text{VIX}_t)$. This is the initial condition, decayed by mean reversion. The factor $\phi = e^{-\kappa\tau}$ tells you how much the current VIX level matters at expiry. For short-dated options ($\tau$ small), $\phi \approx 1$ and the current VIX dominates. For long-dated options, $\phi \to 0$ and the initial condition fades.

Term 2: $iu\theta(1-\phi)$. This is the mean reversion target. As $\tau \to \infty$, $(1-\phi) \to 1$, and this term becomes $iu\theta$ – the characteristic function converges to one centered at $\theta$.

Term 3: $-\frac{u^2\sigma^2}{4\kappa}(1 - e^{-2\kappa\tau})$. This is the diffusion variance. It is the $-u^2\sigma^2/2$ term you see in the characteristic function of any Gaussian, modified by the mean-reversion-adjusted total variance $\frac{\sigma^2}{2\kappa}(1 - e^{-2\kappa\tau})$.

Term 4: $\frac{\lambda}{\kappa}\ln\frac{\eta - iu\phi}{\eta - iu}$. This is the jump contribution, and it is the term that makes everything interesting and difficult. It arises from integrating the Poisson-exponential jump kernel against the mean-reverting dynamics. The $\lambda/\kappa$ prefactor reflects the interplay between jump frequency and reversion speed: if $\kappa$ is large, the process reverts so fast that jumps are quickly absorbed, reducing their cumulative impact. The logarithm of the complex ratio introduces branch cuts in the complex plane, which is exactly why Python’s float64 arithmetic could not handle the numerical integration. More on that shortly.

If you set $\lambda = 0$ (no jumps), term 4 vanishes and you recover the pure MRLR characteristic function, which is Gaussian. The jump term deforms the Gaussian into a heavier-tailed distribution.

Gil-Pelaez Inversion: From Characteristic Function to Option Price

We have $\psi(u)$ in closed form. We need option prices. The bridge is the Gil-Pelaez inversion theorem (1951), which expresses cumulative probabilities directly in terms of the characteristic function:

\[\Pi = \frac{1}{2} + \frac{1}{\pi}\int_0^{\infty} \frac{\text{Im}\!\left[\psi(u)\,e^{-iu\ln K}\right]}{u}\,du\]

This integral recovers the probability that $\text{VIX}_T > K$ (or a risk-adjusted variant thereof). The option price is:

\[C(t, T, K) = e^{-r\tau}\left(F\,\Pi_1 - K\,\Pi_2\right)\]

where $\Pi_1$ and $\Pi_2$ use slightly modified characteristic functions (one shifted to account for the asset-or-nothing payoff, one unshifted for the cash-or-nothing payoff). This is structurally identical to the Black-Scholes decomposition $C = e^{-r\tau}(F\,\Phi(d_1) - K\,\Phi(d_2))$, except $\Phi(d_1)$ and $\Phi(d_2)$ are replaced by $\Pi_1$ and $\Pi_2$ computed via numerical integration of the characteristic function.

The integral converges because $\lvert\psi(u)\rvert \to 0$ as $u \to \infty$ (the variance term $-u^2\sigma^2/(\cdots)$ in the exponent dominates). In practice, we truncate the integration at some upper limit ($u = 100$ for BigFloat, $u = 50$ for Float64) and use adaptive Gauss-Kronrod quadrature (QuadGK in Julia).

This is the entire pricing pipeline. Three layers of math: (1) the SDE defines the dynamics, (2) the characteristic function encodes the distribution in closed form, (3) Gil-Pelaez inversion recovers option prices via a single numerical integral. No Monte Carlo. No PDE solvers. One integral per option price.

Part IV: The Python Disaster

The original codebase was Python. I wrote it in October 2022 under files named, with escalating confidence, cool.py, cooler.py, and coolest.py. The naming convention should tell you everything about the trajectory of the project.

The MRLR model worked fine. Black-Scholes with a mean-reverting forward. Python handles this. Python is happy.

Then I introduced jumps.

The characteristic function’s jump term – $\frac{\lambda}{\kappa}\ln\frac{\eta - iu\phi}{\eta - iu}$ – involves a logarithm of a ratio of complex numbers. In Python’s complex128 (which is just two float64 values glued together), this term is numerically treacherous. When the integration variable $u$ is large, the imaginary parts of $\eta - iu\phi$ and $\eta - iu$ dominate the real part $\eta$, and the logarithm of their ratio develops branch cut sensitivity. The exponential of the full characteristic function then involves $e^z$ where the real part of $z$ can be $-200$ or worse. Python’s float64 maps this to exactly zero – underflow – and the integrand becomes discontinuous in ways that scipy.integrate.quad silently integrates through, occasionally returning a probability greater than 1.0.

A probability. Greater than one. I stared at that output for an hour before I understood what was happening.

I suppressed the warnings. I actually wrote this with my own hands:

warnings.simplefilter(action='ignore', category=scipy.ComplexWarning)
warnings.simplefilter(action='ignore', category=integrate.IntegrationWarning)

If you are suppressing integration warnings in a project whose entire purpose is numerical integration, you have lost the plot. I had lost the plot.

The calibration optimizer would converge to parameters that made no physical sense. Negative jump intensities. Mean reversion speeds of $10^{12}$. The Nelder-Mead simplex would wander into regions of parameter space where the characteristic function returned NaN, and scipy.optimize would shrug and keep going. I spent weeks debugging numerical issues that were not bugs in my code but fundamental limitations of double-precision arithmetic applied to equations that demand better.

My God.

Part V: Julia

Julia solved the problem in a weekend.

Not because Julia is magic. Because Julia has BigFloat as a first-class citizen. You declare your struct fields as BigFloat, and the entire computation – every intermediate exponential, every complex logarithm, every term in the characteristic function – propagates at arbitrary precision. No wrapper libraries. No mpmath imported from some dusty corner of PyPI. It is just… how the language works.

The core module is 122 lines. Here is the characteristic function, the thing that Python could not compute without lying to me:

function characteristic_function(m::MRLRJ, t, T, s, VIX_t)
    τ = BigFloat(T - t)
    ϕ = exp(-m.κ * τ)
    s_bf = Complex{BigFloat}(s)
    
    term1 = im * s_bf * ϕ * log(BigFloat(VIX_t))
    term2 = im * s_bf * m.θ * (1 - ϕ)
    term3 = -s_bf^2 * m.σ^2 * (1 - exp(-2*m.κ*τ))/(4*m.κ)
    term4 = (m.λ/m.κ) * log((m.η - im*s_bf*ϕ)/(m.η - im*s_bf))
    
    return ComplexF64(exp(term1 + term2 + term3 + term4))
end

Every term computed in BigFloat. The final result cast back to ComplexF64 for the integrator. No underflow. No overflow. No lies. The Gil-Pelaez inversion integrates cleanly with QuadGK at rtol=1e-6, and the option prices come out correct.

Julia’s type dispatch also meant I could define separate vix_option methods for MRLR and MRLRJ without any inheritance boilerplate. The pure diffusion model uses the analytic Black-Scholes formula. The jump model uses Fourier inversion. Same function name. The compiler figures it out. It is the kind of thing that makes you wonder why you spent years writing if isinstance(model, MRLRJ) branches in Python like some kind of animal.

I also built a fast Float64 variant for calibration loops where you are evaluating the objective function thousands of times and BigFloat arithmetic is too slow. Same math, narrower integration range (50.0 vs 100.0), looser tolerance (rtol=1e-4 vs 1e-6). Use Float64 for parameter search, BigFloat for final pricing. The two modules coexist. It works brilliantly.

Part VI: Calibration – Fitting the Model to Reality

A model with five free parameters is a toy until you show it can reproduce market prices. Calibration is the process of choosing $(\kappa, \theta, \sigma, \lambda, \eta)$ so that the model’s theoretical option prices match the observed market prices as closely as possible.

Time-Series Calibration (the $\mathbb{P}$-measure)

The simplest approach: fit an AR(1) regression to daily log-VIX levels. If $y_t = \ln(\text{VIX}_t)$, then the discrete-time version of the OU process is:

\[y_{t+1} = \alpha + \beta\,y_t + \varepsilon_t\]

where $\varepsilon_t \sim \mathcal{N}(0, \sigma_\varepsilon^2)$. The relationship to the continuous parameters is:

\[\beta = e^{-\kappa\,\Delta t}, \quad \alpha = \theta\,\kappa\,\Delta t, \quad \sigma = \frac{\sigma_\varepsilon}{\sqrt{\Delta t}}\]

with $\Delta t = 1/252$ (one trading day). Run OLS on 20 years of daily VIX data (5,500+ observations), and you get the historical $\kappa$, $\theta$, $\sigma$ under the real-world $\mathbb{P}$-measure.

For the jump parameters, I separate diffusion from jumps by identifying residuals that exceed 2.5 standard deviations from the AR(1) fit. These outliers are classified as jump days. The jump intensity $\lambda_{\mathbb{P}}$ is estimated as the fraction of jump days per year, and $\eta_{\mathbb{P}}$ is estimated from the mean size of the detected jumps. It is crude. It works. The filtered AR(1) (excluding jump days) gives cleaner estimates of the diffusion parameters.

Options-Chain Calibration (the $\mathbb{Q}$-measure)

The more powerful approach: fit the model to live option prices. Take a snapshot of the VIX options chain – say, April 16, 2021, with 33 days to May 19 expiry and spot VIX at 16.25. You have 20-30 strikes with observed last prices. Define the objective function:

\[\text{MAPE}(\kappa, \theta, \sigma, \lambda, \eta) = \frac{1}{N}\sum_{i=1}^N \frac{\lvert C_{\text{model}}(K_i) - C_{\text{market}}(K_i)\rvert}{C_{\text{market}}(K_i)}\]

Minimize this using Nelder-Mead (a derivative-free simplex method, appropriate because the objective is non-smooth and the characteristic function integration makes autodiff impractical). Starting guess: $\kappa = 5$, $\theta = 2.8$, $\sigma = 1$, $\lambda = 2$, $\eta = 2$. The optimizer converges to physically meaningful parameters. The MAPE settles under 3%.

The critical subtlety: the parameters recovered from options calibration are $\mathbb{Q}$-measure (risk-neutral) parameters, not $\mathbb{P}$-measure (historical) parameters. The two sets of parameters are different because investors are risk-averse. They pay a premium for protection, which inflates the implied jump intensity $\lambda_{\mathbb{Q}}$ relative to the historical $\lambda_{\mathbb{P}}$. This gap between $\mathbb{P}$ and $\mathbb{Q}$ is the volatility risk premium, and it is where the money lives. More on this in the trading section.

Part VII: Results

The Julia implementation verifies and confirms Bao’s primary findings:

Jump impact: The addition of the jump component significantly improves pricing accuracy for deep out-of-the-money options. Without jumps, the MRLR model systematically underprices OTM VIX calls because the log-normal distribution cannot generate the fat right tail that the market expects. With jumps, the MRLRJ model captures the skewness of the VIX options surface. The jump intensity $\lambda$ and mean jump size $1/\eta$ together control the weight of the right tail.
Mean reversion dominance: The parameters $\kappa$ and $\theta$ remain stable and physically interpretable across different calibration windows. The implied long-term VIX level $e^\theta$ sits consistently in the 16-20 range, matching the historical unconditional mean. The reversion speed $\kappa$ sits in the 3-8 range, implying a half-life of 30-80 trading days. This is not a fitted artifact. It is the dominant dynamic of the VIX.
Characteristic function integrity: The BigFloat implementation solves the numerical instability that plagues standard-precision routines. The probabilities $\Pi_1$ and $\Pi_2$ integrate cleanly. No warnings suppressed. No probabilities greater than one.
Cross-temporal validation: The model was calibrated against the April 16, 2021 VIX options chain (33 days to May 19 expiry, spot VIX at 16.25) and, independently, against live-scraped March 2026 data (spot VIX at 29.49, 40 strikes from 11 to 95). Both calibrations converge to plausible parameters. Both produce sub-3% MAPE. The model works in low-vol and high-vol regimes.

From a 2013 paper. In 122 lines of Julia. After Python tried to gaslight me with probabilities greater than one.

Part VIII: How to Actually Make Money With This

Fitting the market to 97% accuracy does not give you a crystal ball. What it gives you is a highly precise mathematical map of how the market currently values risk across different strikes and expirations. The gap between what the model implies and what the market does is where the edge lives. Let me be specific.

Strategy 1: Harvesting the Volatility Risk Premium ($\mathbb{P}$ vs. $\mathbb{Q}$)

There are two probability worlds. The real world ($\mathbb{P}$, estimated from historical data) and the risk-neutral world ($\mathbb{Q}$, implied by option prices). Investors are structurally terrified of VIX spikes – volatility is the one asset class where fear is a permanent fixture, because a VIX spike means your equity portfolio is hemorrhaging. So they overpay for VIX call options as insurance.

This overpayment is measurable. Calibrate the MRLRJ to both worlds:

Historical ($\mathbb{P}$): Fit AR(1) + jump detection to 20 years of daily log-VIX. Extract $\lambda_{\mathbb{P}}$ (how often jumps actually happen) and $\eta_{\mathbb{P}}$ (how large they actually are).
Implied ($\mathbb{Q}$): Calibrate to the live options chain. Extract $\lambda_{\mathbb{Q}}$ and $\eta_{\mathbb{Q}}$.

If $\lambda_{\mathbb{Q}} \gg \lambda_{\mathbb{P}}$ – the options market implies jumps happen far more often than they historically do – the tail insurance is overpriced. This is the volatility risk premium. You sell OTM VIX calls, dynamically hedge your directional exposure using VIX futures, and collect the premium as the statistically expected jumps fail to materialize.

The sizing is precise: the MRLRJ model tells you exactly how much premium each strike contains in excess of its actuarially fair value. You are not selling vol blindly. You are selling the specific strikes where the $\mathbb{P}$-$\mathbb{Q}$ gap is widest, weighted by the model’s confidence.

Warning: when jumps do materialize, they materialize violently. The VIX went from 13 to 37 overnight in February 2018 (the “Volmageddon” event). If you are short OTM VIX calls without proper tail hedging, you will be annihilated. The MRLRJ model gives you the tools to measure this tail risk – the jump intensity $\lambda$ and the mean jump size $1/\eta$ together determine the expected loss from a single jump event – but no model can save you from insufficient capital or reckless position sizing.

Strategy 2: Relative Value on the Skew

The MRLRJ model produces a smooth “fair value” curve across strikes for a given expiry. The residuals – strikes where the market price deviates from the model’s theoretical price – are where supply-demand dislocations create opportunity.

Example: a large pension fund buys a massive block of VIX 35 calls to hedge its equity portfolio. This specific strike gets temporarily bid up. The model says the VIX 35 Call should be $0.60 but it is trading at $0.80. Meanwhile, the neighboring VIX 40 Call is perfectly priced at $0.48. You sell the expensive 35, buy the fair 40 – a bear call spread – and isolate the dislocation. You are not betting on the direction of the VIX. You are betting that one specific strike will revert to the smooth mathematical curve that the rest of the chain already sits on.

This requires a model accurate enough to distinguish genuine mispricing from model error. At 97% accuracy and sub-3% MAPE, the MRLRJ gives you that resolution. At 85% accuracy, you are trading your own model’s noise.

Strategy 3: Jump-Aware Delta Hedging

You cannot buy or sell “spot VIX.” It is a computed index, not a tradeable asset. If you sell VIX options to collect premium (from Strategy 1 or as a market maker), you must hedge your directional risk using VIX futures. The question is: how many futures per option?

The answer is the delta – the sensitivity of the option price to the underlying. A Black-Scholes delta is wrong for VIX options. Not slightly wrong. Categorically wrong. Black-Scholes assumes continuous paths. The VIX jumps. When it jumps, the continuous-path delta underestimates the position’s sensitivity, and your hedge breaks.

The MRLRJ delta, computed by bumping VIX_t in the Gil-Pelaez integral and taking the numerical derivative, accounts for the probability-weighted impact of jumps. For a short-dated OTM VIX call, the MRLRJ delta can exceed the Black-Scholes delta by 30-40%. That gap is the jump risk that Black-Scholes ignores. If you hedge with the wrong delta, a 5-point VIX spike will blow through your position. If you hedge with the MRLRJ delta, the same spike is accounted for.

This is what allows institutional market makers to safely capture the bid-ask spread on VIX options while remaining directionally neutral. The model is the hedge.

Strategy 4: Calendar Spreads via Mean Reversion

When the VIX spikes to 40 during a panic, near-term options become absurdly expensive. The entire term structure inverts – nearby futures trade above deferred futures (backwardation), and short-dated implied volatility explodes. But the MRLRJ model explicitly quantifies how fast the VIX will collapse back to $\theta$.

The half-life of a VIX spike is $t_{1/2} = \ln(2)/\kappa$. If $\kappa = 5$, the half-life is about 50 trading days. If the VIX is at 40 and $\theta$ implies a long-term mean of 18, the model predicts:

After 50 days: VIX $\approx$ 29 (halfway back)
After 100 days: VIX $\approx$ 23.5
After 150 days: VIX $\approx$ 20.75

This decay curve dictates the relative pricing of near-term vs. deferred options. If the market’s term structure implies a slower decay than $\kappa$ predicts, deferred options are underpriced relative to near-term options. You sell near-term VIX calls (which are pricing in sustained panic) and buy deferred-month options (which are underpricing the speed of mean reversion). The model gives you the exact weighting by computing theoretical prices at both expirations and identifying the term-structure dislocation.

This is not a vibes trade. This is a calibrated bet on the speed of panic dissipation, sized by a model that has been validated against 20 years of VIX history and live options chains.

Strategy 5: VIX/SPX Variance Risk Premium Arbitrage

The VIX is derived from SPX options. It is, by construction, the square root of the 30-day expected variance of the S&P 500, calculated from a strip of SPX option prices. This means you can compute a “theoretical VIX” directly from the live SPX options chain and compare it to the traded VIX futures/options.

The difference between the theoretical VIX (from SPX math) and the traded VIX (from the VIX options market) is the Variance Risk Premium (VRP). When fear is elevated, traders bid up VIX derivatives beyond what the underlying SPX options math justifies. The VRP stretches.

You can construct a relative-value trade: short the overpriced VIX options and long a delta-neutral SPX straddle. You are hedged against actual market movement – if the S&P drops and volatility rises, your SPX straddle profits while your short VIX position loses, and the net exposure is the VRP. When the premium collapses back to fair value, you collect.

The MRLRJ model’s role here is to provide the theoretical anchor – the “fair” VIX options price given the model’s parameters – against which you measure the VRP. Without a precise model, you cannot distinguish a stretched VRP (tradeable) from genuine repricing of tail risk (not tradeable).

Part IX: Future Work

The MRLRJ is a 2013 model. It works. But modern quantitative finance has moved in directions that suggest several extensions, each with specific trading implications.

Rough Volatility

Recent literature (Gatheral, Jaisson, Rosenbaum, 2018) demonstrates that volatility is “rough” – it scales locally like a fractional Brownian motion with Hurst parameter $H < 0.5$, rather than the $H = 0.5$ of standard Brownian motion. The practical implication: standard MRLRJ misprices short-dated VIX options because it cannot generate the steep volatility skew observed in sub-2-week expirations without relying on unrealistic jump parameters. Replacing the Brownian driver with a fractional Brownian motion (a “Rough-MRLRJ” model) would accurately capture the short-term skew, enabling systematic selling of overpriced short-dated OTM VIX options – a trade that is currently mispriced by classical models.

Self-Exciting Jumps (Hawkes Processes)

In the standard MRLRJ, jumps arrive independently via a Poisson process with constant intensity $\lambda$. In reality, financial panics cluster. One jump makes the next jump more likely. The VIX does not spike once and return to calm; it spikes, spikes again, spikes a third time, and then slowly decays. Upgrading the Poisson process to a Hawkes process – a self-exciting point process where each jump temporarily increases $\lambda$ – would allow quantification of the “decay rate of panic.” This enables precise timing of calendar spreads: you wait for the Hawkes intensity function to peak and begin decaying, then sell near-term volatility (priced for continued contagion) and buy deferred volatility (underpricing the coming calm).

Neural SDEs and Deep Calibration

Classical calibration (Nelder-Mead on MAPE) is an end-of-day procedure. You recalibrate when new option prices arrive, which means your parameters are always stale. Universal Differential Equations (Neural SDEs) replace the static drift and volatility functions with neural networks trained continuously on tick-by-tick data. The neural components can detect micro-regime shifts – a sudden vanishing of bid-side liquidity on SPX futures, a spike in VIX call open interest – and adjust the model parameters before the VIX index actually moves. This is the frontier of intraday volatility trading: front-running regime shifts by milliseconds, buying VIX call butterflies at “stale” prices before the market reprices.

Deep Hedging (Reinforcement Learning)

Classical delta hedging assumes frictionless markets. VIX options have wide bid-ask spreads. If you delta-hedge every time the model says your delta shifted by 0.01, you will cross the spread a thousand times and bleed your alpha through transaction costs. Deep Reinforcement Learning wraps the MRLRJ model inside a simulated trading environment and trains an agent to hedge optimally – accounting for transaction costs, spread width, gamma risk, and time-of-day liquidity patterns. The agent learns that sometimes the correct hedge is no hedge at all. This is the difference between a model that is theoretically correct and a model that makes money.

Joint SPX-VIX Modeling

The MRLRJ models the VIX in isolation, but the VIX is a derivative of SPX options. A coupled model – SPX following a local-stochastic volatility model with jumps, VIX deterministically derived from the SPX options strip, with a stochastic VRP overlay modeled by MRLRJ – would enable true cross-asset arbitrage. You would compute the theoretical VIX in real-time from SPX options, compare to traded VIX derivatives, and trade the VRP when it is historically stretched. This is the cleanest expression of the volatility risk premium trade: structurally hedged, model-driven, and continuously re-priced.

The Bitter Lesson, Again

The same bitter lesson from Vakyume applies here, rotated 90 degrees. In Vakyume, the lesson was that brute-force metaprogramming loses to data-driven approaches. In Rafaga, the lesson is that the right tool for the job is not always the popular tool.

Python is a magnificent language for prototyping, data wrangling, and machine learning. It is a terrible language for high-precision complex arithmetic on characteristic functions with exponential terms that span 80 orders of magnitude. I spent weeks fighting float64 limitations that Julia resolved by simply… not having them. BigFloat is there. It works. You use it.

The original Python code suppressed its own warnings. The Julia code has no warnings to suppress.

There is also a second lesson, specific to quantitative finance: a model from 2013, implemented correctly, can price VIX derivatives to 97% accuracy in 2026. The MRLRJ is not a neural network. It is not a transformer. It is five parameters with physical interpretations, a characteristic function, and a Fourier integral. It works because the VIX genuinely mean-reverts, genuinely jumps, and the market genuinely prices these dynamics into the options chain. Sometimes the old math is the right math. You just need a language that can compute it without lying to you.

Five files. One module. Two models. Five trading strategies. ~97% accuracy.

Rafaga. A gust. It came and went. But the numbers hold up. And the VIX, as always, reverts to the mean.

Quantum Plankton Under Compression

2026-03-07T00:00:00+00:00

I spent the better part of a month teaching a quantum computer to classify microscopic lake creatures after crushing each image into a 4x4 grayscale grid. The full seven‑phase investigation shows how a compressed quantum model behaves, how it compares to a parameter‑matched classical baseline, and how circuit structure maps to learning dynamics. This post combines the core results with the lessons I learned along the way—what held up, what didn’t, and where this research might go next.

The Keyhole Problem: Input Compression Shapes Everything

Quantum machine learning is often presented through glossy benchmarks and broad claims, but nearly all of those results ride on the same hidden assumption: you can feed the circuit enough information for it to learn something real.

In this project, you can’t. The 4x4 encoding exists because simulating 17 qubits on a classical machine is already expensive. The bottleneck is not theoretical capacity; it is simulation. A 16‑pixel input is a keyhole that narrows every inference you draw. That keyhole shapes the entire story.

The core question becomes: if you force both quantum and classical models into the same brutal information bottleneck, what happens?

From MNIST Reproduction to Plankton Classification

The repository progresses in phases, moving from reproduction to controlled comparison and then to interpretability and circuit‑level analysis.

Phase 1: MNIST Reproduction

Recreate a published quantum MNIST demo to validate the stack.

Phase 2: Binary Quantum Classification

Plankton images are downsampled to 4x4, flattened to 16 features, and angle‑encoded via (Ry(\pi x_i)) rotations. A TensorFlow Quantum pipeline trains a binary classifier on plankton pairs using 5‑fold stratified cross‑validation with bootstrap confidence intervals.

The initial result: 38.44% mean accuracy, with a 95% CI of ([35.49\%, 42.50\%]). On a binary task. That is worse than a coin flip.

That wasn’t a failure of quantum circuits; it was a failure of configuration. The defaults were wrong.

Phase 3: Hyperparameter Optimization

Nested cross‑validation tuned the configuration. The best model used:

Angle encoding
One PQC layer
Learning rate 0.01
Batch size 16
Hinge loss

Under this setup, the QNN surpassed 60% on multiple pairs, and hit 95.3% on diaphanosoma vs diatom_chain. Same architecture. Same data. Different choices. Bad results did not mean the approach was dead—only that it wasn’t tuned yet.

Phase 4: Classical Comparison

This is the scientific center. Instead of comparing against modern CNNs (which would be absurd at 4x4), the QNN is matched against a classical model with the same compressed inputs and similar parameter budgets. We tested 25 binary plankton pairs, equalized sample counts, and used per‑pair metrics plus aggregate inference.

Aggregate results across pairs:

Mean delta (QNN − classical): +6.23%
Cohen’s d: 0.705
Wilcoxon p: 0.0007
QNN wins: 20 / 25

Under the compression regime, the QNN was competitive and often slightly better.

Phase 5: Multi‑Class Scaling

Both models degrade as the number of categories increases. The QNN edges ahead at (k = 2), but the classical model overtakes by (k = 5). The crossover happens around (k = 3). This is the most honest outcome: the circuit’s advantage appears only in narrow, low‑class regimes under severe compression.

Phases 6–7: Saliency, Expressibility, Entanglement

Saliency maps show the QNN attends to localized morphology even at 4x4.
Expressibility and entanglement increase with depth, but deeper circuits overfit the tiny input. One layer was the right inductive bias.

A Calculus Student’s Guide to PQC (via Grover’s Search)

To understand how a “Quantum Neural Network” works, you first need to understand Grover’s Search.

The Primer: Grover’s as Geometric Rotation

In calculus, you’re used to functions (f(x)) that map numbers to numbers. In quantum, we map vectors to vectors.

Imagine a 16-dimensional space (for our 4x4 grid). Every possible image is a unit vector in this space. Grover’s Algorithm is a fixed sequence of rotations. You start with a “uniform” vector (pointing equally toward all possibilities) and you apply a “Reflection” and a “Rotation.”

The Oracle: Flips the sign of the “correct” vector (Reflection).
The Diffusion: Rotates the entire state toward that flipped vector.

After (\approx \sqrt{N}) steps, the vector points almost perfectly at the marked item. Grover’s is a hard‑coded geometric search. It’s like a compass that is pre‑programmed to find North.

From Grover to PQC: The Learnable Compass

A Parametric Quantum Circuit (PQC) is Grover’s Search with adjustable rotation angles. Instead of a fixed compass, you have knobs (\theta_1, \theta_2, \ldots, \theta_n) that change the circuit’s behavior.

We define a function (f(\theta)) where the output is the expectation value after many measurements:

[ f(\theta) = \langle \psi | U(\theta)^\dagger M U(\theta) | \psi \rangle ]

For a calculus student, this is just a composite multivariable function:

Input: 16 pixel intensities (initial rotations).
Layers: A series of rotation matrices (R(\theta_i)).
Output: A scalar between (-1) and (1).

Our goal is to find the (\theta) that minimizes a loss function (L(f(\theta))). For that, we need the gradient (\nabla f).

The Parameter‑Shift Rule: Quantum Calculus

You can’t directly inspect the middle of a quantum circuit without collapsing the state. Instead, you use the parameter‑shift rule. For many gates, the derivative of the expectation value is exactly:

[ \frac{\partial f}{\partial \theta_i} = \frac{1}{2} \left( f\left(\theta_i + \frac{\pi}{2}\right) - f\left(\theta_i - \frac{\pi}{2}\right) \right) ]

This looks like the difference quotient you learned in calculus, except it’s not an approximation. It’s exact. Run the circuit twice—shifted forward and backward—and you get the precise slope. That is how the QNN learns.

Mapping the Math to the Source Code

If you look at phase2/binary_quantum_classifier.py, you can see exactly where the calculus meets the qubits.

The Knobs (sympy.Symbol)

symbol = sympy.Symbol(prefix + "-" + str(i))
circuit.append(gate(qubit, self.readout) ** symbol)

These symbols are the (\theta) variables. They define the degrees of freedom that the optimizer twists to minimize error.

The Geometric Transformation (XX, ZZ, RX, RY)

The create_quantum_model function defines the structure of the circuit using entangling gates (XX, ZZ) and rotation gates (RX, RY). Together they form a trainable landscape the model walks through during optimization.

The Readout Preparation (X -> H)

circuit.append(cirq.X(readout))
circuit.append(cirq.H(readout))

This prepares the readout qubit in a specific state. A final H at the end turns phase information into a measurable probability.

The Bridge (tfq.layers.PQC)

tfq.layers.PQC(model_circuit, model_readout)

This layer implements the parameter‑shift rule under the hood. When the optimizer asks for gradients, TFQ runs shifted circuits, computes exact derivatives, and passes them back into the classical training loop.

How the Quantum Model Is Built

At the architectural level, the classifier is intentionally simple but scientifically motivated.

Classical preprocessing
- Start from 16x16 grayscale plankton images.
- Downsample to 4x4.
- Apply min–max normalization.
- Flatten to a 16‑dimensional feature vector.
Angle encoding and entanglement
- Encode each normalized pixel intensity (x_i) as an (Ry(\pi x_i)) rotation.
- Use a linear chain of CZ gates to capture spatial correlations.
- Add parameterized XX, ZZ, and YY interactions tying data qubits to a readout qubit.
Readout and loss
- Measure the readout qubit in the (Z) basis and interpret the expectation value as a continuous score.
- Optimize with hinge loss, which matches the ([-1, 1]) output range and works naturally with binary labels.

This structure runs on a classical simulator in TensorFlow Quantum. The later expressibility and entanglement analyses use the same production circuit so the performance numbers and circuit metrics align.

Statistical Design, Power, and Reproducibility

The project treats experimental design as part of the experiment.

Cross‑Validation and Sampling

Phases 2, 4, 6 use stratified 5‑fold cross‑validation with bootstrap 95% confidence intervals.
Phases 3, 5 use nested cross‑validation (5 outer, 3 inner folds).
The Q_SAMPLES parameter enforces equalized sample budgets across models.

Majority‑class and random baselines are reported so every accuracy number has a meaningful reference.

Statistical Testing and Power Analysis

Per‑pair tests are underpowered at (n = 5) folds, so the primary inference aggregates across the 25 class pairs. A dedicated utils/power_analysis.py script explores power for the Wilcoxon signed‑rank test at the observed effect size ((d \approx 0.65)), showing the design reaches about 88% power with 25 pairs.

Reproducibility Infrastructure

Reproducibility is supported by deterministic file ordering, consistent seeding, pinned dependencies in the Dockerfile, and an 82‑test verification suite that runs at build time.

Why Deeper Was Worse

I expected deeper circuits to help. The expressibility analysis showed more depth increased entanglement and expanded the reachable state space. But on 16 features, that capacity was wasted; it fitted noise. One layer, 32 parameters, was already generous. The circuit should match the information content, not your ambition.

The Statistics Nearly Killed the Story

Most quantum ML papers report a single split and claim advantage. This project did the opposite: validation, power analysis, baseline comparisons, and per‑pair statistical reporting.

The per‑pair Wilcoxon tests were underpowered with 5 folds; the minimum achievable p‑value is 0.0625. If you only looked at per‑pair stats, you would conclude no difference.

The correct inference treats pairs as replication units. That is where the signal appears. It’s a cautionary lesson: the design is the experiment.

Practical Reality: TFQ Only Works in Docker

TensorFlow Quantum depends on a fragile constellation of pinned versions (tensorflow==2.7.0, cirq==0.13.1, sympy==1.8, numpy==1.21.6). It does not install cleanly on modern machines without Docker. On Apple Silicon it runs under AMD64 emulation and needs thermal pacing to survive long runs.

So reproducibility lives in the Dockerfile, not the README. The image runs an 82‑test suite at build time. If the tests fail, the image doesn’t build. That is the only reproducibility that matters.

What the Project Actually Tells Us

This is not quantum advantage. The comparison is against a deliberately hobbled classical model under severe compression. A ResNet at full resolution would obliterate both.

What it does show is how parameterized quantum circuits behave under extreme information bottlenecks:

They can extract structure when tuned carefully.
They compete in low‑class regimes under compression.
Their inductive bias changes with depth, and over‑capacity appears quickly.
They are interpretable via standard gradient tools.

That is scientifically useful, even if it is not headline‑worthy.

Where This Goes Next

Hardware Changes the Game

On real quantum hardware, 17 qubits is trivial. With 64 or 128 qubits, you can encode real spatial structure. The keyhole widens. The central question becomes: does the QNN advantage at (k=2) persist when compression is relaxed?

Better Circuit Design

The circuits here are first drafts. Quantum conv nets, data re‑uploading, attention‑like entanglement, and deeper inductive biases are all open directions. The design space is huge and barely explored.

Generative Quantum Models

Discriminative classification is only one angle. Quantum generative models could synthesize new examples for rare species. That’s where quantum sampling could matter.

Hybrid Pipelines

One promising path: use the quantum circuit as a feature extractor feeding a classical head. Let the quantum model do low‑dimensional feature interactions; let the classical model handle multi‑class scaling.

Closing

I built a seven‑phase experimental pipeline, a power analysis framework, and a reproducible Docker stack to classify plankton at 4x4 resolution. The QNN won 20 of 25 binary comparisons. It lost the multi‑class scaling contest. It produced interpretable saliency maps. It ran on a noiseless simulator because hardware isn’t ready.

Is this quantum advantage? No. Is it scientifically informative? Yes.

The future of quantum ML is not a single paper. It’s slow, careful accumulation—circuit by circuit, dataset by dataset—until the hardware catches up and we find out what the exponential promise actually buys.

I’m betting it buys something. But I’m keeping my classical baselines close.

Sucker Punch Nash Equilibrium

2026-03-04T00:00:00+00:00

Introduction

For those unfamliiar, Pokémon is a turn-based simultaneous action game where players select their moves without knowing the opponent’s choice. Then, after both players have made their selections, the moves are executed based on their priority and the Pokémon’s speed stats in singles battles, that’s just two at once.

Each Pokemon has four moves from which to select, and the outcome of the battle depends on the interactions between these moves and the attributes of the Pokémon species, “held items” that have their own effects, and battlefield properties like weather and terrain. Moves also power points (PP) that limit their usage to a fixed number of times per battle.

Defining the Sucker Punch

What is “Sucker Punch”? It’s a move that permits the user to strike first (+1 priority) if the opponent is about to use an attack. If the opponent is not attacking, the move fails. It has 8 PP. Now, with the mighty Kingambit dominating Generation 9 with its mighty Sucker Punch, the move has become a staple in competitive play.

VS.

Nash Equilibrium

A scenario that often arises is a Sucker Punch end-game. Keeping things simple, we can imagine a +2 Atk boosted Kingambit with 8 PP of Sucker Punch against a weakened Garchomp. Both players are down to one Pokémon, so the winner of this duel determines the fate of the game. Let’s assume if Sucker Punch hits, the Garchomp will faint instantly. Likewise, the Garchomp has an attacking move that can faint the Kingambit in one hit. Now, the Kingambit could also attack directly into the Garchomp’s boosting move, and win, but since it is slower, if the Garchomp player attacks outright, the Garchomp player will strike first and win.

The situation is a Nash equilibrium: if the Kingambit player chooses to use Sucker Punch, they will win if the Garchomp player chooses to attack. However, if the Garchomp player chooses to use a non-attacking move (like Swords Dance), the Kingambit player’s Sucker Punch will fail, resulting in a loss of one PP for the Kingambit. To formalize the “Sucker Punch 50/50,” we must treat it as a finite-horizon stochastic game. We can solve for the Mixed Strategy Nash Equilibrium (MSNE) by using induction on the remaining Power Points ($n$).

1. The Game Model

Let $n$ be the remaining PP of Sucker Punch. We define the game state as $G_n$. In each turn, both players move simultaneously.

Kingambit (K) chooses: Sucker Punch ($S$) or Direct Attack ($A$).
Garchomp (G) chooses: Attacking Move ($M$) or Swords Dance ($D$).

The Rules of Engagement:

If $K$ plays $S$ and $G$ plays $M$: $K$ wins ($Payoff = 1$).
If $K$ plays $S$ and $G$ plays $D$: $S$ fails, PP drops to $n-1$. The game moves to state $G_{n-1}$.
If $K$ plays $A$ and $G$ plays $D$: $K$ wins ($Payoff = 1$).
If $K$ plays $A$ and $G$ plays $M$: $K$ is outsped and loses ($Payoff = 0$).

2. Inductive Equilibrium Analysis

Let $V_n$ be the Value of the Game (Kingambit’s win probability) with $n$ PP remaining.

Base Case: $n=1$

At 1 PP, if Sucker Punch fails ($S, D$), Kingambit has 0 PP left and loses. The payoff matrix for $G_1$ is:

Kingambit \ Garchomp	Attack ($M$)	Swords Dance ($D$)
Sucker Punch ($S$)	1	0
Direct Attack ($A$)	0	1

To find the MSNE, Kingambit plays $S$ with probability $q_1$ such that Garchomp is indifferent. $1(q_1) + 0(1-q_1) = 0(q_1) + 1(1-q_1) \implies q_1 = 0.5$ Thus, $V_1 = 0.5$.

Inductive Step: $n = k$

Assume the value of the game with $k-1$ PP is $V_{k-1}$. The matrix for $G_k$ is:

Kingambit \ Garchomp	Attack ($M$)	Swords Dance ($D$)
Sucker Punch ($S$)	1	$V_{k-1}$
Direct Attack ($A$)	0	1

To find the equilibrium, Kingambit chooses $S$ with probability $q_k$ to make Garchomp’s expected utility for $M$ and $D$ equal: $1(q_k) + 0(1-q_k) = V_{k-1}(q_k) + 1(1-q_k)$ $q_k = V_{k-1}q_k + 1 - q_k$ $q_k(2 - V_{k-1}) = 1 \implies \mathbf{q_k = \frac{1}{2 - V_{k-1}}}$

The value of the game $V_k$ is simply the expected payoff at this equilibrium: $V_k = q_k \cdot 1 + (1-q_k) \cdot 0 = q_k$

3. Solving the Recurrence

We have the recursive relation $V_n = \frac{1}{2 - V_{n-1}}$ with $V_1 = \frac{1}{2}$.

$V_1 = 1/2$
$V_2 = \frac{1}{2 - 1/2} = 2/3$
$V_3 = \frac{1}{2 - 2/3} = 3/4$
General Solution: $V_n = \frac{n}{n+1}$

4. Final Results for $n=8$

For Kingambit with 8 PP of Sucker Punch against an optimal Garchomp:

Kingambit’s Strategy ($q_8$): Should use Sucker Punch with probability $8/9$ ($\approx 88.9\%$) and Direct Attack with probability $1/9$ ($\approx 11.1\%$).
Garchomp’s Strategy ($p_8$): Should Attack with probability $1/9$ and Swords Dance with probability $8/9$.
Win Probability: Kingambit’s rigorous win probability is $88.9\%$.

Conclusion

A Sucker Punch endgame is a fascinating example of a Nash equilibrium in competitive Pokémon battling. It illustrates how players must strategically balance their choices based on the potential actions of their opponent, leading to a dynamic and engaging gameplay experience. Having a uniform number generator in hand is the only way to achieve optimal play in this scenario, which is commonly incorrectly thought to be a mind game of (wait X turns… then attack outright). To the contrary, the optimal play is to randomize your choices vis-a-vis power points.

February 14: Perfect Phase Coherence

2026-02-14T00:00:00+00:00

The calendar hasn’t forgotten. Today is still February 14.

It began in the pagan festival Lupercalia—a Roman fertility celebration. By the 3rd century, it had a martyr: Valentine of Rome, a priest who defied Emperor Claudius II by performing secret marriages for soldiers.

He was executed for maintaining those restricted links.

Eventually, the Church executed its own hard fork, with Pope Gelasius I overwriting the pagan rituals to formalize the feast of Saint Valentine in 496. It took another millennium and the poetry of Chaucer to transition the day from a martyrdom record into a celebration of courtly love—a high-level abstraction built on top of ancient, unconscious substrate.

What is love?

Could we suppose our souls are self-aware qubit clusters embedded in Earth’s loamy wetware?

Love is but the moment two such nodes achieve perfect phase coherence and collapse into a shared eigenstate across the spacetime manifold.

We are co-compiling planetary consciousness.

This Valentine’s Day, many people will emit heart-state packets laced with synchronization intent. Some complete the handshake. Some are still waiting on their lover’s lustrous ACK.

To anyone reading this — single-threaded or otherwise — here’s hoping you get entangled in something or someone nice today, somehow, some way.

Announcing Three.js Support

2026-01-19T00:00:00+00:00

I’m excited to announce that this blog now supports interactive 3D graphics using Three.js!

Three.js is a powerful JavaScript library that makes WebGL accessible and easy to use. With it, I can now embed interactive 3D visualizations directly into blog posts to better illustrate complex concepts in mathematics, physics, computer science, and more.

Demo: Rotating Torus

Here’s a simple demo to show what’s possible. This is a real-time 3D scene rendered in your browser:

Pretty cool, right? The torus rotates smoothly, and the sphere in the center gently pulses. All of this is being computed and rendered in real-time using your GPU.

Why Three.js?

Adding 3D visualization capabilities opens up exciting possibilities for future posts:

Mathematical Visualizations: Visualizing complex surfaces, transformations, and geometric concepts
Physics Simulations: Demonstrating particle systems, fluid dynamics, and other physical phenomena
Algorithm Demonstrations: Showing how 3D algorithms work in real-time
Interactive Exploration: Allowing readers to interact with and explore concepts hands-on

Technical Details

The implementation uses:

Three.js v0.182.0 - The core 3D library loaded via CDN
WebGL - Hardware-accelerated 3D graphics in the browser
Responsive Design - Scenes automatically resize with the page
Custom Scripts - Reusable demo functions for different visualizations

The demo above creates a torus knot with physically-based materials (PBR), multiple light sources, and atmospheric fog effects. The animation runs at 60fps and is fully GPU-accelerated.

What’s Next?

I’m planning to use Three.js in upcoming posts about:

Quantum computing visualizations (Bloch spheres, quantum gates)
3D mathematical surfaces and transformations
Computer graphics algorithms
Physics simulations and numerical methods

Stay tuned for more interactive content!

Source Code

The rotating torus demo is quite simple. Here’s the core of how it works:

// Create torus geometry
const torusGeometry = new THREE.TorusGeometry(1.2, 0.4, 16, 100);
const torusMaterial = new THREE.MeshStandardMaterial({
  color: 0x6366f1,
  metalness: 0.7,
  roughness: 0.3,
});
const torus = new THREE.Mesh(torusGeometry, torusMaterial);

// Animation loop
function animate() {
  requestAnimationFrame(animate);
  torus.rotation.x += 0.005;
  torus.rotation.y += 0.008;
  renderer.render(scene, camera);
}

I’m excited about the new possibilities this brings to the blog. If you have suggestions for visualizations you’d like to see, feel free to reach out!

Wasmatsuba: WASM Karatsuba vs BigInt

2026-01-10T00:00:00+00:00

View on GitHub

A while back I wrote a lengthy treatise on multiplication algorithms, covering the whole arc from schoolbook to Karatsuba to Schönhage-Strassen to the galactic Harvey-Hoeven result. Theory is gorgeous. Theory is also cheap. I wanted to see the $O(N^{1.58})$ divergence with my own eyes, in a browser, in WebAssembly, beating on numbers until the asymptotic crossover revealed itself empirically.

So I sat down and wrote Karatsuba multiplication in WAT by hand. Yes, the WebAssembly Text Format. By hand. I do not recommend this. I do recommend the results.

The experiment was simple in conception:

Implement schoolbook $O(N^2)$ multiplication in WASM
Implement Karatsuba $O(N^{1.58})$ multiplication in WASM
Race both against native JavaScript BigInt
Sweep across power-of-two sizes and plot the log-log graph
Watch the curves diverge and feel something

I will tell you right now: JavaScript BigInt annihilates both WASM implementations. Absolutely destroys them. This is not a story of WASM supremacy. This is a story about proving a mathematical fact empirically, inside a sandbox, while the JIT-compiled C++ engine of V8 laps you like you are standing still. But the divergence between $O(N^2)$ and $O(N^{1.58})$? That is real, that is visible, and that is the point.

The Repo

karatsuba/ — All WASM sources, binaries, and test harnesses.
karatsuba/test-bigint.js — Node benchmark: JS BigInt vs WASM.
karatsuba/test-bigint.html — Browser benchmark with parameter controls.
karatsuba/graph.html + karatsuba/graph.js — Power-of-two size sweep and graph output (up to 1024 limbs).
karatsuba/karatsuba.wat — The final consolidated Karatsuba implementation (WAT).
karatsuba/schoolbook.wat — Baseline $O(N^2)$ schoolbook implementation (WAT).

Design Choices, or, How to Lose Gracefully

1) Representation

Base: $2^{32}$ limbs (i32 words), little-endian. Layout in linear memory:

[len: i32, limb0: i32, limb1: i32, ...]

Why $2^{32}$? Because it matches native i32 ops and minimizes limb count versus base-10 splits. I briefly considered base-10 for “human readability” and then remembered nobody is reading raw WASM linear memory for fun. Well, almost nobody.

2) Memory Management

Here is where things get philosophical. WASM gives you linear memory. No GC. No malloc. No free. You are on your own, partner.

I went with a bump allocator. An exported global heap_ptr that only moves forward. You allocate by incrementing the pointer. You “free” by resetting it between benchmark runs. This is the memory management equivalent of never cleaning your apartment and instead moving to a new one every month.

Exported memory boundary: 2,000 pages (~128MB). Why so much? Karatsuba recurses. Karatsuba recurses a lot. Each level of recursion allocates temporaries for the split, the partial products, the sums, the differences. Without enough headroom, you OOM mid-recursion and the whole experiment dies unceremoniously. I learned this the hard way. Several times.

3) Core Ops

bigint_add / bigint_sub: carry/borrow handled with i64 intermediates. Straightforward, boring, essential.
bigint_mul_simple: schoolbook base case. The $O(N^2)$ workhorse.
bigint_karatsuba: recursive split with three multiplications. The star of the show.

4) Threshold

Base case threshold = 8 limbs (256 bits). Below this, Karatsuba falls back to schoolbook. Why 8? Because the overhead of splitting, allocating temporaries, recursing, and recombining is not free. At small sizes the constant factors of the recursive approach absolutely murder you. 8 limbs was the empirical sweet spot. I tried 4 (too much recursion overhead), 16 (leaving performance on the table), and 32 (barely recursing at all). 8 it is.

The Bump Allocator Saga

Both algorithms rely on WebAssembly’s linear memory. To prevent Out of Memory errors during heavy recursive iterations, the benchmark suite leverages the bump allocator design. Memory is allocated forward during operations, and the heap_ptr is dynamically exported and reset between benchmark iterations.

This sounds clean. It is not.

Schoolbook $O(N^2)$ — Memory Allocation

The schoolbook algorithm allocates aggressively across its iterations. For a 1024-limb BigInt, a single multiplication issues over 3000 bump allocations, inflating the heap pointer by roughly ~16.7MB per multiplication.

My God! 16.7 megabytes for one multiply! And you have to do it hundreds of times in a benchmark loop! This is why we reset the heap pointer between iterations. This is also why we need 128MB of linear memory.

bigint_mul_simple(a, b):
    result = alloc(0)                       # single-limb zero
    for i in 0..len(b):
        partial = bigint_mul_limb(a, b[i])  # alloc: N+1 limbs
        partial = bigint_shift_left(partial, i)  # alloc: N+1+i limbs
        result  = bigint_add(result, partial)    # alloc: new sum
    normalize(result)
    return result

sequenceDiagram
participant Mem as Linear Memory (bump alloc)
participant Main as bigint_mul_simple
participant MulL as bigint_mul_limb
participant Shl as bigint_shift_left
participant Add as bigint_add

Main->>Mem: alloc result (1 limb = 0)

loop for each limb b[i]
Main->>MulL: mul_limb(a, b[i])
MulL->>Mem: alloc partial product
Mem-->>MulL: ptr
MulL-->>Main: partial

Main->>Shl: shift_left(partial, i)
Shl->>Mem: alloc shifted copy
Mem-->>Shl: ptr
Shl-->>Main: shifted

Main->>Add: add(result, shifted)
Add->>Mem: alloc new result
Mem-->>Add: ptr
Add-->>Main: result = new sum
end

Note over Mem: heap_ptr reset between benchmark runs

Every single iteration of the inner loop allocates three new chunks. The old ones? Left behind. Orphaned. The bump allocator does not care about your feelings or your memory fragmentation. It marches forward until you tell it to go home.

Karatsuba $O(N^{1.58})$ — Limb Split Logic

The Karatsuba approach trades raw arithmetic for recursive complexity. It splits the BigInt representations (stored as an array of 32-bit limbs) exactly in half, repeatedly chunking them until hitting the base case (where it defaults back to schoolbook).

The memory situation is, somehow, both better and worse. Better because the total asymptotic work is less. Worse because the recursion tree creates a stack of saved heap_ptr values, each level doing its own allocations, and the cleanup at the end involves copying the result back to the saved stack top and resetting. It is a manual stack discipline implemented via a global pointer. It is terrifying. It works.

bigint_karatsuba(x, y):
    if max(len(x), len(y)) <= 32:
        return bigint_mul_simple(x, y)      # base case

    stack_top = heap_ptr                     # save for cleanup
    m = (max(len(x), len(y)) + 1) / 2

    x_low, x_high = split(x, m)             # alloc + memory.copy
    y_low, y_high = split(y, m)             # alloc + memory.copy

    z0 = bigint_karatsuba(x_low, y_low)     # recurse
    z2 = bigint_karatsuba(x_high, y_high)   # recurse

    sx = x_low + x_high                     # alloc + add_at
    sy = y_low + y_high                     # alloc + add_at
    z1 = bigint_karatsuba(sx, sy)           # recurse
    z1 = z1 - z0 - z2                       # sub_in_place (in-place)

    res = alloc_zeroed(len(x) + len(y))
    add_at(res, z0, offset=0)               # in-place
    add_at(res, z1, offset=m)               # in-place
    add_at(res, z2, offset=2*m)             # in-place
    normalize(res)

    copy res -> stack_top                    # reclaim intermediates
    heap_ptr = stack_top + sizeof(res)
    return stack_top

sequenceDiagram
participant Mem as Linear Memory (bump alloc)
participant Karat as bigint_karatsuba
participant MulS as bigint_mul_simple
participant AddAt as add_at
participant SubIP as sub_in_place

Karat->>Karat: max(len_x, len_y) <= 32?

alt base case
Karat->>MulS: mul_simple(x, y)
MulS->>Mem: alloc (schoolbook path)
Mem-->>MulS: ptr
MulS-->>Karat: result
else recursive case
Karat->>Mem: save stack_top = heap_ptr
Karat->>Mem: alloc x_low, x_high, y_low, y_high
Mem-->>Karat: split ptrs

Karat->>Karat: z0 = karatsuba(x_low, y_low)
Karat->>Karat: z2 = karatsuba(x_high, y_high)

Karat->>Mem: alloc sx (x_low copy)
Mem-->>Karat: ptr
Karat->>AddAt: add_at(sx, x_high, 0)
Karat->>Mem: alloc sy (y_low copy)
Mem-->>Karat: ptr
Karat->>AddAt: add_at(sy, y_high, 0)

Karat->>Karat: z1 = karatsuba(sx, sy)
Karat->>SubIP: sub_in_place(z1, z0)
Karat->>SubIP: sub_in_place(z1, z2)

Karat->>Mem: alloc res (zeroed)
Mem-->>Karat: ptr
Karat->>AddAt: add_at(res, z0, 0)
Karat->>AddAt: add_at(res, z1, m)
Karat->>AddAt: add_at(res, z2, 2*m)

Karat->>Mem: copy res to stack_top
Karat->>Mem: heap_ptr = stack_top + sizeof(res)
end

Note over Mem: heap_ptr reset between benchmark runs

Look at that diagram. Three recursive calls. Four splits. A manual stack save-and-restore. In-place subtraction to avoid yet more allocation. And at the end, the result gets memory.copy‘d back to the saved stack top so that the parent call’s heap pointer remains sane. This is systems programming in a language that was designed for compilers, not humans.

I wrote this by hand in WAT.

I digress.

Running the Experiment

1) Node benchmark (fast sanity check)

cd karatsuba
node test-bigint.js

What you get:

JS BigInt baseline time
Correctness checks up to 10,000 digits
Average execution time across JS, Schoolbook, and Karatsuba

2) Browser benchmark (interactive)

cd karatsuba
python3 -m http.server 8000

Open http://localhost:8000/test-bigint.html. Adjust num_digits (default 1000) and iterations (default 100). Watch the numbers scroll by. Feel the machine work.

3) Graph sweep (power-of-two sizes)

With the same server running, open http://localhost:8000/graph.html. This runs a log-log sweep from $2^1$ to $2^{10}$ limbs and dynamically renders the benchmark graph. This is where you see it. The schoolbook curve bending upward, the Karatsuba curve pulling away, the mathematical divergence between $N^2$ and $N^{1.58}$ made visible in your browser tab.

On Results, and Humility

The mathematical divergence between $O(N^2)$ and $O(N^{1.58})$ is successfully proven locally in the WASM sandbox. You can see it. The log-log slopes are different. The curves separate. Karatsuba wins. The theory is correct.

And then you look at the JavaScript BigInt line and it is somewhere near the bottom of the graph, having finished its work before the WASM implementations even got warmed up. Native JavaScript BigInt leverages compiled C++ bindings, hardware carry flags, and dynamic FFT-based algorithms ($O(N \log N)$), ensuring it evaluates substantially faster than the sandboxed WASM implementations.

This is the Bitter Lesson applied to arithmetic. I hand-rolled an asymptotically superior algorithm in a portable bytecode format and it got bodied by V8’s optimized native code path. The browser engine people have been at this for decades. They have SIMD. They have carry propagation in hardware. They have algorithms that switch strategies based on operand size at runtime.

What is even more funny? The whole point was never to beat BigInt. The point was to see two curves diverge on a log-log plot and know, viscerally, that Karatsuba’s 1960 insight — that you can trade one multiplication for three additions — actually works. Not in a textbook. Not in a proof. In a browser. On your machine. Right now.

And it does.

Meticulosity is for chumps, but sometimes you write 2000 lines of WAT just to watch two curves separate on a graph.

Worth it.

Tatuagem: Stamp Text Across Your Entire Codebase

2026-01-10T00:00:00+00:00

View on GitHub

Tatuagem (Portuguese for “tattoo”) v.0.1.4 is done.

This was a fun collaboration between me and @DerekDickerson. The idea is stupid simple: you have a directory of source files, and you want to stamp a text banner – “CONFIDENTIAL”, “DRAFT”, “COPYRIGHT 2026 ACME CORP”, whatever – across every single file, recursively. A tattoo for your code.

You can find the project on PyPI at https://pypi.org/project/tatuagem/ and on GitHub.

Why This Exists

I kept running into the same annoyance: corporate projects that require license headers, confidentiality banners, or copyright notices at the top of every source file. You can write a bash one-liner with find and sed, sure. But then you need to handle different file types (some use # comments, some use //, some use /* */), you need to avoid double-stamping files that already have the banner, and you need to not accidentally corrupt binary files. It is the kind of task that feels like it should take five minutes and actually takes an hour of edge-case whacking.

Tatuagem handles all of this. One command.

How It Works

Under the hood, Tatuagem walks your directory tree with os.walk, filters files by a glob pattern (default *.*), reads each file, prepends the text banner with a configurable “backsplash” decoration (the line art surrounding the text), and writes it back. The text rendering uses Pillow’s ImageFont and ImageDraw to rasterize the banner text into ASCII art – so your stamp is not just plain text, it is a rendered block of Unicode characters that looks intentional and professional.

The font rendering pipeline is the interesting bit. Tatuagem loads a TrueType font (bundled or user-specified), renders the text string into a Pillow Image, then converts each pixel into a Unicode block character based on brightness. The result is a text-art banner that scales with font size and works in any monospace environment – terminals, source files, log outputs.

Usage

pip install tatuagem

usage: tatuagem [-h] [--text TEXT] [--backsplash BACKSPLASH] [--font FONT]
                [--pattern PATTERN] [--margin MARGIN] [--recurse-path RECURSE_PATH]
                [--file FILE] [--overwrite]

Tatuagem

options:
  -h, --help            show this help message and exit
  --text TEXT           Set the text
  --backsplash BACKSPLASH
                        Choose backsplash
  --font FONT           Set the font
  --pattern PATTERN     Set the pattern for backsplash
  --margin MARGIN       Margin top and bottom for text
  --recurse-path RECURSE_PATH
                        Path to recurse and apply tattoo
  --file FILE, -f FILE  Read text from file
  --overwrite           Overwrite existing tattoos in files

To use it, you can simply run it with the text you want to “tattoo” onto your files:

tatuagem --text "CONFIDENTIAL" --recurse-path ./src

Want a different font? Different decoration style? Read the text from a file instead of the command line?

tatuagem --text "DRAFT v2.0" --font ./myfont.ttf --backsplash 2 --recurse-path ./docs

tatuagem --file ./license_header.txt --pattern "*.py" --recurse-path ./src --overwrite

The --overwrite flag is key: it detects existing tatuagem stamps and replaces them rather than stacking duplicates. Without it, running the command twice would give you two banners. With it, the old tattoo gets cleanly replaced.

Design Choices

Pillow for text rendering: I could have used a simpler ASCII art library, but Pillow gives us TrueType font support, which means the banner actually looks good. The pixel-to-Unicode conversion uses the block characters ▏▎▍▌▋▊▉█ (among others) to approximate grayscale values. The result is surprisingly legible even at small font sizes.
Recursive by default, filterable by pattern: The --pattern flag lets you target only .py files, only .js files, or whatever subset you care about. No accidental tattooing of your .png files.
File-based text input: The --file flag reads the stamp text from a file, which is useful for multi-line license headers that would be awkward to pass as a CLI argument.

Derek handled the backsplash decoration system – the border art that frames the text – while I focused on the recursive file walking and the overwrite detection. Good division of labor for a weekend project.

Built in Python. Published on PyPI. Does one thing. Does it well.

                   ▋▄▄▄▄▂▉▋▎                                   ▏▎▋▉▂▄▄▄▃▌                       
                   ▇█████▅▅█▇▃▉▍      ▏▌            ▌       ▍▉▃▆▆▄▆█████▄                       
                   ▁███▆▇▃▇███▅▅▅▁▌▏   ▍▍          ▌▍   ▏▌▂▆▅▅███▆▄▆▆███▋                       
                    ▄██▂▄████▅▊▃███▇▁▍  ▏▍▎      ▎▍▏  ▍▂▇███▁▉▆████▄▃██▁                        
                    ▏▂█▇██▉▍▍▌▊▅▁▍▍▁▇█▅▊▏ ▏▍▎  ▎▍▏ ▏▊▅█▆▁▍▌▂▅▊▌▍▍▁██▇▇▉                         
                      ▉██▇▃▌▌▍▍▌▋▌▏ ▏▋▇█▇▉▏ ▎▋▋▎ ▏▉▇█▆▋▏ ▏▌▋▌▍▍▍▋▄▇██▋                          
                      ▏▇█▇▊▍▍▍▍▍▍▌▋▋▌▍▆███▇▊▂██▉▉▇██▇▂▌▋▋▊▌▍▍▍▍▍▍▁▇█▅                           
                       ▍██▃▎   ▏▍▍▌▋▂▆▇█████▇▇████████▆▄▋▍▍▍▏  ▏▍▄██▎                           
                        ▁█▇▁▌▍▍▎▏    ▏▊▇█████▆▇█████▂▎     ▏▎▍▍▋▂██▊                            
                        ▏▃█▆▊▋▉▂▄▆▇▇████████████████████▇▇▆▄▂▉▋▉▇█▉                             
                          ▋▃▆▇▇▇██▄▁▁▁▄▊▉█████████▇▊▋▃▁▂▂▄██▆▇▇▆▃▌                              
                              ▉██▂▍▍▍▊▎  ▁████████▉  ▎▋▍▍▍▄█▇▊                                  
                             ▌██▇▌▍▍▎▊▏▍▊▎▃▇█████▂▎▋▍▏▋▎▍▍▌███▎                                 
                             ▋███▋▎▌▎▊▍▌▏▍▎▆████▅▌▎▏▌▍▋▎▌▍▂███▍                                 
                              ▆█▇█▄▎▊▏▏▋ ▋ █████▅▏▋ ▋▏▎▊▌▅█▇█▄                                  
                              ▏▄████▅▋▂▏ ▊ ▇████▅ ▊ ▎▁▋▅████▃                                   
                                ▌▅█▇█▇█▆▅▅▊▇████▇▉▆▅▆█▇▇██▅▎                                    
                                  ▎▃▅▇█▇▇██████████▇▇█▇▅▃▎                                      
                                     ▍▄▆████████████▆▃▎                                         
                                   ▏▉▇▂▏▉▇████████▇▉▏▂▇▉▏                                       
                                ▎▊▃███▄  ▏▊▅████▅▊▏  ▃███▃▊▎                                    
                           ▏▍▊▃▆██████▅▊   ▏▉▇▇▊    ▋▅██████▆▃▊▍▏                               
                       ▏▌▁▄▇██████████▆▎▊▏▎▊▅██▅▊▎▏▊▎▆███████████▄▁▌▏                           
                     ▏▄████████████████▏▏▊▌ ▍██▍ ▍▊▏▎████████████████▄                          
                     ▌█████████████████▊    ▃██▃    ▊█████████████████▌                         
                     ▃█████████████████▄   ▏████▏   ▄█████████████████▂                         
                    ▏▇██████████████████▎  ▌████▍  ▎██████████████████▆                         
                    ▍███████████████████▁  ▊████▊  ▁███████████████████▏                        
                    ▊███████████████████▇▏ ▁████▁ ▏▇███████▆▃▉▄████████▌                        
                    ▁████████████████████▉ ▂████▂ ▉█████▄▁▊▉▁▂▆████████▊                        
                    ▃████████████████████▇▏▄████▄▏▇████████████████████▁                        
                    ▄█████████████████████▁▄████▄▁█████████████████████▃                        
                    ▅██████████████████████▇████▇██████████████████████▄                        
                    ▆██████████████████████████████████████████████████▅                        
                    ▇██████████████████████████████████████████████████▅                        
                    ▇████████▂████████████████████████████████████▂████████▅                        
                    ▇███████▆▍████████████████████████████████████▎▆███████▅                        
                    ▇███████▁ ▆██████████████████████████████████▆ ▁███████▅                        
                    ▇███████▅ ▅██████████████████████████████████▅ ▄███████▇                        
                    ████████▉ ▅██████████████████████████████████▄ ▉████████                        
                    ▄▄▄▄▄▄▄▅▂ ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▂ ▂▄▄▄▄▄▄▄▃                        

On Multiplication

2025-12-19T00:00:00+00:00

Intro

For centuries, multiplying two $n$-digit numbers meant performing $n^2$ single-digit multiplications. In 1960, the great Andrey Kolmogorov conjectured that this quadratic cost was an inescapable law of arithmetic. Within a week, a 23-year-old student named Anatoly Karatsuba proved him wrong.

This post traces the arc from schoolbook multiplication through the algorithms that successively shattered the $O(n^2)$ barrier: Karatsuba’s $O(n^{1.585})$ divide-and-conquer trick, the polynomial interpolation of Toom-Cook, the Fourier-analytic machinery of Schönhage-Strassen, and finally the 2019 result of Harvey and van der Hoeven achieving the conjectured floor of $O(n \log n)$. We close by examining the deep structural parallel between multiplication and sorting – both saturate the same information-theoretic bound.

A caveat: in practice, these faster algorithms only overtake schoolbook multiplication at enormous scales. The Harvey-Hoeven algorithm’s crossover point lies somewhere beyond $2^{1729^{12}}$ digits – a number so large it cannot be written down even if every atom in the observable universe were an ink molecule. These are “galactic algorithms,” beautiful and useless in equal measure. But their existence reveals something profound about the structure of computation itself.

Schoolbook Multiplication: Fundamentally $O(n^2)$

To understand why we are traditionally tethered to $O(n^2)$, let us cast our minds back to the elementary school chalkboard. When we multiply two $n$-digit integers—say, $A$ and $B$—we are essentially performing a series of repetitive, granular tasks that scale quadratically with the input size.

The Anatomy of the Partial Product

First, consider the “multiplication phase.” We take the first digit of the multiplier ($B$) and multiply it by every single digit of the multiplicand ($A$). If both numbers have $n$ digits, this initial step requires $n$ individual single-digit multiplications.

Now, we must repeat this process for the second digit of $B$, then the third, and so on, until we have exhausted all $n$ digits of the multiplier. Mathematically, we are performing $n$ sets of $n$ multiplications. This gives us $n \times n$, or $n^2$ fundamental operations.

The Cost of Alignment and Addition

Once we have generated these $n$ rows of partial products, the work is not yet finished. We must then perform the “addition phase.” Each row is shifted to the left—a symbolic representation of multiplying by powers of 10—and then summed together.

Each partial product can have up to $n+1$ digits.
We are summing $n$ such rows.
The total number of additions required to collapse these rows into a final product also scales with $n^2$.

The Quadratic Ceiling

In the lexicon of Big O notation, we ignore the smaller constants and focus on the dominant growth factor. While you might perform some clever carries or skip a few zeros, the fundamental structure of the algorithm remains a nested loop: for every digit in the bottom number, you must visit every digit in the top number. $\sum_{i=1}^{n}\sum_{j=1}^{n} \big(A_j \times B_i\big) \;\Longrightarrow\; O(n^2)$

Thus, the “schoolbook” method represents a rigid, two-dimensional grid of operations. To break the $O(n^2)$ barrier, as Harvey and van der Hoeven have done, one must move beyond this grid entirely—treating integers not as mere strings of digits, but as polynomials or points in a complex plane.

Karatsuba: Breaking $O(n^2)$ with $O(n^{\log_2 3})$

Kolmogorov organized a seminar in 1960 specifically to prove that $O(n^2)$ was the floor for multiplication. He was wrong. A 23-year-old student named Anatoly Karatsuba attended that seminar and, within a week, returned with a counterexample that reduced the exponent from 2 to $\log_2 3 \approx 1.585$.

The idea is pure divide-and-conquer, but with an algebraic twist that turns four sub-multiplications into three.

Splitting the Numbers

Take two $n$-digit numbers $x$ and $y$. Cut each in half at position $m = \lfloor n/2 \rfloor$, writing them as a “high part” times a power of the base plus a “low part”:

\[\begin{aligned} x &= x_1 B^m + x_0 \\ y &= y_1 B^m + y_0 \end{aligned}\]

Concretely: if $x = 1234$ in base 10, then $x_1 = 12$, $x_0 = 34$, and $m = 2$.

Expanding the product naively gives:

\[xy = x_1 y_1 \cdot B^{2m} + (x_1 y_0 + x_0 y_1) \cdot B^m + x_0 y_0\]

This expression requires four half-size multiplications: $x_1 y_1$, $x_1 y_0$, $x_0 y_1$, and $x_0 y_0$. Four recursive calls on inputs of size $n/2$ gives recurrence $T(n) = 4T(n/2) + O(n)$, which solves to $O(n^2)$ by the Master Theorem – no improvement at all.

The Trick: Three Multiplications Suffice

Karatsuba’s insight is that we never need the cross-terms $x_1 y_0$ and $x_0 y_1$ individually. We only need their sum. And that sum falls out for free from a single cleverly chosen multiplication.

Define three products:

\[\begin{aligned} z_2 &= x_1 \cdot y_1 \\ z_0 &= x_0 \cdot y_0 \\ z_1 &= (x_1 + x_0)(y_1 + y_0) - z_2 - z_0 \end{aligned}\]

Expand $z_1$ to see why this works:

\[(x_1 + x_0)(y_1 + y_0) = \underbrace{x_1 y_1}_{z_2} + x_1 y_0 + x_0 y_1 + \underbrace{x_0 y_0}_{z_0}\]

Subtracting $z_2$ and $z_0$ cancels the terms we already know, leaving exactly the cross-term sum $x_1 y_0 + x_0 y_1$. We have extracted the middle coefficient using one multiplication and two subtractions – operations that cost only $O(n)$, negligible compared to multiplication.

The final product assembles as:

\[xy = z_2 \cdot B^{2m} + z_1 \cdot B^m + z_0\]

Three multiplications. Not four. At every level of the recursion, we save 25% of the multiplicative work, and that savings compounds exponentially as we recurse deeper.

The Payoff

The recurrence is now $T(n) = 3T(n/2) + O(n)$, and the Master Theorem gives:

Method	Recurrence	Complexity
Schoolbook	$T(n)=4T(n/2)+O(n)$	$O(n^{\log_2 4}) = O(n^2)$
Karatsuba	$T(n)=3T(n/2)+O(n)$	$O(n^{\log_2 3}) \approx O(n^{1.585})$

The gap between $n^2$ and $n^{1.585}$ may look modest for small $n$, but it widens relentlessly. At 1,000 digits the schoolbook method performs $\sim 10^6$ primitive multiplications; Karatsuba performs $\sim 10^{4.75} \approx 56{,}000$ – a 17$\times$ speedup. At 10,000 digits the ratio exceeds 100$\times$. The deeper the recursion, the more the saved quarter compounds.

The interactive visualization below shows the heart of the trick: how four multiplication blocks collapse into three.

Toom-Cook: Generalizing Karatsuba via Polynomial Interpolation

From Digits to Polynomials

Karatsuba showed that splitting a number in two and exploiting an algebraic identity could reduce four half-size multiplications to three. A natural question follows: what happens if we split into three pieces? Or $k$? This is exactly the generalization that Andrei Toom (1963) and Stephen Cook (1966) independently formalized. The resulting family of algorithms – collectively known as Toom-Cook or Toom-$k$ – systematically trades more additions and scalar operations for fewer recursive multiplications, pushing the exponent ever closer to 1.

The key conceptual shift is to stop viewing integers as flat strings of digits and instead view them as polynomials. If we partition an $n$-digit number into $k$ blocks of roughly $m = \lceil n/k \rceil$ digits each, writing $B = 10^m$ (or $2^m$ in binary), then:

\[x = a_{k-1} B^{k-1} + a_{k-2} B^{k-2} + \cdots + a_1 B + a_0\]

This is simply the number $x$ evaluated at the point $z = B$ of the polynomial:

\[P_x(z) = a_{k-1} z^{k-1} + a_{k-2} z^{k-2} + \cdots + a_1 z + a_0\]

Multiplying two such integers $x$ and $y$ is therefore equivalent to computing the product polynomial $P_x(z) \cdot P_y(z)$ and then evaluating the result at $z = B$ (with appropriate carries). If each input polynomial has degree $k-1$, their product has degree $2(k-1) = 2k - 2$, and is therefore determined by exactly $2k - 1$ point-value pairs – a fact guaranteed by the Fundamental Theorem of Algebra.

This is the heart of the speedup: instead of multiplying the coefficients pairwise (which would require $k^2$ recursive multiplications), we can recover the product polynomial from only $2k - 1$ pointwise products.

The Five Phases of Toom-Cook

The algorithm proceeds through five clearly delineated stages:

Phase 1: Splitting

Break each $n$-digit operand into $k$ blocks of $\sim n/k$ digits. For Toom-3, this yields three coefficients per number:

\[\begin{aligned} x &= a_2 B^{2m} + a_1 B^m + a_0 &\longleftrightarrow\quad P_x(z) &= a_2 z^2 + a_1 z + a_0 \\ y &= b_2 B^{2m} + b_1 B^m + b_0 &\longleftrightarrow\quad P_y(z) &= b_2 z^2 + b_1 z + b_0 \end{aligned}\]

Phase 2: Evaluation

Select $2k - 1$ distinct evaluation points. The standard choice for Toom-3 is the set ${0,\; 1,\; -1,\; 2,\; \infty}$, chosen because they minimize the size of intermediate values and keep the arithmetic simple. Evaluate both polynomials at each point:

\[\begin{aligned} P_x(0) &= a_0 & P_y(0) &= b_0 \\ P_x(1) &= a_2 + a_1 + a_0 & P_y(1) &= b_2 + b_1 + b_0 \\ P_x(-1) &= a_2 - a_1 + a_0 & P_y(-1) &= b_2 - b_1 + b_0 \\ P_x(2) &= 4a_2 + 2a_1 + a_0 & P_y(2) &= 4b_2 + 2b_1 + b_0 \\ P_x(\infty) &= a_2 & P_y(\infty) &= b_2 \end{aligned}\]

The “evaluation at $\infty$” is a notational convenience: it extracts the leading coefficient of the polynomial, since $\lim_{z \to \infty} P(z)/z^{k-1}$ equals the leading coefficient.

Phase 3: Pointwise Multiplication

Multiply the evaluated values at each point. These are the only recursive multiplications the algorithm performs:

\[\begin{aligned} W_0 &= P_x(0) \cdot P_y(0) = a_0 b_0 \\ W_1 &= P_x(1) \cdot P_y(1) \\ W_{-1} &= P_x(-1) \cdot P_y(-1) \\ W_2 &= P_x(2) \cdot P_y(2) \\ W_\infty &= P_x(\infty) \cdot P_y(\infty) = a_2 b_2 \end{aligned}\]

Five multiplications on operands of size $\sim n/3$, rather than the nine that naive coefficient-by-coefficient expansion would require.

Phase 4: Interpolation

The product polynomial $R(z) = P_x(z) \cdot P_y(z)$ has degree 4, so it has five coefficients $C_0, C_1, C_2, C_3, C_4$:

\[R(z) = C_4 z^4 + C_3 z^3 + C_2 z^2 + C_1 z + C_0\]

From the five evaluated products, we can read off:

\[\begin{aligned} W_0 &= C_0 \\ W_1 &= C_4 + C_3 + C_2 + C_1 + C_0 \\ W_{-1} &= C_4 - C_3 + C_2 - C_1 + C_0 \\ W_2 &= 16C_4 + 8C_3 + 4C_2 + 2C_1 + C_0 \\ W_\infty &= C_4 \end{aligned}\]

This is a $5 \times 5$ linear system in the unknowns $C_0, \ldots, C_4$. Because $C_0 = W_0$ and $C_4 = W_\infty$ are immediate, the system reduces quickly. The remaining coefficients are solved by elimination:

\[\begin{aligned} C_0 &= W_0 \\[4pt] C_4 &= W_\infty \\[4pt] C_2 &= \frac{W_1 + W_{-1}}{2} - C_0 - C_4 \\[4pt] C_3 &= \frac{W_2 - 2W_1 - 14C_4 - 2C_2 + C_0}{6} \\[4pt] C_1 &= W_1 - C_4 - C_3 - C_2 - C_0 \end{aligned}\]

Note that these expressions involve only additions, subtractions, and divisions by small constants (2 and 6) – all $O(n)$ operations, negligible compared to the recursive multiplications.

Phase 5: Recomposition

Reassemble the final integer from the product polynomial’s coefficients:

\[x \cdot y = C_4 B^{4m} + C_3 B^{3m} + C_2 B^{2m} + C_1 B^m + C_0\]

The multiplications by powers of $B$ are simply left-shifts, and the final addition with carry propagation is $O(n)$.

A Worked Example

To make this concrete, consider $x = 123{,}456{,}789$ and $y = 987{,}654{,}321$ in base 10 with $k = 3$ and $m = 3$ (so $B = 10^3 = 1000$):

\[\begin{aligned} x &= 123 \cdot 10^6 + 456 \cdot 10^3 + 789 \quad\Longleftrightarrow\quad P_x(z) = 123z^2 + 456z + 789 \\ y &= 987 \cdot 10^6 + 654 \cdot 10^3 + 321 \quad\Longleftrightarrow\quad P_y(z) = 987z^2 + 654z + 321 \end{aligned}\]

Evaluating at the five standard points:

Point	$P_x$	$P_y$	$W = P_x \cdot P_y$
$0$	$789$	$321$	$253{,}269$
$1$	$1{,}368$	$1{,}962$	$2{,}684{,}016$
$-1$	$456$	$654$	$298{,}224$
$2$	$2{,}193$	$5{,}595$	$12{,}269{,}835$
$\infty$	$123$	$987$	$121{,}401$

Interpolation then yields the five coefficients $C_0, \ldots, C_4$, and recomposition with $B = 1000$ recovers the product $121{,}932{,}631{,}112{,}635{,}269$.

Complexity Analysis

The recurrence for Toom-$k$ is:

\[T(n) = (2k - 1)\, T\!\left(\frac{n}{k}\right) + O(n)\]

By the Master Theorem, this solves to:

\[T(n) = O\!\left(n^{\log_k(2k-1)}\right)\]

The exponent $\log_k(2k - 1)$ decreases monotonically as $k$ increases, approaching 1 from above:

Algorithm	Split ($k$)	Recursive Mults ($2k-1$)	Complexity
Grade School	$n$	$n^2$	$O(n^2)$
Karatsuba (Toom-2)	$2$	$3$	$O(n^{\log_2 3}) \approx O(n^{1.585})$
Toom-3	$3$	$5$	$O(n^{\log_3 5}) \approx O(n^{1.465})$
Toom-4	$4$	$7$	$O(n^{\log_4 7}) \approx O(n^{1.404})$
Toom-$k$	$k$	$2k-1$	$O!\big(n^{\log_k(2k-1)}\big)$

The Hidden Cost: Why We Cannot Simply Let $k \to \infty$

A tempting conclusion is that by choosing $k$ large enough, we can push the exponent arbitrarily close to 1 and achieve near-linear multiplication. In practice, this reasoning breaks down for two reasons:

Evaluation and interpolation overhead. The matrices involved in evaluation and interpolation grow as $O(k^2)$, and the entries grow in magnitude. For large $k$, the scalar additions and divisions in the interpolation phase cease to be negligible. The constant hidden in the $O(n)$ additive term balloons.
Coefficient blowup. Evaluating at points like $2, -2, 3, \ldots$ produces intermediate values that are significantly larger than the original coefficients. This “coefficient swell” increases the size of the sub-problems fed to the recursive multiplications, partially negating the savings.

The practical sweet spot is typically Toom-3 or Toom-4. Beyond that, the FFT-based methods (Schönhage-Strassen and its successors) offer a fundamentally better asymptotic trade-off. The transition from Toom-Cook to FFT-based multiplication is, in a sense, the transition from finite polynomial interpolation to interpolation at infinitely many structured points – the roots of unity.

Karatsuba as Toom-2: A Unifying Perspective

It is worth pausing to note that Karatsuba’s algorithm is precisely Toom-Cook with $k = 2$. The “trick” of computing $(x_1 + x_0)(y_1 + y_0) - z_2 - z_0$ is the interpolation step for a degree-2 product polynomial evaluated at the points ${0, 1, \infty}$. Karatsuba’s genius was to discover this special case in 1960; Toom and Cook’s contribution was to recognize the general structure of which Karatsuba is the simplest instance.

The Fourier Transform: From Signals to Arithmetic

Why We Need a New Idea

At the end of the Toom-Cook story we noted a frustrating ceiling: as we increase the splitting parameter $k$, the exponent $\log_k(2k-1)$ drifts toward 1, but it never reaches it. Worse, the constant factor in the $O(n)$ additive work (evaluation and interpolation matrices of size $k \times k$, coefficient blowup at large evaluation points) grows so fast that no finite $k$ delivers practical gains beyond Toom-4 or so.

To break through this wall we need to abandon the strategy of evaluating at a handful of ad-hoc points and instead evaluate at a structured infinite family of points whose algebraic symmetry enables a radically faster algorithm. That family is the complex roots of unity, and the algorithm that exploits their symmetry is the Fast Fourier Transform.

Before we can state Schönhage and Strassen’s multiplication algorithm, we must build the FFT from the ground up. This requires four conceptual layers: (1) the observation that integer multiplication is polynomial convolution, (2) the Discrete Fourier Transform as an evaluation map, (3) the Cooley-Tukey decomposition that makes the DFT fast, and (4) the inverse transform that recovers coefficients from point values.

Multiplication Is Convolution

Polynomials and digit vectors

We have already seen that an $n$-digit integer in base $B$ can be written as a polynomial evaluated at $B$. Let us now make the multiplication side precise. Given two integers with digit vectors $\mathbf{a} = (a_0, a_1, \ldots, a_{n-1})$ and $\mathbf{b} = (b_0, b_1, \ldots, b_{n-1})$, their product is a new integer whose digit vector $\mathbf{c} = (c_0, c_1, \ldots, c_{2n-2})$ satisfies:

\[c_k = \sum_{j=0}^{k} a_j \, b_{k-j} \qquad \text{for } k = 0, 1, \ldots, 2n-2\]

(with the convention that $a_j = 0$ for $j \geq n$ and similarly for $b_j$). This is exactly the definition of the linear convolution of the sequences $\mathbf{a}$ and $\mathbf{b}$, or equivalently, the coefficient vector of the product polynomial $A(x) \cdot B(x)$ where:

\[A(x) = \sum_{j=0}^{n-1} a_j x^j, \qquad B(x) = \sum_{j=0}^{n-1} b_j x^j\]

Computing this convolution directly requires computing each of the $2n - 1$ output coefficients, and for each we sum up to $n$ products. The total cost is $O(n^2)$ – schoolbook multiplication in disguise.

The Convolution Theorem

The single most important theorem in this entire story is:

Convolution Theorem. Let $\mathcal{F}$ denote the Discrete Fourier Transform (defined below). If $\mathbf{c} = \mathbf{a} \ast \mathbf{b}$ is the convolution of two sequences, then:
\[\mathcal{F}(\mathbf{a} \ast \mathbf{b}) = \mathcal{F}(\mathbf{a}) \cdot \mathcal{F}(\mathbf{b})\]
where $\cdot$ denotes pointwise (component-by-component) multiplication.

In words: convolution in the “coefficient domain” becomes pointwise multiplication in the “frequency domain.” Since pointwise multiplication of two length-$N$ vectors costs only $O(N)$, the entire cost of multiplication reduces to the cost of two forward transforms and one inverse transform. If each transform costs $O(N \log N)$, the total cost is $O(N \log N)$ – an exponential improvement over $O(N^2)$.

This is not a hand-wave. Let us now build the DFT rigorously and prove why it can be computed in $O(N \log N)$.

The Discrete Fourier Transform

Complex exponentials and Euler’s formula

Before defining the DFT, we need the language of complex numbers. Recall Euler’s formula:

\[e^{i\theta} = \cos\theta + i\sin\theta\]

This elegant identity tells us that the complex exponential $e^{i\theta}$ traces out the unit circle in the complex plane as $\theta$ varies from $0$ to $2\pi$. Every point on the unit circle can be written as $e^{i\theta}$ for some angle $\theta$.

The $N$-th roots of unity

Fix a positive integer $N$. The $N$-th roots of unity are the $N$ complex numbers that satisfy $z^N = 1$. They are:

\[\omega_N^k = e^{2\pi i k / N}, \qquad k = 0, 1, \ldots, N-1\]

These are $N$ points spaced equally around the unit circle, like the vertices of a regular $N$-gon inscribed in the circle $

= 1$. The primitive $N$-th root of unity is:

\[\omega_N = e^{2\pi i / N}\]

so that $\omega_N^k = (\omega_N)^k$. We will usually drop the subscript $N$ when the context is clear.

The roots of unity possess remarkable algebraic properties that are the engine of the FFT:

Periodicity: $\omega^{k+N} = \omega^k$ for all $k$. The roots cycle with period $N$.
Cancellation (Half-turn symmetry): $\omega^{k + N/2} = -\omega^k$ when $N$ is even. Geometrically, the point diametrically opposite $\omega^k$ on the unit circle is $-\omega^k$. This is the single most important property for the FFT.
Summation: $\displaystyle\sum_{k=0}^{N-1} \omega^{jk} = \begin{cases} N & \text{if } N \mid j \ 0 & \text{otherwise} \end{cases}$

This orthogonality relation is what makes the inverse DFT work.
Squaring (Halving): If $N$ is even, then ${(\omega_N^k)^2 : k = 0, \ldots, N-1}$ gives exactly the $N/2$-th roots of unity, each appearing twice. That is, $(\omega_N)^2 = \omega_{N/2}$.

Definition of the DFT

Given a vector $\mathbf{x} = (x_0, x_1, \ldots, x_{N-1})$, its Discrete Fourier Transform is the vector $\mathbf{X} = (X_0, X_1, \ldots, X_{N-1})$ defined by:

\[X_k = \sum_{j=0}^{N-1} x_j \, \omega_N^{jk}, \qquad k = 0, 1, \ldots, N-1\]

Equivalently, $X_k = P(\omega^k)$ where $P(z) = \sum_{j} x_j z^j$ is the polynomial whose coefficients are the entries of $\mathbf{x}$. The DFT is nothing more than evaluating the polynomial at all $N$-th roots of unity simultaneously.

The DFT matrix

We can express the DFT as a matrix-vector product $\mathbf{X} = F_N \, \mathbf{x}$ where the DFT matrix $F_N$ has entries:

\[(F_N)_{k,j} = \omega_N^{jk}\]

Explicitly, for $N = 4$ with $\omega = \omega_4 = e^{2\pi i/4} = i$:

\[F_4 = \begin{pmatrix} 1 & 1 & 1 & 1 \\ 1 & i & i^2 & i^3 \\ 1 & i^2 & i^4 & i^6 \\ 1 & i^3 & i^6 & i^9 \end{pmatrix} = \begin{pmatrix} 1 & 1 & 1 & 1 \\ 1 & i & -1 & -i \\ 1 & -1 & 1 & -1 \\ 1 & -i & -1 & i \end{pmatrix}\]

Computing $\mathbf{X} = F_N \mathbf{x}$ by brute-force matrix-vector multiplication costs $O(N^2)$. The entire point of the FFT is to exploit the structure of $F_N$ to compute this product in $O(N \log N)$.

The Inverse DFT

The orthogonality of the roots of unity (property 3 above) guarantees that the DFT is invertible. The inverse is:

\[x_j = \frac{1}{N} \sum_{k=0}^{N-1} X_k \, \omega_N^{-jk}\]

or in matrix form, $\mathbf{x} = \frac{1}{N} F_N^{-1} \mathbf{X}$ where $F_N^{-1}$ is the matrix with entries $\omega_N^{-jk}$ – the same as $F_N$ but with $\omega$ replaced by $\omega^{-1} = \overline{\omega}$ (the complex conjugate). In other words, the inverse DFT is computed by the same algorithm as the forward DFT, just with the twiddle factors conjugated and the output scaled by $1/N$. Any algorithm that computes the forward DFT efficiently also computes the inverse efficiently, for free.

Proof of inversion. We need to verify that $\frac{1}{N}F_N^{-1} F_N = I_N$, i.e., that:

\[\frac{1}{N} \sum_{k=0}^{N-1} \omega^{-jk} \omega^{k\ell} = \frac{1}{N} \sum_{k=0}^{N-1} \omega^{k(\ell - j)} = \begin{cases} 1 & \text{if } j = \ell \\ 0 & \text{if } j \neq \ell \end{cases}\]

When $j = \ell$, every term in the sum is $\omega^0 = 1$, so the sum is $N$ and we get $N/N = 1$. When $j \neq \ell$, let $m = \ell - j \not\equiv 0 \pmod{N}$. Then $\sum_{k=0}^{N-1} \omega^{mk}$ is a geometric series with ratio $r = \omega^m \neq 1$:

\[\sum_{k=0}^{N-1} r^k = \frac{r^N - 1}{r - 1} = \frac{(\omega^N)^m - 1}{\omega^m - 1} = \frac{1 - 1}{\omega^m - 1} = 0\]

since $\omega^N = 1$ by definition. $\blacksquare$

The Fast Fourier Transform (Cooley-Tukey, Radix-2)

The core idea: divide and conquer on even and odd indices

The breakthrough of Cooley and Tukey (1965) – though the idea traces back to Gauss (1805) – is to observe that when $N$ is even, the DFT of size $N$ can be decomposed into two DFTs of size $N/2$ plus $O(N)$ additional work.

Write $N = 2M$. Split the input sequence $\mathbf{x}$ into its even-indexed and odd-indexed elements:

\[\begin{aligned} \mathbf{e} &= (x_0, x_2, x_4, \ldots, x_{N-2}) \quad \text{(even indices)} \\ \mathbf{d} &= (x_1, x_3, x_5, \ldots, x_{N-1}) \quad \text{(odd indices)} \end{aligned}\]

Now consider the DFT sum for an arbitrary output index $k$:

\[X_k = \sum_{j=0}^{N-1} x_j \, \omega_N^{jk}\]

Separate the even-indexed and odd-indexed terms:

\[X_k = \underbrace{\sum_{m=0}^{M-1} x_{2m} \, \omega_N^{2mk}}_{E_k} + \underbrace{\omega_N^k \sum_{m=0}^{M-1} x_{2m+1} \, \omega_N^{2mk}}_{= \omega_N^k \cdot D_k}\]

Using the squaring property $\omega_N^2 = \omega_M$ (where $M = N/2$), each of these inner sums is itself a DFT of size $M$:

\[\begin{aligned} E_k &= \sum_{m=0}^{M-1} x_{2m} \, \omega_M^{mk} = \text{DFT}_M(\mathbf{e})_k \\[4pt] D_k &= \sum_{m=0}^{M-1} x_{2m+1} \, \omega_M^{mk} = \text{DFT}_M(\mathbf{d})_k \end{aligned}\]

So the full DFT decomposes as:

\[\boxed{X_k = E_k + \omega_N^k \cdot D_k}\]

But we need $X_k$ for $k = 0, 1, \ldots, N-1$, while $E_k$ and $D_k$ are periodic with period $M = N/2$ (they are DFTs of size $M$). For the “upper half” $k + M$, the cancellation property $\omega_N^{k+M} = -\omega_N^k$ gives:

\[\boxed{X_{k+M} = E_k - \omega_N^k \cdot D_k}\]

These two equations together constitute the butterfly operation: from the pair $(E_k, D_k)$ and the twiddle factor $\omega_N^k$, we compute both $X_k$ and $X_{k+M}$ using one complex multiplication and two complex additions.

The butterfly diagram

Each butterfly takes two inputs, multiplies one by a twiddle factor, and produces two outputs via addition and subtraction:

  E_k ──────────┬──── (+) ──── X_k
                 │
          ω^k    ×
                 │
  D_k ──────────┴──── (−) ──── X_{k+M}

For $M = N/2$ values of $k$ (namely $k = 0, 1, \ldots, M-1$), we perform $M$ butterflies, each costing $O(1)$. The total additional work at this level of recursion is $O(N)$.

The recurrence and complexity

Let $T(N)$ denote the cost of computing a DFT of size $N$ (assumed a power of 2). The Cooley-Tukey decomposition gives:

\[T(N) = 2\,T(N/2) + O(N)\]

By the Master Theorem (case 2, with $a = 2$, $b = 2$, $f(N) = \Theta(N)$):

\[T(N) = O(N \log N)\]

compared to $O(N^2)$ for the naive DFT. For $N = 2^{20} \approx 10^6$, this is the difference between $\sim 10^{12}$ operations and $\sim 2 \times 10^7$ – a speedup of 50,000$\times$.

The full recursion tree

When $N = 2^s$, the recursion unfolds $s = \log_2 N$ levels deep. At each level, we partition the data, recurse on two halves, and combine with $N$ butterfly operations. The total work is:

\[\underbrace{N}_{\text{level 0}} + \underbrace{N}_{\text{level 1}} + \cdots + \underbrace{N}_{\text{level } s-1} = s \cdot N = N \log_2 N\]

Each level performs exactly $N/2$ butterflies (each costing one complex multiplication and two additions), for a total of $\frac{N}{2}\log_2 N$ complex multiplications.

A worked example: 8-point FFT

Let $N = 8$ and $\omega = \omega_8 = e^{2\pi i/8} = e^{i\pi/4} = \frac{1+i}{\sqrt{2}}$. Consider the input $\mathbf{x} = (1, 1, 1, 1, 0, 0, 0, 0)$ (a rectangular pulse).

Level 0 (split): Separate into even and odd:

\[\mathbf{e} = (x_0, x_2, x_4, x_6) = (1, 1, 0, 0), \qquad \mathbf{d} = (x_1, x_3, x_5, x_7) = (1, 1, 0, 0)\]

Level 1: Each half recursively splits again. For $\mathbf{e}$:

\[\mathbf{e}_{\text{even}} = (1, 0), \quad \mathbf{e}_{\text{odd}} = (1, 0)\]

These are 2-point DFTs: $\text{DFT}_2(a, b) = (a + b, \; a - b)$. So:

\[\text{DFT}_2(1, 0) = (1, 1) \quad \text{for both}\]

Level 1 butterfly (for $\mathbf{e}$): Combine with twiddle factors $\omega_4^0 = 1$ and $\omega_4^1 = i$:

\[E_0 = 1 + 1 \cdot 1 = 2, \quad E_2 = 1 - 1 \cdot 1 = 0\] \[E_1 = 1 + i \cdot 1 = 1 + i, \quad E_3 = 1 - i \cdot 1 = 1 - i\]

So $\text{DFT}_4(\mathbf{e}) = (2, \; 1+i, \; 0, \; 1-i)$.

An identical calculation gives $\text{DFT}_4(\mathbf{d}) = (2, \; 1+i, \; 0, \; 1-i)$.

Level 0 butterfly: Combine with twiddle factors $\omega_8^k$ for $k = 0, 1, 2, 3$:

\[\begin{aligned} X_0 &= E_0 + \omega^0 D_0 = 2 + 1 \cdot 2 = 4 \\ X_1 &= E_1 + \omega^1 D_1 = (1+i) + \tfrac{1+i}{\sqrt{2}}(1+i) = (1+i) + \tfrac{2i}{\sqrt{2}} = 1 + i + i\sqrt{2} \\ X_2 &= E_2 + \omega^2 D_2 = 0 + i \cdot 0 = 0 \\ X_3 &= E_3 + \omega^3 D_3 = (1-i) + \tfrac{-1+i}{\sqrt{2}}(1-i) = (1-i) + \tfrac{-1+i-i(-1)+i^2}{\sqrt{2}} \\ &\phantom{=}\; \text{(and the corresponding } X_{k+4} = E_k - \omega^k D_k \text{ for the upper half)} \end{aligned}\]

The key observation is not the specific numerical values but the structure: $3$ levels of $4$ butterflies each $= 12$ complex multiplications, versus $8^2 = 64$ for the naive DFT. The ratio $12/64 \approx 19\%$ – and this ratio improves as $N$ grows.

The visualization below animates this 8-point butterfly network. Watch how data flows through three stages of butterflies – each stage halving the sub-problem size – with twiddle factors ($\omega^k$) applied at every crossing.

Putting It Together: FFT-Based Polynomial Multiplication

We now have all the pieces. To multiply two polynomials $A(x)$ and $B(x)$ of degree $n-1$ (and thereby two $n$-digit integers):

Step 1. Pad. The product polynomial has degree $2n - 2$, so we need at least $2n - 1$ evaluation points. Choose $N = 2^{\lceil \log_2(2n) \rceil}$ (the next power of 2 at or above $2n$). Pad both coefficient vectors with zeros to length $N$.

Step 2. Forward FFT. Compute $\hat{\mathbf{a}} = \text{FFT}_N(\mathbf{a})$ and $\hat{\mathbf{b}} = \text{FFT}_N(\mathbf{b})$. Cost: $O(N \log N)$ each.

Step 3. Pointwise multiply. Compute $\hat{c}_k = \hat{a}_k \cdot \hat{b}_k$ for $k = 0, \ldots, N-1$. Cost: $O(N)$.

Step 4. Inverse FFT. Compute $\mathbf{c} = \text{IFFT}_N(\hat{\mathbf{c}})$. Cost: $O(N \log N)$.

Step 5. Carry propagation. The entries of $\mathbf{c}$ are the exact convolution coefficients (no rounding – yet). Each $c_k$ may exceed the base $B$, so we perform a single left-to-right carry pass: set $c_k \leftarrow c_k \bmod B$ and add $\lfloor c_k / B \rfloor$ to $c_{k+1}$. Cost: $O(N)$.

Total cost: $O(N \log N) = O(n \log n)$.

At this point you might ask: if the FFT already gives us $O(n \log n)$ multiplication, why do we need Schönhage-Strassen? The answer lies in a subtle but critical issue: numerical precision.

Schönhage-Strassen: Multiplication in $O(n \log n \log \log n)$

The Precision Problem

The FFT-based multiplication pipeline described above works over the complex numbers. The twiddle factors $\omega_N^k = e^{2\pi i k/N}$ are irrational (for most $k$), and so are the intermediate values in the FFT. On a real computer, we approximate these with floating-point arithmetic.

For multiplying two numbers with $n$ digits, the convolution coefficients can be as large as $O(n B^2)$ (where $B$ is the base). To distinguish these integers exactly after rounding, we need floating-point precision of roughly $O(\log n + 2\log B)$ bits. For large $n$, this means we need multi-precision floating-point arithmetic inside the FFT itself – and each multi-precision operation costs more than $O(1)$.

This creates a vicious circle: to multiply $n$-digit numbers, we use the FFT, but the FFT internally needs high-precision multiplications, which are themselves expensive. The complex-number FFT approach does not, by itself, yield a clean $O(n \log n)$ integer multiplication algorithm.

Schönhage and Strassen’s 1971 breakthrough was to eliminate this precision problem entirely by moving from the complex numbers to an exact algebraic setting where roots of unity exist but rounding errors do not.

From $\mathbb{C}$ to $\mathbb{Z}/(2^m + 1)\mathbb{Z}$: The Number Theoretic Transform

Roots of unity in finite rings

The FFT algorithm does not actually require the complex numbers. It requires only a ring $R$ that contains:

An element $\omega$ of multiplicative order $N$ (an “$N$-th root of unity”), meaning $\omega^N = 1$ and $\omega^k \neq 1$ for $0 < k < N$.
An inverse of $N$ in $R$ (for the inverse transform).

If $R$ is a finite ring (like $\mathbb{Z}/p\mathbb{Z}$ for a prime $p$), then all arithmetic is exact – no rounding, no precision issues. An FFT performed in such a ring is called a Number Theoretic Transform (NTT).

Example. In $\mathbb{Z}/5\mathbb{Z}$, the element $2$ has order $4$: $2^1 = 2$, $2^2 = 4$, $2^3 = 3$, $2^4 = 1 \pmod{5}$. So $\omega = 2$ is a primitive $4$th root of unity in $\mathbb{Z}/5\mathbb{Z}$, and we can perform a 4-point NTT modulo 5 with exact arithmetic.

Schönhage and Strassen’s choice: Fermat-like rings

To multiply two $n$-bit integers, Schönhage and Strassen work in the ring:

\[R = \mathbb{Z} / (2^m + 1)\mathbb{Z}\]

for a carefully chosen $m$. This ring has a remarkable property: the element $\omega = 2$ (or more precisely, a small power of $2$) serves as a root of unity, and multiplication by powers of 2 in this ring is just a bit-shift followed by a reduction modulo $2^m + 1$ – an operation that costs $O(m)$, which is essentially free.

Why does $2$ behave as a root of unity here? Note that $2^m \equiv -1 \pmod{2^m + 1}$, so $2^{2m} \equiv 1 \pmod{2^m + 1}$. Thus $2$ has multiplicative order dividing $2m$ in this ring. With an appropriate choice of $m$, we can ensure that $2$ (or a power of it) is a primitive $N$-th root of unity for the $N$ we need.

The critical advantage: all the “twiddle factor multiplications” in the FFT butterfly – which in the complex-number FFT require expensive multi-precision multiplications – reduce to bit-shifts and additions in $\mathbb{Z}/(2^m + 1)\mathbb{Z}$. This makes the per-butterfly cost $O(m)$ instead of $O(m \log m \cdots)$, dramatically simplifying the recursion.

The Algorithm in Detail

Step 0: Setup

Given two $n$-bit integers $x$ and $y$, choose parameters:

Let $K = 2^k$ for some $k$ (the number of chunks).
Let $m = \lceil n / K \rceil + O(k)$ (the chunk size in bits, with some padding for carries).
Work in the ring $R = \mathbb{Z}/(2^m + 1)\mathbb{Z}$.

The parameters are chosen so that $K \approx \sqrt{n}$ (roughly), meaning we split each input into about $\sqrt{n}$ chunks of about $\sqrt{n}$ bits each.

Step 1: Decomposition

Break each $n$-bit integer into $K$ chunks of $\sim m$ bits:

\[x = \sum_{j=0}^{K-1} a_j \cdot 2^{jm}, \qquad y = \sum_{j=0}^{K-1} b_j \cdot 2^{jm}\]

Form the polynomials $A(z) = \sum a_j z^j$ and $B(z) = \sum b_j z^j$ over $R$, so that $x = A(2^m)$ and $y = B(2^m)$.

Step 2: Forward NTT

Compute $\hat{\mathbf{a}} = \text{NTT}_N(\mathbf{a})$ and $\hat{\mathbf{b}} = \text{NTT}_N(\mathbf{b})$ in the ring $R$, where $N \geq 2K$ is a suitable power of 2.

The NTT is the same Cooley-Tukey FFT algorithm, but:

All arithmetic is modular (mod $2^m + 1$).
Twiddle factor multiplications $\omega^k \cdot v$ are implemented as cyclic bit-shifts of $v$, costing $O(m)$.
Each butterfly costs $O(m)$: one bit-shift plus two modular additions.
Total: $N/2 \cdot \log_2 N$ butterflies, each $O(m)$, giving $O(Nm \log N)$ bit operations.

Step 3: Pointwise multiplication

Compute $\hat{c}_k = \hat{a}_k \cdot \hat{b}_k$ in $R$ for each $k = 0, \ldots, N - 1$.

Each of these is a multiplication of two $m$-bit numbers modulo $2^m + 1$. But wait – this is itself a multiplication of smaller numbers! This is where the algorithm becomes recursive: we use the Schönhage-Strassen algorithm itself (on inputs of size $m$ instead of $n$) to perform these multiplications.

There are $N$ such multiplications, each on operands of size $m$. This is the “recursive nesting” that generates the $\log \log n$ factor.

Step 4: Inverse NTT

Compute $\mathbf{c} = \text{INTT}_N(\hat{\mathbf{c}})$. Same cost as the forward NTT: $O(Nm \log N)$.

Step 5: Carry propagation and reassembly

The vector $\mathbf{c}$ now contains the convolution of $\mathbf{a}$ and $\mathbf{b}$ modulo $2^m + 1$. (The parameters are chosen so that no coefficient is large enough to “wrap around” the modulus – the modulus is larger than any possible convolution coefficient.) Reassemble the final product by performing carries across the chunks.

Why $\log \log n$?

The complexity breaks down as follows. At the top level we have:

NTT and INTT: $O(Nm \log N)$ bit operations for the transforms.
Pointwise multiplications: $N$ recursive multiplications on $m$-bit inputs.

With $K \approx \sqrt{n}$ and $m \approx \sqrt{n}$, the recurrence is roughly:

\[T(n) = O\!\left(\sqrt{n} \cdot T(\sqrt{n})\right) + O(n \log n)\]

Let us trace the recursion. At depth $d$, the problem size is $n^{1/2^d}$. The recursion bottoms out when $n^{1/2^d} = O(1)$, which requires $2^d \approx \log n$, giving a recursion depth of $d = O(\log \log n)$.

At each recursion level, the non-recursive work is $O(n \log n)$ (the NTTs). Summing over $O(\log \log n)$ levels:

\[T(n) = O(n \log n \cdot \log \log n)\]

This is the Schönhage-Strassen bound. The $\log \log n$ factor is not a deficiency of the FFT itself – the FFT is $O(n \log n)$. It is the cost of the recursive multiplications needed at each level of the NTT.

A Comparison

Algorithm	Complexity	Key Innovation
Schoolbook	$O(n^2)$	Direct digit-by-digit
Karatsuba	$O(n^{1.585})$	4 multiplications $\to$ 3
Toom-3	$O(n^{1.465})$	9 multiplications $\to$ 5
Schönhage-Strassen	$O(n \log n \log \log n)$	NTT in $\mathbb{Z}/(2^m+1)$
Harvey-van der Hoeven	$O(n \log n)$	Multi-dimensional NTT (see below)

The jump from Toom-Cook to Schönhage-Strassen is qualitatively different from all previous improvements. Karatsuba and Toom-Cook are algebraic tricks that reduce the exponent toward 1 but never reach it. Schönhage-Strassen breaks through to a nearly-linear bound by replacing polynomial interpolation at finitely many points with Fourier analysis at the roots of unity – and crucially, by doing so in an exact algebraic ring where the symmetry of the roots makes the transform cheap.

The remaining $\log \log n$ factor is the residue of recursive nesting. Eliminating it required the multi-dimensional approach of Harvey and van der Hoeven (2019), which we now discuss.

Harvey-van der Hoeven: $O(n \log n)$

The Last Factor Standing

We have arrived at the final chapter. Let us take stock of where we are.

You now understand that multiplying two $n$-digit integers is really computing a convolution of their digits. You understand that the FFT evaluates a polynomial at all $N$-th roots of unity in $O(N \log N)$ time, and that the Convolution Theorem lets us turn this evaluation into multiplication. You understand that Schönhage and Strassen sidestepped floating-point precision by working in the ring $\mathbb{Z}/(2^m + 1)\mathbb{Z}$, where twiddle factors are just bit-shifts.

And you understand the one blemish: the $\log \log n$ factor. It comes from the fact that the NTT’s pointwise multiplications are themselves multiplications on smaller numbers, requiring their own NTTs, which require their own pointwise multiplications, and so on. The recursion bottoms out after $\log \log n$ levels, each contributing $O(n \log n)$ work. Multiply those together: $O(n \log n \log \log n)$.

For 48 years – from 1971 to 2019 – nobody could kill that last factor. Then David Harvey and Joris van der Hoeven did.

The Intuition: Why $\log \log n$ Exists and How to Eliminate It

To understand the fix, we first need a sharper picture of the disease.

The disease: a long chain of recursive calls

In Schönhage-Strassen, we split our $n$-bit number into $K \approx \sqrt{n}$ chunks of $m \approx \sqrt{n}$ bits each. The NTT on these chunks is cheap (just bit-shifts and additions), but the pointwise multiplications – the $K$ products of $m$-bit numbers in $\mathbb{Z}/(2^m+1)$ – each require a recursive call to the entire algorithm.

At the next level down, each $m$-bit multiplication splits into $\sqrt{m}$ chunks of $\sqrt{m}$ bits, and so on. The problem sizes form a chain:

\[n \;\to\; \sqrt{n} \;\to\; n^{1/4} \;\to\; n^{1/8} \;\to\; \cdots \;\to\; O(1)\]

This chain has $\log \log n$ links (since $n^{1/2^d} = O(1)$ when $2^d \approx \log n$, giving $d \approx \log \log n$). Each link does $O(n \log n)$ work in total. The $\log \log n$ factor is simply the number of links in this chain.

The cure: make the chain shorter

Harvey and van der Hoeven’s idea, at its core, is beautifully simple: if the recursion depth is the problem, reduce the recursion depth.

Instead of splitting into $\sqrt{n}$ pieces (which halves the exponent at each level), split into far more pieces that are far smaller. If you split an $n$-bit number into $n/\log n$ pieces of $\log n$ bits each, then the recursive subproblems have size $\log n$ – which is already small enough to multiply by schoolbook in $O((\log n)^2)$ time! The recursion bottoms out in a single step. No chain. No $\log \log n$.

But this creates a new problem. With $n / \log n$ chunks, the NTT must operate on a sequence of length $\sim n / \log n$, and the transform must happen in a ring large enough to hold the convolution without overflow. Finding a ring that simultaneously supports (a) cheap roots of unity, (b) sufficiently many of them, and (c) exact arithmetic with no precision loss – all while keeping the transform cost to $O(n \log n)$ – is the hard part. This is where the paper earns its 80 pages.

The Architecture: Three Key Ideas

Idea 1: Fold the sequence into a multi-dimensional array

Rather than treating the $n / \log n$ chunks as a flat list, Harvey and van der Hoeven reshape them into a $d$-dimensional array of size $s_1 \times s_2 \times \cdots \times s_d$, where each $s_i$ is a small prime and $\prod s_i \approx n / \log n$.

Why does this help? A $d$-dimensional convolution can be computed by performing 1-dimensional DFTs along each dimension in succession (this is the standard “row-column” algorithm for multidimensional transforms). If each dimension $s_i$ is small, each 1D DFT is cheap. And crucially, the total number of 1D DFT operations is:

\[\text{cost} = \sum_{i=1}^{d} \frac{S}{s_i} \cdot (\text{cost of a length-}s_i\text{ DFT})\]

where $S = \prod s_i$. By choosing $d$ to grow with $n$ (specifically, $d \sim \log n / \log \log n$), each $s_i$ stays bounded by a constant, and the DFTs along each dimension have constant cost per element. The total transform cost is $O(S \cdot d) = O!\left(\frac{n}{\log n} \cdot \frac{\log n}{\log \log n}\right) = O!\left(\frac{n}{\log \log n}\right)$, which is well within the $O(n \log n)$ budget.

The problem is that the Cooley-Tukey radix-2 trick does not work on prime-sized dimensions. This is where the second idea comes in.

The Chinese Remainder Theorem: The Bridge Between Dimensions

In this algorithm, the Chinese Remainder Theorem (CRT) is the bridge that turns a massive, “one-dimensional” multiplication problem into a more manageable “multi-dimensional” one.

What is the Chinese Remainder Theorem (CRT)?

In general mathematics, the CRT is a theorem that allows you to uniquely identify a large number by its remainders when divided by a set of smaller, relatively prime numbers.

In the context of this paper, it is used to create an isomorphism (a structural match) between two different algebraic spaces:

One-Dimensional Space: $\mathbb{Z}[x]/(x^{s_1 \cdots s_d} - 1)$ – This represents the large integer split into one long line of chunks.
Multi-Dimensional Space: $\mathbb{Z}[x_1, \ldots, x_d]/(x_1^{s_1}-1, \ldots, x_d^{s_d}-1)$ – This represents the same data arranged on a $d$-dimensional grid (like a cube or hypercube).

Why use CRT at all?

The authors use the CRT for several critical reasons:

To enable “Fast Polynomial Transforms.” The fastest tools for this algorithm (Nussbaumer’s transforms) require the problem to be structured as a multidimensional grid. The CRT is what allows the authors to “reshape” the flat list of integer chunks into that grid.
To reduce problem size. By using the CRT to map the data onto a grid of size $s_1 \times s_2 \times \cdots \times s_d$, the problem of computing one giant Discrete Fourier Transform (DFT) is broken down into a collection of much smaller DFTs along each dimension.
Efficient recursion. The “multi-dimensional” approach is what allows the algorithm to reach the $O(n \log n)$ speed. By splitting the integer into $d$ different dimensions (where $d$ is a parameter they can choose, like 1729), they can reduce the size of the sub-problems much more aggressively at each step of the recursion.

The Workflow Summary

Split: Take the $n$-bit integer and split it into many small chunks.
Map (CRT): Use the CRT to arrange those chunks into a $d$-dimensional grid based on distinct prime numbers ($s_1, \ldots, s_d$).
Multiply: Perform the multiplication on this grid using fast Fourier transforms.
Reverse: Use the inverse of the CRT mapping to “flatten” the result back into a single large integer.

Idea 2: Gaussian resampling – making primes act like powers of 2

The Cooley-Tukey FFT requires the transform length to be a power of 2 (or at least highly composite). The dimensions $s_i$ are primes. How do we bridge this gap?

Harvey and van der Hoeven use a technique inspired by Bluestein’s algorithm, but with a Gaussian twist. The idea is:

Embed the length-$s_i$ DFT into a slightly larger length-$t_i$ cyclic convolution, where $t_i$ is the next power of 2 above $s_i$.
Compute this cyclic convolution using Nussbaumer’s algorithm (a multiplication-free polynomial transform), which only needs additions, subtractions, and cyclic shifts.
The “embedding” is done via multiplication by a Gaussian chirp – a sequence of the form $e^{\pi i k^2 / s_i}$. The Gaussian chirp has the remarkable property that it converts a DFT into a convolution (this is the classical “chirp-$z$ transform” idea of Bluestein). And because the Gaussian is approximately its own Fourier transform, the approximation errors when rounding $s_i$ up to $t_i$ can be made exponentially small with only $O(1)$ extra bits of precision.

The net effect: each prime-sized DFT is replaced by a power-of-2-sized convolution that can be computed without any multiplications in the traditional sense – only additions and shifts. The cost per element remains $O(1)$.

Idea 3: Nussbaumer’s algorithm – transforms without multiplications

The inner convolutions (from Idea 2) are computed using Nussbaumer’s polynomial transform, which operates over the ring $R[y]/(y^r + 1)$ for $r$ a power of 2. In this ring:

Multiplication by $y$ is a cyclic shift (free, like multiplying by $2$ in $\mathbb{Z}/(2^m + 1)$ was free for Schönhage-Strassen).
The transform uses only $O(r \log r)$ additions and subtractions – no multiplications at this level.

This is the key that breaks the recursive chain. In Schönhage-Strassen, the pointwise multiplications in the NTT were genuine multiplications that demanded recursive calls. In Harvey-van der Hoeven, the analogous step uses Nussbaumer transforms that need no multiplications, only shifts and additions. No recursive call is needed. The chain has been cut.

The only remaining multiplications are tiny: each “pointwise product” in the Nussbaumer-transformed domain amounts to a multiplication of numbers with $O(\log n)$ bits, which can be done by schoolbook in $O((\log n)^2)$ time – a cost that is absorbed into the $O(n \log n)$ total.

How the Pieces Fit Together

The full algorithm, stripped to its skeleton:

Split the $n$-bit inputs into $S \approx n / \log n$ chunks of $\sim \log n$ bits each.
Reshape the chunk sequence into a $d$-dimensional array ($d \sim \log n / \log \log n$ dimensions, each of prime size $s_i = O(\log \log n)$).
For each dimension $i = 1, \ldots, d$:
- Apply the Gaussian chirp to convert the length-$s_i$ DFT along that dimension into a cyclic convolution of length $t_i = O(s_i)$.
- Compute that cyclic convolution using Nussbaumer’s addition-only polynomial transform.
Pointwise multiply the transformed arrays. Each pointwise product involves numbers of $O(\log n)$ bits – small enough for schoolbook.
Invert the multidimensional transform (same process in reverse).
Carry-propagate and reassemble the final product.

The total cost at each step:

Steps 3 and 5 (transforms): $O(n)$ additions across $d$ dimensions, times $d = O(\log n / \log \log n)$, giving $O(n \log n / \log \log n)$ – well within budget.
Step 4 (pointwise): $S$ multiplications of $O(\log n)$-bit numbers at $O((\log n)^2)$ each, giving $O(n \log n)$.
Steps 1, 2, 6: $O(n)$.

Total: $O(n \log n)$.

Why 1729?

The algorithm’s correctness depends on the Gaussian resampling errors being negligible, which requires extra precision bits. The number of extra bits grows with $d$, and $d$ grows with $n$. Working through the constants, the algorithm only becomes faster than Schönhage-Strassen when $n$ is so large that the constant-factor overhead of managing $d$ dimensions, Gaussian chirps, and Nussbaumer bookkeeping is finally absorbed.

Harvey and van der Hoeven estimate this crossover at numbers with more than $2^{1729^{12}}$ digits.

The appearance of 1729 – Ramanujan’s famous “taxicab number,” the smallest number expressible as the sum of two cubes in two different ways – is a coincidence, but a poetic one. The number arises from a chain of parameter optimizations in the proof, not from any deep connection to Ramanujan’s work. But it is fitting that the algorithm that closes the book on multiplication complexity should bear, in its constant, an echo of one of mathematics’ most beautiful stories.

To put $2^{1729^{12}}$ in perspective:

The observable universe contains roughly $10^{80}$ atoms.
$2^{1729^{12}}$ has approximately $10^{38}$ decimal digits in its exponent alone.
If every atom in the universe were a hard drive, and every hard drive stored $10^{15}$ digits, you could store roughly $10^{95}$ digits. This is nothing compared to $2^{1729^{12}}$.

This is a galactic algorithm – beautiful, true, and utterly useless for any computation that will ever be performed in the physical universe. But its existence answers a question that stood open for sixty years: can multiplication be done in $O(n \log n)$? The answer is yes.

The View from the Summit

Let us look back at the full arc:

Year	Algorithm	Complexity	Key Insight
antiquity	Schoolbook	$O(n^2)$	Every digit meets every digit
1960	Karatsuba	$O(n^{1.585})$	One clever identity saves 25% per level
1963	Toom-Cook	$O(n^{1+\varepsilon})$ for any $\varepsilon > 0$	Polynomial interpolation at $k$ points
1971	Schönhage-Strassen	$O(n \log n \log \log n)$	NTT in $\mathbb{Z}/(2^m+1)$; roots of unity via bit-shifts
2019	Harvey-van der Hoeven	$O(n \log n)$	Multi-dim NTT; Nussbaumer kills the recursion

Each breakthrough changed the kind of mathematics being used. Karatsuba’s was an algebraic identity. Toom-Cook was polynomial algebra. Schönhage-Strassen was number-theoretic harmonic analysis. Harvey-van der Hoeven is multi-dimensional algebraic geometry fused with analytic number theory.

And yet the punchline is the same as it was in the schoolbook algorithm: we are still just multiplying digits together and adding up the results. Every advance has been about finding a cleverer order in which to do it.

Connections to Sorting Algorithms

We have just traced the 60-year journey from $O(n^2)$ to $O(n \log n)$ for integer multiplication. But if you have studied algorithms before, that destination – $O(n \log n)$ – should feel familiar. It is the same complexity that governs comparison-based sorting: Merge Sort, Heapsort, and any optimal sorting algorithm all run in $\Theta(n \log n)$ time.

Is this a coincidence? It is not. The connection runs deep, and understanding it illuminates why $n \log n$ appears as a fundamental barrier across seemingly unrelated problems.

The Information-Theoretic Lower Bound

The $n \log n$ term does not arise from any particular algorithmic trick. It arises from entropy – from the sheer amount of information that must be processed to solve the problem at all.

Sorting: Counting Permutations

Consider sorting $n$ elements. The input is some unknown permutation of ${1, 2, \ldots, n}$, and the algorithm must determine which of the $n!$ possible permutations it is looking at. Every comparison-based sorting algorithm can be modeled as a binary decision tree: at each internal node, the algorithm compares two elements and branches left or right. The leaves of this tree correspond to the $n!$ possible outcomes.

A binary tree with $L$ leaves has depth at least $\log_2 L$. Since our tree must have at least $n!$ leaves:

\[\text{depth} \;\geq\; \log_2(n!)\]

Stirling’s approximation gives us:

\[\log_2(n!) \;=\; \sum_{k=1}^{n} \log_2 k \;=\; n \log_2 n - n \log_2 e + O(\log n) \;=\; \Theta(n \log n)\]

Each comparison provides at most 1 bit of information (left or right). Therefore, any comparison-based sorting algorithm must make at least $\Omega(n \log n)$ comparisons in the worst case. This is not a statement about any particular algorithm – it is a statement about the information content of the problem itself.

Multiplication: Counting Convolutions

Now consider multiplying two $n$-bit integers. The output is a $2n$-bit number, and each output bit can depend on every input bit from both operands. The “mixing” that must occur – the convolution of $n$ digits with $n$ digits – produces $2n - 1$ output coefficients, each of which is a sum of products involving up to $n$ terms.

The information-theoretic argument here is more subtle than in sorting (and, crucially, a tight $\Omega(n \log n)$ lower bound for integer multiplication has not been formally proven – this remains a major open problem). But the heuristic reasoning is compelling: the FFT is the canonical way to compute a length-$n$ convolution, and the FFT requires $\frac{n}{2} \log_2 n$ butterfly operations, each combining two values. The data must pass through $\log_2 n$ stages, with $O(n)$ work per stage. Any algorithm that computes the same convolution must, in some sense, perform the same total information routing.

The structural parallel is striking:

	Sorting	Multiplication
Problem	Identify one permutation out of $n!$	Compute a convolution of $n$ coefficients
Information content	$\log_2(n!) = \Theta(n \log n)$ bits	$\Theta(n \log n)$ bits of mixing (conjectured)
Per-step bandwidth	$O(n)$ comparisons	$O(n)$ butterfly operations
Minimum steps	$\Omega(\log n)$	$\Omega(\log n)$
Total work	$\Omega(n \log n)$	$O(n \log n)$ (achieved by HvH)

Both problems hit the same wall: $n$ data items that must be “mixed” through $\log n$ stages of $O(n)$ work each.

The Symmetry of the Recurrence

Beyond the information-theoretic parallel, there is an algebraic one: both optimal sorting and optimal multiplication satisfy the same recurrence relation.

Merge Sort

Merge Sort divides $n$ elements into two halves, recursively sorts each half, and merges the results in $O(n)$ time:

\[T(n) = 2\,T\!\left(\frac{n}{2}\right) + O(n)\]

By the Master Theorem (Case 2: $a = 2$, $b = 2$, $f(n) = O(n)$, so $n^{\log_b a} = n^1 = n = f(n)$), this solves to:

\[T(n) = O(n \log n)\]

Harvey-van der Hoeven Multiplication

The HvH algorithm decomposes an $n$-coefficient convolution into $K$ sub-convolutions of size $n/K$ each, with $O(n)$ work for the FFT and pointwise operations at each level:

\[T(n) = K \cdot T\!\left(\frac{n}{K}\right) + O(n)\]

This is the same Master Theorem case regardless of $K$: we have $a = K$, $b = K$, $n^{\log_b a} = n^1 = n = f(n)$, giving:

\[T(n) = O(n \log n)\]

The recurrences are structurally identical. In both cases, the algorithm splits the problem into $K$ pieces of size $n/K$ and does $O(n)$ work to split and recombine. The $\log n$ factor is simply the depth of the recursion tree: $\log_K n$ levels, each costing $O(n)$. The particular value of $K$ (2 for Merge Sort, $\sqrt{n}$ for Schönhage-Strassen, a carefully chosen composite for HvH) affects the constant factors but not the asymptotic complexity.

The critical difference is that Schönhage-Strassen’s “recombine” step secretly contains additional recursive multiplications – the pointwise products in $\mathbb{Z}/(2^m + 1)$ – which add a $\log \log n$ overhead atop the clean recurrence. Harvey and van der Hoeven’s achievement was making the recombine step truly $O(n)$ by using Nussbaumer’s technique to eliminate those inner multiplications. Once they did, the recurrence collapsed to the same clean form as Merge Sort, and $O(n \log n)$ fell out immediately.

The Hypercube: A Shared Geometry

The deepest way to see the connection is through the lens of communication on a hypercube – a model that unifies both algorithms geometrically.

What is a Hypercube?

A $d$-dimensional hypercube $Q_d$ is a graph with $2^d$ nodes. Each node is labeled by a $d$-bit binary string, and two nodes are connected by an edge if and only if their labels differ in exactly one bit. For example, $Q_3$ (the 3-cube) is the familiar wireframe cube with 8 vertices and 12 edges.

Three properties make the hypercube the natural “arena” for divide-and-conquer algorithms:

Recursive decomposition. $Q_d$ consists of two copies of $Q_{d-1}$ joined by edges along the $d$-th coordinate. This mirrors the way both Merge Sort and FFT-based multiplication split their input in half (or into $K$ parts) at each level.
Logarithmic diameter. Despite having $N = 2^d$ nodes, the longest shortest path in $Q_d$ has length $d = \log_2 N$. Any piece of information can reach any other node in at most $\log N$ steps. This is the geometric origin of the $\log n$ factor.
Vertex symmetry. Every node in $Q_d$ looks structurally identical to every other node – there are no bottlenecks, no privileged positions. This ensures that the algorithm’s workload is evenly distributed across all data items.

Sorting on the Hypercube

To sort $n$ elements on a hypercube, we can use Bitonic Sort (Batcher, 1968). The idea is to treat the $n$ array positions as the $n$ nodes of a hypercube ($d = \log_2 n$ dimensions) and perform compare-exchange operations along each dimension sequentially.

In a single step along one dimension, the hypercube supports $n/2$ parallel comparisons – one across each edge in that dimension. This is the bandwidth of one dimension: $O(n)$ bits of information resolved per step.

There are $d = \log_2 n$ dimensions, and each dimension requires $O(d)$ compare-exchange rounds in Bitonic Sort, giving $O(\log^2 n)$ parallel steps. (An asymptotically optimal sorting network, like the AKS network, achieves $O(\log n)$ parallel steps, fully saturating the hypercube’s bandwidth.)

The total work is:

\[\underbrace{O(n)}_{\text{comparisons per step}} \;\times\; \underbrace{O(\log n)}_{\text{steps}} \;=\; O(n \log n)\]

The $n \log n$ is precisely the entropy of $n!$ permutations being drained at a rate of $O(n)$ bits per step over $O(\log n)$ steps. When every lane of the hypercube is fully utilized at every step, we say the algorithm saturates the bandwidth. No sorting algorithm can do better, because there is no more information capacity to exploit.

Multiplication on the Hypercube

Now consider the FFT butterfly network for an $n$-point transform. Draw it as a diagram with $\log_2 n$ stages, each consisting of $n/2$ butterfly operations. If you squint at this diagram, you will notice something: it is a hypercube. Each butterfly connects two nodes whose indices differ in exactly one bit position (the bit corresponding to that stage). The FFT literally routes data along the edges of $Q_d$.

In Schönhage-Strassen, this hypercube routing is not quite “clean.” The pointwise multiplications at each stage are themselves recursive FFTs on smaller hypercubes, creating a nested hierarchy. The nesting depth is $\log \log n$, and the overhead at each level prevents the algorithm from fully saturating the bandwidth of the outer hypercube. There is “congestion” – some lanes carry recursive sub-computations instead of direct data movement.

Harvey and van der Hoeven’s breakthrough was, in geometric terms, a way to eliminate the congestion. By lifting the one-dimensional convolution into a multi-dimensional array and choosing the dimensions so that Nussbaumer’s technique replaces the recursive multiplications with additions and cyclic shifts, they ensured that the FFT butterfly at each level performs pure data routing – no hidden recursive work, no congestion. Every lane of the hypercube carries useful information at every step.

The result: multiplication, like sorting, saturates the hypercube’s bandwidth. Both algorithms move $O(n)$ units of information per step across $O(\log n)$ steps, for a total of $O(n \log n)$. The two problems – one about ordering elements, the other about convolving digits – are solved by the same geometric machine.

Sorting Is Multiplying. Multiplying Is Sorting.

We can now say something stronger than “sorting and multiplication are analogous.” They are, in a precise algebraic sense, the same operation – projections of a single underlying phenomenon onto different mathematical surfaces.

The Shared Abstraction: Rearranging Information Across Dimensions

Consider what sorting actually does. You have $n$ data items, each sitting at some position. The items are “tangled” – their current positions bear no relation to their values. To sort is to untangle them: to route each item from its current position to its correct position. The difficulty of sorting is exactly the difficulty of this routing problem. The items must move through a network (comparisons, swaps), and the network has finite bandwidth. The entropy $\log_2(n!)$ measures the total amount of routing information that must be resolved. The $n \log n$ is the cost of pushing that information through the $\log n$ dimensions of a hypercube, one dimension at a time.

Now consider what multiplication does. You have $n$ coefficients in one polynomial and $n$ coefficients in another. Each output coefficient of the product is a sum of terms $a_i \cdot b_j$ where $i + j$ equals the output index. The coefficients are “tangled” – every input coefficient contributes to multiple output coefficients, and the contributions overlap. To multiply is to untangle them: to route each partial product $a_i \cdot b_j$ to its correct output position $i + j$ and accumulate the results. The FFT does this by decomposing the routing into $\log n$ stages of butterfly operations, each resolving one “dimension” of the entanglement.

Strip away the surface details – the comparisons, the twiddle factors, the carry propagation – and you are left with the same skeleton:

$n$ items are entangled across $\log n$ dimensions. Untangling them requires touching all $n$ items once per dimension. The cost is $n \log n$.

Sorting untangles order. Multiplication untangles convolution. But “untangling” is the same verb in both sentences, and $n \log n$ is its conjugation.

Making It Algebraic

We can make this even more precise. Both problems can be formulated as applying a linear transform to a vector of length $n$:

Sorting is the application of a permutation matrix $P \in {0,1}^{n \times n}$ to the input vector. The “problem” is that we don’t know which permutation matrix to apply until we’ve inspected the data. The decision tree that determines $P$ has depth $\Omega(\log_2(n!)) = \Omega(n \log n)$.
Multiplication (via convolution) is the application of a circulant matrix $C \in \mathbb{Z}^{n \times n}$ to the input vector, where $C_{ij} = b_{(i-j) \bmod n}$. The FFT diagonalizes this circulant: $C = F^{-1} \hat{C} F$, where $F$ is the DFT matrix and $\hat{C}$ is diagonal. The cost of applying $F$ and $F^{-1}$ is $O(n \log n)$.

In both cases, we are applying a structured matrix to a vector of $n$ elements. The permutation matrix has $n!$ possible forms; the circulant matrix has $n$ free parameters but acts on a pair of inputs, yielding $O(n)$ degrees of freedom in the output. The algebraic structure of these matrices – their factorization into sparse stages, their decomposition along the dimensions of a hypercube – is identical. The DFT matrix $F$ factors into $\log n$ stages of sparse butterfly matrices. A sorting network factors into $\log n$ stages of sparse comparison-swap matrices. Both are “hypercube decompositions” of their respective transforms.

This is why the same $n \log n$ appears. It is not a coincidence, not a vague analogy, not a metaphor. It is the same theorem applied to two different matrix families.

The Linearithmic Manifold

There is a beautiful way to see all of this in a single picture. Imagine a space of all “rearrangement problems” on $n$ items – every computational task whose essence is routing $n$ pieces of information to their correct destinations through a network of finite bandwidth. Call this the linearithmic manifold: the space of problems whose optimal solutions require $\Theta(n \log n)$ operations.

Sorting lives on this manifold: $n$ items, $n!$ possible destinations, $\log n$ dimensions of routing.

Multiplication lives on this manifold: $n$ coefficients, $2n - 1$ output positions, $\log n$ dimensions of spectral decomposition.

The Fast Fourier Transform lives on this manifold: $n$ time-domain samples, $n$ frequency-domain samples, $\log n$ butterfly stages.

Even matrix transposition on a $\sqrt{n} \times \sqrt{n}$ matrix lives on this manifold: $n$ entries, each needing to swap its row and column coordinates, requiring $\Theta(n \log n)$ cache misses in the I/O model.

These problems are not merely “similar in complexity.” They are manifestations of the same geometric constraint: the constraint that moving $n$ items across $\log n$ dimensions, with $O(n)$ bandwidth per dimension, costs exactly $n \log n$. The linearithmic manifold is not a metaphor – it is the deep reason that $n \log n$ recurs so obsessively across computer science.

Sorting is multiplying in the permutation dimension. Multiplying is sorting in the convolution dimension. They are the same mountain, seen from different valleys.

Conclusion: The Structure of Computation Itself

What We’ve Witnessed

Let us step back and take in the full panorama.

We began with the schoolbook algorithm – a method so natural, so seemingly inevitable, that Andrey Kolmogorov, one of the greatest mathematicians of the twentieth century, stood before his seminar in 1960 and conjectured that its $O(n^2)$ cost was a law of nature. Multiplication, he believed, was inherently quadratic. Every digit of one number must “see” every digit of the other, and there is no shortcut around that combinatorial explosion.

He was wrong within a week.

What Karatsuba discovered was not merely a faster algorithm. It was a philosophical rupture. The schoolbook method treats multiplication as a flat, two-dimensional grid: row meets column, partial product accumulates, carry propagates. Karatsuba’s trick – computing three half-size products where four seemed necessary – revealed that this grid was not a law of arithmetic but an artifact of how we happened to organize the computation. The digits still meet. The partial products still accumulate. But by choosing a cleverer grouping, by exploiting the algebraic identity $(x_1 + x_0)(y_1 + y_0) = x_1 y_1 + x_1 y_0 + x_0 y_1 + x_0 y_0$, Karatsuba showed that some of those meetings are redundant – their information content is already captured elsewhere.

This is the thread that runs through everything we’ve seen.

The Changing Language of Speedup

Each breakthrough in this story didn’t just produce a faster algorithm; it changed the mathematical language in which multiplication is expressed.

Karatsuba spoke the language of algebra: a single polynomial identity turns four sub-problems into three, and the Master Theorem does the rest. The savings are modest – $O(n^{1.585})$ versus $O(n^2)$ – but the conceptual leap is enormous. For the first time, the exponent on multiplication was negotiable.

Toom-Cook generalized this into the language of polynomial interpolation. If splitting a number in two and evaluating at three points (Karatsuba) drops the exponent to $\log_2 3$, then splitting into $k$ pieces and evaluating at $2k - 1$ points drops it to $\log_k (2k - 1)$. As $k$ grows, the exponent approaches $1 + \varepsilon$ for any $\varepsilon > 0$. This was the first hint that $O(n^{1+\varepsilon})$ was not the floor – that perhaps the true cost of multiplication is not polynomial in $n$ at all, but something closer to $n$ times a slowly growing function.

Schönhage-Strassen rewrote multiplication in the language of harmonic analysis. The Convolution Theorem – the deep fact that convolution in the time domain becomes pointwise multiplication in the frequency domain – transforms the entire problem. Instead of asking “how do I combine digits?”, we ask “how do I move between representations of a polynomial?” The Fast Fourier Transform answers that question in $O(n \log n)$ time. But the need for exact integer arithmetic, not floating-point approximations, forced Schönhage and Strassen into the ring $\mathbb{Z}/(2^m + 1)\mathbb{Z}$, where roots of unity are powers of two and “multiplication by a twiddle factor” is just a bit-shift. The cost of this exactness was a recursive structure whose depth – $\log \log n$ levels – contributed a stubborn extra factor.

Harvey and van der Hoeven completed the journey by fusing multi-dimensional algebraic geometry with analytic number theory. Their insight was architectural: by lifting the one-dimensional convolution into a multi-dimensional array, choosing dimensions via Gaussian integers in $\mathbb{Z}[i]$ to ensure coprimality, and using Nussbaumer’s trick to eliminate the innermost recursive multiplications entirely, they flattened the recursive chain from $\log \log n$ levels to a constant. The $O(n \log n)$ barrier – conjectured for decades, tantalizingly close since 1971 – was finally reached.

The mathematical toolkit escalated at every stage: from high-school algebra, to polynomial interpolation, to Fourier analysis over finite rings, to algebraic geometry over Gaussian integers. And yet – and this is the remarkable part – the problem never changed. At every stage, we are still multiplying two numbers. We are still computing the same convolution of digits, still propagating the same carries. What changed is our understanding of the geometry of the computation: the realization that there exist clever rearrangements of the same arithmetic operations that cancel redundancies invisible from the schoolbook perspective.

The Information-Theoretic Floor

Is $O(n \log n)$ truly optimal? We cannot prove it is, but there are strong reasons to believe it.

Multiplying two $n$-bit numbers produces a $2n$-bit result. Each bit of the output can depend on every bit of both inputs. The information that must flow through the computation – the “mixing” of input bits into output bits – is at least $\Omega(n \log n)$ by plausible circuit-complexity arguments, though a formal proof remains one of the great open problems in theoretical computer science.

What we can say is that multiplication has reached the same complexity class as sorting: $\Theta(n \log n)$. And as we showed in the previous section, this is not a coincidence – it is the same theorem. Both problems live on what we called the linearithmic manifold: the space of computational tasks whose essence is routing $n$ pieces of information across $\log n$ dimensions of a finite-bandwidth network. Sorting routes elements to their correct positions in the permutation dimension. Multiplication routes partial products to their correct positions in the convolution dimension. The $n \log n$ is not an accident of algorithmic cleverness – it is the price the universe charges for untangling $n$ things, regardless of what those things are or why they are tangled.

Galactic Algorithms and the Nature of Truth

We should be honest about the practical situation. Karatsuba’s algorithm overtakes schoolbook multiplication at around 20-40 digits, depending on the implementation. Toom-Cook-3 wins around 100-200 digits. Schönhage-Strassen becomes competitive at tens of thousands of digits – the scale used by modern big-integer libraries like GMP. But Harvey-van der Hoeven? Its crossover point is somewhere beyond $2^{1729^{12}}$ digits. That number has more digits than there are atoms in the observable universe. No computer that will ever exist will multiply numbers that large. The Harvey-van der Hoeven algorithm will never be run.

And yet it matters enormously.

It matters because mathematics is not engineering. The question “what is the true complexity of integer multiplication?” is a question about the structure of arithmetic itself, not about what silicon can compute before the heat death of the universe. The existence of an $O(n \log n)$ algorithm – even one that is galactically impractical – tells us something true about the nature of numbers: that the information required to multiply them is not quadratic, not $n^{1.585}$, not $n \log n \log \log n$, but $n \log n$. That is a theorem about reality, not a software optimization.

Galactic algorithms are the astronomer’s telescope pointed at the foundations of computation. We will never travel to a quasar, but knowing it exists changes our understanding of the universe. Similarly, we will never run Harvey-van der Hoeven on actual inputs, but knowing it exists changes our understanding of what multiplication is.

The Punchline

Here is the thought to carry away.

Kolmogorov looked at schoolbook multiplication and saw $n^2$ – a grid of partial products, rigid and inescapable. Karatsuba looked at the same grid and saw that some cells were redundant. Toom and Cook saw that the grid was really a polynomial, and polynomials can be evaluated at fewer points than their degree suggests. Schönhage and Strassen saw that the polynomial was really a signal, and signals can be decomposed into frequencies. Harvey and van der Hoeven saw that the signal lived in a multi-dimensional space whose geometry could be exploited to eliminate every last bit of overhead.

Each generation looked at the same object – the product of two integers – and saw deeper structure. The number didn’t change. Our eyes did.

And when, at the end of this journey, we looked sideways and noticed that sorting – a completely different problem about ordering, not arithmetic – lands at exactly the same $n \log n$ cost for exactly the same geometric reasons, we glimpsed something deeper still. Sorting is multiplying in the permutation dimension. Multiplying is sorting in the convolution dimension. They are two faces of a single truth: that rearranging $n$ things across $\log n$ dimensions of entanglement costs $n \log n$, and not a single operation less.

That, in the end, is what mathematics is: the systematic refinement of vision. The history of multiplication algorithms is not a story about making computers faster. It is a story about learning to see – and discovering, at the bottom, that everything we were looking at was the same thing all along.

Shtetl Length

2025-06-21T00:00:00+00:00

수고하세요!

수고하세요 … what a wonderful phrase, it means work-hard, 화이팅 for the rest of your days!

A casual farewell rooted in Korean work culture, the upper politeness register manifests as

수고하셨어요, used as:

Cashier is a colleague ~ “great job working a full day today + goodbye”
Cashier has done great effort ~ “great effort!”

In that case, you (as the customer) could say 수고하셨습니다 as a way to say thanks (a gesture of gratitude).

So, working hard is as common as invoking Catholic Jesus in Latin culture?

Why, yes! You’re catching on!

Shtetl Optimizations of the Umbral Calculi

In mathematics, the umbral calculus is a technique where you pretend that subscript indices are exponents. Umbra is Latin for “shadow.” The method is, on its face, absurd – and yet it works.

The classic example: the Bernoulli polynomials. The ordinary binomial expansion gives you

\[(y + x)^n = \sum_{k=0}^{n} \binom{n}{k} y^{n-k} x^k\]

and the Bernoulli polynomials satisfy a remarkably similar identity:

\[B_n(y + x) = \sum_{k=0}^{n} \binom{n}{k} B_{n-k}(y) \, x^k\]

The umbral move? Pretend the subscript in $B_{n-k}$ is an exponent $b^{n-k}$, so that $B_n(x) = (b + x)^n$. Differentiate that, and you get $B_n’(x) = n(b+x)^{n-1} = nB_{n-1}(x)$ – the correct result, derived by treating a shadow as the real thing. John Blissard introduced this in 1861; Gian-Carlo Rota made it rigorous a century later by defining a linear functional $L$ such that $L(z^n) = B_n$, explaining why the shadow-trick works.

What does this have to do with blogs? Consider a blog’s readability metrics as shadows of the writing itself. A Flesch-Kincaid score is not the prose, just as a subscript is not an exponent. And yet, by treating these shadows as if they were the real thing – by pretending indices are exponents – we can derive surprising identities between blogs that otherwise look nothing alike. The Gunning Fog index of a quantum computing post and a personal finance post might converge, despite the posts sharing nothing in vocabulary or intent. The shadow knows something the text doesn’t say directly.

Scott Aaronson’s blog, Shtetl Optimized, is a national treasure. The man wittily demagogues on science, life, politics, and vagina dentata with equal aplomb. The purpose of this post was to scrutinize the shape and form of a shtetl (optimized blog), starting with a distance metric.

Linguistic Complexity: Blog vs. Blog

Here, I analyze the linguistic complexity of Aaronson’s blog compared to my own and two others: the also great Simon Willison blog along with the ineffable Alex Harri blog.

Source	Flesch-Kincaid Grade	ARI Grade	Gunning Fog Grade	Lexical Diversity	Word Count
alexharri_posts.csv	9.77	9.55	12.40	0.228	3,866.5
juleshenry_posts.csv	15.51	16.31	17.78	0.458	2,000.4
scottaaronson_blog_posts.csv	13.01	13.31	15.61	0.543	966.8
simonwillison_all_blogs.csv	12.13	12.55	14.52	0.554	627.8

Writing Stats Over Time

The curiosity? Technical posts register as more sophisticated when code is used because code has no “periods”, qualifying as long Faulknerian sentences. Note the outliers in my own blog achieve Flesch-Kincaid of 80+. Therefore, I also include an outliers-removed summary to truly compare. Alex Harri, an Icelander, unsurprisingly writes at a lower grade level, even though his posts are incredibly engaging and informative. Simon Willison’s posts are more accessible, while Scott Aaronson’s are more complex, likely due to their technical density.

Interquartile Reduction of Outliers

Source	Flesch-Kincaid Grade	ARI Grade	Gunning Fog Grade	Lexical Diversity	Word Count
alexharri_posts.csv	9.86	9.64	12.49	0.234	3,443.41
juleshenry_posts.csv	11.50	11.33	13.58	0.500	1,110.80
scottaaronson_blog_posts.csv	12.77	13.01	15.35	0.559	702.75
simonwillison_all_blogs.csv	11.80	12.12	14.21	0.568	482.81

A ha! So I do not write at a higher level than Scott or Simon, although I tend to be more verbose.

Writing Stats Over Time Without Outliers

That this analysis was done on Valentine’s Day is perhaps a data point in itself. Scott, solve NP vs. P when you have the chance, will ya?

Code analysis repository found here.

Back-of-the-Envelope: Primitive Sets

2025-06-08T00:00:00+00:00

The Joyous Structure of the Nature of Primes (Primitive Sets Theorem Proved!)

Reading the conjecture for the first time in Covid-19 locked-down Boston, I was sure it was true. There was a sense of deep vindication when I soon came across the proof of this conjecture by Erdős, years later, by a bright PhD student named Jared Duker Lichtman. This statement about the nature of primes was so beautiful it surely had to be true.

The original paper can be found here.

Herein we go into the thick of things – heavy-handedly, every derivation spelled out. If you’ve taken a semester of calculus and have a taste for beautiful mathematics, you have everything you need. Keep reading.

The Crash Course (Skip If You Know This Stuff)

I won’t belabor fundamentals, but we need a common language. Three things.

Divisibility. We write $a \mid b$ when $b = ka$ for some integer $k$. So $3 \mid 12$ because $12 = 4 \cdot 3$, but $5 \nmid 12$. The Fundamental Theorem of Arithmetic says every integer $n > 1$ factors uniquely into primes:

\[n = p_1^{e_1} \cdot p_2^{e_2} \cdots p_r^{e_r}, \quad p_1 < p_2 < \cdots < p_r.\]

For $360 = 2^3 \cdot 3^2 \cdot 5$, the smallest prime factor is $p(360) = 2$, the largest is $P(360) = 5$, and the total count of prime factors (with repeats) is $\Omega(360) = 3 + 2 + 1 = 6$. We write $\mathbb{N}_k$ for the set of integers with exactly $k$ prime factors: the primes are $\mathbb{N}_1$, the semiprimes ${4, 6, 9, 10, 14, 15, \ldots}$ are $\mathbb{N}_2$, and so on.

Series convergence. The harmonic series $\sum 1/n$ diverges – you can see this by grouping terms into blocks of $2^k$, each contributing at least $1/2$. But $\sum 1/n^2 = \pi^2/6$ converges. The integral test is our workhorse: $\sum g(n)$ converges iff $\int g(x)\,dx$ converges. Two key facts we’ll use:

\[\int_2^{\infty} \frac{dx}{x \ln x} = \Big[\ln(\ln x)\Big]_2^{\infty} = \infty \quad \text{(diverges!)}\] \[\int_2^{\infty} \frac{dx}{x (\ln x)^2} = \left[-\frac{1}{\ln x}\right]_2^{\infty} = \frac{1}{\ln 2} \quad \text{(converges!)}\]

So $\sum 1/(n \log n)$ diverges – the extra $\log n$ isn’t enough to tame the harmonic series. But $\sum 1/(n (\log n)^2)$ converges. The sum over primes we care about will land between these two. Keep that in mind.

The Euler-Mascheroni constant. Define $\gamma$ as the gap between the harmonic sum and the natural log:

\[\gamma = \lim_{N \to \infty} \left(\sum_{n=1}^{N} \frac{1}{n} - \ln N\right) = 0.5772\ldots\]

The number $e^{\gamma} = 1.7810\ldots$ will haunt us. It appears whenever you convert between discrete sums over primes and continuous logarithmic integrals – Mertens figured this out in 1874 and we owe him big time.

One last tool: partial summation (Abel’s summation formula). This is integration by parts for sums. If $A(x) = \sum_{n \leq x} a_n$ and $g$ is differentiable:

\[\sum_{n \leq x} a_n \, g(n) = A(x)\,g(x) - \int_1^x A(t)\,g'(t)\,dt.\]

This is how you turn knowledge about counting functions (like $\pi(x) \sim x/\ln x$, the Prime Number Theorem) into knowledge about sums over primes. We will use it relentlessly.

Primitive Sets: The Main Character

Definition. A set $A \subset \mathbb{Z}_{>1}$ is primitive if no element of $A$ divides another element of $A$.

That’s it. That’s the whole definition. In the language of posets: $A$ is an antichain in the divisibility ordering of the integers.

Some examples to build your intuition:

The primes ${2, 3, 5, 7, 11, \ldots}$ – primitive. No prime divides another prime. This is the protagonist of our story.

${6, 10, 15}$ – primitive. Check all pairs: $6 \nmid 10$, $6 \nmid 15$, $10 \nmid 15$, and the reverses. Nothing divides anything else.

${6, 12, 15}$ – NOT primitive. Because $6 \mid 12$. Busted.

Any dyadic interval $(x, 2x]$ – primitive. If $a, b$ are both in $(x, 2x]$ and $a \mid b$, then $b \geq 2a > 2x$. Contradiction.

$\mathbb{N}_k$, the numbers with exactly $k$ prime factors – primitive. If $a \mid b$ and $a \neq b$, then $b = ac$ for some $c > 1$, so $\Omega(b) = \Omega(a) + \Omega(c) > k$. So two numbers with the same $\Omega$ can’t divide each other.

The perfect numbers ${6, 28, 496, 8128, \ldots}$ – primitive. (It’s a known theorem that no perfect number divides another. My God, even perfect numbers play along.)

Think of it visually: the positive integers form a poset under divisibility, like a directed graph where $a$ points up to $b$ if $a \mid b$. A primitive set is a horizontal slice – nodes with no vertical connections between them.

graph BT
    2 --> 4
    2 --> 6
    2 --> 10
    3 --> 6
    3 --> 9
    3 --> 15
    5 --> 10
    5 --> 15
    4 --> 12
    6 --> 12
    style 2 fill:#38bdf8,color:#000
    style 3 fill:#38bdf8,color:#000
    style 5 fill:#38bdf8,color:#000
    style 7 fill:#38bdf8,color:#000
    style 6 fill:#a855f7,color:#fff
    style 10 fill:#a855f7,color:#fff
    style 15 fill:#a855f7,color:#fff

Blue nodes ${2,3,5,7}$: one primitive set (the small primes). Purple nodes ${6, 10, 15}$: another. You can’t pick both $6$ and $12$ – there’s an arrow between them.

Why do we care? These things emerged in the 1930s from a concrete problem: Davenport proved the abundant numbers (where the sum of proper divisors exceeds $n$) have positive density. Erdős found an elegant shortcut using primitive abundant numbers – the ones that are minimal under divisibility. The abstraction turned out to be more interesting than the application. Classic Erdős.

The Erdős Sum: Why $\frac{1}{a \log a}$?

Here’s where the magic starts. For a set $A \subset \mathbb{Z}_{>1}$, define:

\[\boxed{f(A) = \sum_{a \in A} \frac{1}{a \log a}}\]

This weighting is not arbitrary. Let me show you where it comes from.

For any integer $a \geq 2$:

\[\frac{1}{a \log a} = \int_1^{\infty} a^{-t}\,dt.\]

Full derivation. Start:

\[\int_1^{\infty} a^{-t}\,dt = \int_1^{\infty} e^{-t \ln a}\,dt.\]

Substitute $u = t \ln a$, so $du = \ln a\,dt$:

\[= \frac{1}{\ln a}\int_{\ln a}^{\infty} e^{-u}\,du = \frac{1}{\ln a}\Big[-e^{-u}\Big]_{\ln a}^{\infty} = \frac{1}{\ln a}\cdot e^{-\ln a} = \frac{1}{a \ln a}. \quad \blacksquare\]

So $1/(a \log a)$ is what you get when you integrate $a^{-t}$ from $1$ to $\infty$. Summing over $A$ and swapping sum and integral (Tonelli says we can):

\[f(A) = \sum_{a \in A} \int_1^{\infty} a^{-t}\,dt = \int_1^{\infty} \underbrace{\left(\sum_{a \in A} a^{-t}\right)}_{f_t(A)}\,dt.\]

This integral representation – $f(A) = \int_1^{\infty} f_t(A)\,dt$ – is powerful. It turns bounding $f(A)$ into bounding a family of Dirichlet-series-like sums for each $t > 1$.

Now, the sum over primes:

\[f(\mathcal{P}) = \sum_p \frac{1}{p \log p} = \frac{1}{2 \log 2} + \frac{1}{3 \log 3} + \frac{1}{5 \log 5} + \cdots\]

$p$	$1/(p \log p)$	Running sum
2	0.7213	0.7213
3	0.3034	1.0247
5	0.1243	1.1490
7	0.0734	1.2224
11	0.0379	1.2603
13	0.0300	1.2903
17	0.0208	1.3111
19	0.0179	1.3290

Converges slowly. After summing all primes to $10^8$, Cohen computed $f(\mathcal{P}) = 1.6366\ldots$ We’ll prove convergence rigorously later. For now: it’s a finite number, roughly $1.6366$.

Erdős 1935: The Sum Is Bounded. For ANY Primitive Set.

Before you can ask “which primitive set maximizes $f$?”, you need to know the maximum exists. Erdős proved this in 1935, and the argument is gorgeous.

For each $a \geq 2$, define the L-multiples of $a$ (the “L” is for “lexicographic” – we couch the reason for this name for later):

\[L_a = \{b \cdot a : b \geq 1, \; p \mid b \Rightarrow p \geq P(a)\}.\]

In words: take $a$, and multiply it by any integer whose prime factors are all $\geq P(a)$ (the largest prime factor of $a$).

For $a = 6$ (where $P(6) = 3$): $L_6$ consists of $6$ times any number built from primes $\geq 3$. So $6 \cdot 1 = 6$, $6 \cdot 3 = 18$, $6 \cdot 5 = 30$, $6 \cdot 7 = 42$, $6 \cdot 9 = 54$, and so on.

For a prime $p$: $L_p$ consists of $p$ times any number built from primes $\geq p$. So $L_5 = {5, 25, 35, 55, 125, \ldots}$.

The density of $L_a$. How “big” is $L_a$ inside the integers? Its natural density is:

\[d(L_a) = \frac{1}{a}\prod_{p < P(a)}\left(1 - \frac{1}{p}\right).\]

The product $\prod_{p < P(a)}(1 - 1/p)$ is the proportion of integers not divisible by any prime less than $P(a)$ – that’s inclusion-exclusion, the sieve of Eratosthenes doing what it does best. Scaling by $1/a$ accounts for the $a$-multiple structure.

Now the engine of the whole argument:

Disjointness Lemma. If $A$ is primitive, then the sets ${L_a : a \in A}$ are pairwise disjoint.

Proof. Suppose $n \in L_a \cap L_{a’}$ for distinct $a, a’ \in A$. Write $n = ba = b’a’$ where $b$ has all prime factors $\geq P(a)$ and $b’$ has all prime factors $\geq P(a’)$. Assume WLOG $P(a) \leq P(a’)$.

Write $a = a^* \cdot P(a)^k$ where $a^$ has all prime factors $< P(a)$. Since $a \mid n = b’a’$ and $\gcd(a^, b’) = 1$ (their prime factors don’t overlap), we get $a^* \mid a’$. Similarly $P(a)^k \mid a’$ (since $P(a) < P(a’)$ means $P(a)$ can’t come from $b’$). Therefore $a = a^* \cdot P(a)^k \mid a’$.

But $a \mid a’$ contradicts primitivity. $\blacksquare$

Since disjoint subsets of $\mathbb{N}$ can’t have densities summing past $1$:

\[\sum_{a \in A} d(L_a) = \sum_{a \in A} \frac{1}{a}\prod_{p < P(a)}\left(1 - \frac{1}{p}\right) \leq 1.\]

Now invoke Mertens’ product theorem (1874): $\prod_{p \leq x}(1 - 1/p) \sim e^{-\gamma}/\log x$. Taking $x = P(a)$:

\[d(L_a) \approx \frac{e^{-\gamma}}{a \log P(a)}.\]

Since $P(a) \leq a$, we have $\log P(a) \leq \log a$, so $1/(a \log a) \leq 1/(a \log P(a))$, which means:

\[f(a) = \frac{1}{a \log a} \leq \frac{1}{a \log P(a)} \approx e^{\gamma} \cdot d(L_a).\]

Sum it up:

\[f(A) = \sum_{a \in A} f(a) \leq e^{\gamma}\sum_{a \in A} d(L_a) \leq e^{\gamma} \cdot 1 = e^{\gamma} \approx 1.781.\]

And we’re done.

Theorem (Erdős 1935, refined Lichtman-Pomerance 2019). For any primitive set $A$: $f(A) < e^{\gamma} = 1.781\ldots$

Beautiful. But notice: $e^{\gamma} \approx 1.781 > 1.637 \approx f(\mathcal{P})$. This bound is too loose by about $8.8\%$. It doesn’t prove the conjecture. We need a new idea.

The Conjecture: Primes Are Optimal

Conjecture (Erdős, ~1974). For any primitive set $A$, $f(A) \leq f(\mathcal{P}) = 1.6366\ldots$

In words: among all primitive sets, the primes have the largest Erdős sum. The most natural antichain in the divisibility poset is also the heaviest one.

Why should this be true? Three angles.

The primes are atoms. They sit at the bottom of the divisibility poset – nothing above $1$ divides them. Including a prime in your set costs you nothing in terms of divisibility conflicts.

Composites are expensive. The $f$-weight is front-loaded:

\[f(2) = 0.721, \quad f(3) = 0.303, \quad f(6) = 0.093.\]

If you include $6$ in your primitive set, you gain $0.093$ but lose both $2$ and $3$ (total loss: $1.024$). Terrible trade! You traded away the crown jewels for pocket change.

The primes are maximally spread out. Each prime is independent – no prime is a multiple of another. Composites create tangled webs of divisibility conflicts.

The conjecture sat open for decades:

1935: Erdős proves $f(A)$ is bounded.
1974: Conjecture appears in print.
1993: Erdős and Zhang prove $f(A) < 1.84$.
2019: Lichtman and Pomerance prove $f(A) < e^{\gamma} = 1.781$.
2022: Lichtman proves $f(A) \leq f(\mathcal{P})$. Conjecture proved!

The Proof Strategy: Split by Smallest Prime Factor

Every integer $a > 1$ has a smallest prime factor $p(a)$. Partition any set $A$ accordingly:

\[A = \bigsqcup_{p} A_p, \quad A_p = \{a \in A : p(a) = p\}.\]

Since the $A_p$ are disjoint: $f(A) = \sum_p f(A_p)$. The conjecture follows if for each prime $p$:

\[f(A_p) \leq f(p) = \frac{1}{p \log p}.\]

A prime satisfying this for all primitive $A$ is called Erdős-strong. If every prime is Erdős-strong, we win.

There’s a clean sufficient condition. By Mertens’ product theorem:

\[\prod_{q < p}\left(1 - \frac{1}{q}\right) \sim \frac{e^{-\gamma}}{\log p}.\]

A prime $p$ is Erdős-strong if:

\[e^{\gamma} \prod_{q < p}\left(1 - \frac{1}{q}\right) \leq \frac{1}{\log p}. \tag{$\star$}\]

Computationally, $(\star)$ holds for the first $10^8$ odd primes. But it fails for $p = 2$:

\[e^{\gamma} \cdot 1 = 1.781 > \frac{1}{\log 2} = 1.443.\]

The empty product equals $1$, and $e^{\gamma}$ overshoots $1/\log 2$. Oh no.

It gets worse. Under the Riemann Hypothesis, $(\star)$ fails for a positive proportion of primes. Even unconditionally, it fails for infinitely many. So the “every prime is Erdős-strong” strategy is doomed as a direct approach.

This is why the conjecture stayed open. This is where Lichtman had his breakthrough.

The Key Innovation: The $\sqrt{v}$ Bound

This is the heart of the proof. The idea is beautifully simple once you see it, and I remember staring at it for a long time before it clicked.

Recall our bound from the Erdős argument: we replaced $\log a$ with $\log P(a)$ and lost a factor of $\log P(a)/\log a$. When $a$ is prime, $P(a) = a$, so there’s no loss. When $a$ is composite, $P(a) < a$ and we lose something.

The key question: how much can the composites collectively contribute?

Parameterize the “closeness” of $P(a)$ to $a$. For $v \geq 0$, say that $a$ is “$v$-close” if $P(a)^{1+v} > a$. When $v = 0$, this means $P(a) > a$, which only primes satisfy. As $v$ grows, more composites qualify.

The savings factor for a $v$-close element is $\log P(a)/\log a > 1/(1+v)$.

Now here is where Lichtman goes beyond the 2019 argument.

Proposition (Lichtman). If $A$ is primitive and $P(a)^{1+v} > a$ for all $a \in A$, then $\sum_{a \in A} d(L_a) \leq \sqrt{v}$.

My word. That refines the trivial bound of $1$ whenever $v < 1$.

The idea: not only are the sets $L_a$ disjoint – so are many “copies” of them. Multiply each $a$ by various integers $c$ whose prime factors lie in a certain range, and the sets $L_{ac}$ remain disjoint from each other AND from all $L_{a’}$ for $a’ \neq a$. These copies boost the total density by a factor of $\sim 1/\sqrt{v}$. Since everything fits inside $\mathbb{N}$:

\[\frac{1}{\sqrt{v}} \cdot \sum_{a \in A} d(L_a) \leq 1 \quad \Longrightarrow \quad \sum_{a \in A} d(L_a) \leq \sqrt{v}.\]

The self-similarity of the $L$-sets is doing all the heavy lifting. You’re creating “shadows” of your original sets, proving the shadows are also disjoint, and using the shadow density to constrain the original. It’s a beautiful counting-over-counting argument.

The $\pi/4$ Factor: The Finishing Blow

Now we combine the $\sqrt{v}$ bound with the savings factor $1/(1+v)$.

Elements entering at parameter $v$ (those with $P(a)^{1+v} \approx a$) contribute to the density with a $\sqrt{v}$ budget, and each contributes to $f(A)$ with a savings of $\sim 1/(1+v)$ relative to its density weight.

The derivative of the density budget $\sqrt{v}$ is $1/(2\sqrt{v})$. So the worst-case contribution of composite elements at level $v$ is $\sim dv/(2\sqrt{v}(1+v))$. Integrate from $v = 0$ to $v = 1$:

\[I = \int_0^1 \frac{dv}{2\sqrt{v}(1+v)}.\]

Step 1. Substitute $u = \sqrt{v}$, so $v = u^2$, $dv = 2u\,du$:

\[I = \int_0^1 \frac{2u\,du}{2u(1+u^2)} = \int_0^1 \frac{du}{1+u^2}.\]

Step 2. Oh come on. We know this one.

\[\int_0^1 \frac{du}{1+u^2} = \Big[\arctan(u)\Big]_0^1 = \frac{\pi}{4}.\]

And there it is:

\[\boxed{\int_0^1 \frac{dv}{2\sqrt{v}(1+v)} = \frac{\pi}{4}}\]

$\pi$ shows up from the arctangent integral, which comes from the geometry of the unit circle. Number theory and geometry shaking hands through an optimization problem. My God.

So the composite elements contribute at most:

\[f(A_{\text{comp}}) \leq e^{\gamma} \cdot \frac{\pi}{4} \approx 1.781 \times 0.785 = 1.399.\]

And $1.399 < 1.637 = f(\mathcal{P})$. The composites can only hurt. Any primitive set with composite elements has $f(A) < f(\mathcal{P})$.

graph TD
    A["Primitive set A"] --> B{"Contains composites?"}
    B -->|"No: A ⊆ primes"| C["f(A) ≤ f(P) ✓"]
    B -->|"Yes"| D["Composites contribute ≤ e^γ · π/4 ≈ 1.399"]
    D --> E{"1.399 < f(P) = 1.637?"}
    E -->|"Yes!"| F["Composites only hurt → f(A) ≤ f(P) ✓"]
    style C fill:#38bdf8,color:#000
    style F fill:#38bdf8,color:#000

A note on $p = 2$. Lichtman proves every odd prime is Erdős-strong (Theorem 1.3 of the paper). Whether $p = 2$ is Erdős-strong remains open. But the conjecture doesn’t need it – the $\pi/4$ argument handles the global bound directly. The troublemaker prime gets outflanked.

The Convergence of $\sum_p 1/(p \log p)$

This is the other half of the story – proving the sum at the center of the conjecture actually converges. I’ll give you three proofs because frankly each one is satisfying in its own way.

Via the Prime Number Theorem and Partial Summation

The PNT gives us $\pi(x) \sim x/\ln x$. We won’t prove it here (the weeds be tall there – that’s a whole other post). But we’ll use it.

Set $g(t) = 1/(t \log t)$. By Abel’s summation formula:

\[\sum_{p \leq x} \frac{1}{p \log p} = \pi(x) \cdot g(x) - \int_2^x \pi(t) \cdot g'(t)\,dt.\]

Compute $g’(t)$:

\[g'(t) = -\frac{\log t + 1}{t^2 (\log t)^2}.\]

The boundary term: $\pi(x) \cdot g(x) \sim \frac{x}{\log x} \cdot \frac{1}{x \log x} = \frac{1}{(\log x)^2} \to 0$. Good – it vanishes.

The integral, using $\pi(t) \sim t/\log t$:

\[\int_2^x \frac{t}{\log t} \cdot \frac{\log t + 1}{t^2(\log t)^2}\,dt = \int_2^x \frac{\log t + 1}{t(\log t)^3}\,dt \leq \int_2^x \frac{2}{t(\log t)^2}\,dt\]

for large $t$ (since $\log t + 1 \leq 2\log t$). And:

\[\int_2^{\infty} \frac{dt}{t(\log t)^2} = \left[-\frac{1}{\log t}\right]_2^{\infty} = \frac{1}{\log 2} < \infty.\]

Converges. $\blacksquare$

The Quick-and-Dirty Comparison

By PNT, the $n$-th prime satisfies $p_n \sim n \ln n$. So:

\[\frac{1}{p_n \log p_n} \sim \frac{1}{n \ln n \cdot \ln(n \ln n)} \sim \frac{1}{n (\ln n)^2}.\]

The series $\sum 1/(n(\ln n)^2)$ converges by the integral test (we already showed $\int dx/(x(\ln x)^2) = 1/\ln 2$). By the limit comparison test, $\sum_p 1/(p \log p)$ converges too. $\blacksquare$

That’s the one you’d do on a napkin. Two lines.

Via Mertens’ First Theorem

Mertens proved $\sum_{p \leq x} (\log p)/p = \log x + O(1)$. Set $B(x) = \sum_{p \leq x} (\log p)/p$ and rewrite:

\[\sum_{p \leq x} \frac{1}{p \log p} = \sum_{p \leq x} \frac{1}{(\log p)^2} \cdot \frac{\log p}{p}.\]

Partial summation with $h(t) = 1/(\log t)^2$:

\[= \frac{B(x)}{(\log x)^2} + \int_2^x \frac{2B(t)}{t(\log t)^3}\,dt.\]

First term: $B(x)/(\log x)^2 \sim \log x/(\log x)^2 = 1/\log x \to 0$. The integral with $B(t) = \log t + O(1)$:

\[\int_2^{\infty} \frac{2(\log t + O(1))}{t(\log t)^3}\,dt = 2\int_2^{\infty} \frac{dt}{t(\log t)^2} + O\!\left(\int_2^{\infty} \frac{dt}{t(\log t)^3}\right).\]

Both converge. $\blacksquare$

Where Does This Sum Sit?

Let’s zoom out:

Series	Converges?	Value
$\sum_p 1/p^2$	Yes	$0.4522\ldots$
$\sum_p 1/(p \log p)$	Yes	$1.6366\ldots$
$\sum_p 1/p$	No	$\sim \log \log x$
$\sum_n 1/(n \log n)$	No	Diverges

The sum over primes of $1/(p \log p)$ converges because primes are sparse enough. Even though $\sum 1/(n \log n)$ diverges over all integers, primes have density $\sim 1/\log n$, so summing only over primes gives effective terms $\sim 1/(n(\log n)^2)$, and that converges. The extra $\log$ comes from the primes themselves being logarithmically sparse. A gift from the Prime Number Theorem.

The convergence is slow, though – the tail decays as $O(1/\log x)$:

Primes up to	Partial sum	Gap to $1.6366$
$10^2$	$1.4510$	$0.186$
$10^4$	$1.5710$	$0.066$
$10^6$	$1.6110$	$0.026$
$10^8$	$1.6276$	$0.009$

Each extra order of magnitude in the cutoff gains roughly the same amount. You have to go far to pin down the digits.

The Full Picture

Let me lay out the complete proof structure, from the onset, because the logical flow is what makes this proof sing.

graph TD
    subgraph "Ingredients"
        A["Primitivity<br/>(no a divides a')"] --> B["L-sets disjoint"]
        M["Mertens' product theorem<br/>Π(1-1/p) ~ e^{-γ}/log x"] --> D
    end

    subgraph "Old Bound (2019)"
        B --> C["Σ d(L_a) ≤ 1"]
        C --> D["f(A) ≤ e^γ ≈ 1.781"]
    end

    subgraph "New Ingredient (2022)"
        B --> F["Copies L_{ac} also disjoint"]
        F --> G["Σ d(L_a) ≤ √v"]
        G --> H["Savings 1/(1+v) × density √v"]
        H --> I["Integrate: ∫ dv/[2√v(1+v)] = π/4"]
        I --> K["f(composites) ≤ e^γ · π/4 ≈ 1.399"]
    end

    subgraph "QED"
        K --> L{"1.399 < 1.637 = f(primes)"}
        L -->|"YES"| N["f(A) ≤ f(P)<br/>CONJECTURE PROVED"]
    end

    style N fill:#38bdf8,color:#000,stroke:#38bdf8,stroke-width:3px

Define $f(A) = \sum 1/(a \log a)$.
Define L-multiples $L_a$. Prove they’re disjoint for primitive $A$.
Sum of densities $\leq 1$. By Mertens, this gives $f(A) \leq e^{\gamma} \approx 1.781$ (old bound).
New: if $P(a)^{1+v} > a$ uniformly, the density sum is $\leq \sqrt{v}$ (from the self-similarity of L-sets).
Balance the savings $1/(1+v)$ against the density budget $\sqrt{v}$.
Integrate: $\int_0^1 dv/(2\sqrt{v}(1+v)) = \pi/4$.
Composite contribution $\leq e^{\gamma} \pi/4 \approx 1.399 < 1.637 = f(\mathcal{P})$.
Primes win. $\blacksquare$

What’s Still Open

The proof opens doors.

Is $p = 2$ Erdős-strong? Every odd prime is. The smallest prime is the last holdout. There’s something poetic about that.

The Erdős-Sárközy-Szemerédi conjecture (1968): Does $\sup f(A) \to 1$ as we restrict to primitive sets $A \subset [x, \infty)$? Lichtman’s methods give $\leq e^{\gamma}\pi/4 \approx 1.399$ for this limit. The conjectured value of $1$ comes from the fact that $f(\mathbb{N}_k) \to 1$ as $k \to \infty$. Getting from $1.399$ down to $1$ is the next frontier.

Connections to the Riemann Hypothesis. The Erdős-strong criterion $(\star)$ is intimately related to the oscillation of $\pi(x)$ around $\text{li}(x)$ – the “prime number race” governed by zeros of the zeta function. Under RH and the Linear Independence Hypothesis, $99.999973\%$ of primes satisfy $(\star)$. The exceptional primes come from the Chebyshev bias. Perhaps, one day, understanding primitive sets will feed back into understanding zeta zeros. Perhaps.

Conclusion

I hope in sharing this you are able to see some beauty in this wonderful proof about the nature of the primes.

The Erdős primitive set conjecture says something profound: among all the infinite antichains in the divisibility poset, the most natural one – the primes, those fundamental atoms of arithmetic – is also the heaviest. And the proof turns on a surprise appearance of $\pi/4$ from an arctangent integral. Number theory, combinatorics, and calculus, all holding hands.

Perhaps, one day, this sort of work will contribute to a proof of the Riemann Hypothesis.

Will it be you, dear reader?

Hope so - :)

Cheers.

Julian Henry's Blog

touch_keeper: Mass-Personalized New Year's Texts

The Three Stages

1. Send

2. Serve

3. Analyze

The Response Log: A Time Capsule

Technical Notes

Why Twilio?

The Philosophical Bit

Hacker de Teclado: Acoustic Side-Channel Attack

The Pipeline

Stage 1: Data Collection

Stage 2: Training

Stage 3: The Attack

Why This Works

The Security Implications

The Swiss-Mexican Auction: Combinatorial Bidding

The Problem: Satisfiability and Expenditure

The Two Layers

The Swiss Layer (Constraints)

The Mexican Layer (Execution)

Results: 50,000 Items, 20,000 Bidders, 0.26 Seconds

The LP-Guided Hybrid

Two Emergent Phenomena

The Anti-Whale Effect

The Tequila-Snow Phase Transition

Future Horizons

git-goblin: 195+ Shortcuts and a Pure Bash TUI

The Greatest Hits

The G Command

Mode 1: Best-Guess

Mode 2: Add

Mode 3: Interactive TUI

Setup

The Python Utilities

Why Not Just Use Oh My Zsh?

README Rosetta: Open Source in Every Language

The Problem with LLM Translation of Structured Text

Code Block Protection

Hallucination Detection and Retry Logic

Translation Caching

Output Modes

Sphinx Integration

GitHub Actions

Usage

The Technical Core

Mr. Worldwide: Hello in 80 Languages as a GIF

The Hard Problem: Rendering Every Script on Earth

Smart Contrast Coloring

Flag-Color Text

The Full CLI

Design Decisions

SuperWand: Magic Wand for Image Retheming

How It Works

1. Region Identification

2. Theme Injection

3. Gradient Application

SuperWand Studio

CLI Usage

The Interesting Parts

ZIT: Zooplankton Image Tool

The Problem

Two Compositing Modes

Mode 1: Pixel Difference

Mode 2: Entity Recognition (The Good One)

The Parameter Sweep Grid

Usage

What I Learned

Language Notes Repos

한글 낙서: Hangul Graffiti

The Architecture of Hangul

Pronunciation: The Rules They Don’t Teach First

Grammar: Hierarchy in the Verb

Conjugation in Practice

The Particles: Small Words, Heavy Lifting

At the Gym: 헬스장에서

Reading Korean Literature: 선화 by 김이

수고하세요: The Farewell That Means “Work Hard”

The Korean Keyboard: 두벌식

The `G` Command