How we put text behind your head

Our most-copied feature is also our most misunderstood: captions that sit behind the person talking. Viewers read it as a After Effects flex; competitors assume it’s a segmentation call and a prayer. It’s neither — it’s an alpha matte computed twice, by two very different systems, feeding one compositor. This post is the actual architecture, including the parts that fell over.

The problem, stated precisely

To draw text behind a person you need, for every frame, an alpha matte: a grayscale map where 1.0 is person, 0.0 is background, and — critically — the values in between belong to hair, motion blur, and soft edges. With that matte the render is a sandwich: background video, then the text, then the person (the frame multiplied by its matte), then anything that rides in front.

The hard part isn’t the sandwich; it’s that “compute a matte” has two contradictory specs. The editor needs one immediately — a creator who clicks the Kumar card and stares at a spinner for four minutes is gone. The export needs one that holds up frame-by-frame at 1080p with no flicker, because a matte that jitters between frames reads as a glitch even when every individual frame looks fine. No single system does both, so we run two.

Path 1: a 244 KB segmenter in your tab

The preview matte comes from MediaPipe’s selfie segmenter compiled to WASM, running per frame right in the browser. The model is 244 KB — small enough that the honest bottleneck was never inference, it was everything around it:

• Own your model files. We originally loaded MediaPipe’s runtime from its public CDN, which meant our editor’s first paint had a third-party dependency we didn’t control. Everything — the WASM runtime included — now ships from our Cloudflare-backed CDN under a versioned, immutable path, so browsers and the edge cache it without making the editor depend on an upstream release URL.
• Warm before you’re asked. Segmenter init (WASM fetch + compile) is slow enough to notice, so it starts at upload, not at first use. By the time footage is ready to preview, the segmenter has been warm for minutes.
• One instance, leased. Our favorite self-inflicted bug: the warm-up path and the preview path each constructed their own segmenter, racing each other for the same WASM download — burning double the memory to make the feature slower. The fix was boring and correct: one shared instance behind a lease, and an effect-ordering test so it can’t regress silently.

Segmentation output is nearly binary — it answers “which pixels are person,” not “how transparent is this strand of hair.” For a live preview that’s the right trade. For the file you post, it isn’t.

Path 2: recurrent matting on L4s

The export matte is produced by Robust Video Matting running on NVIDIA L4s — about 34 ms per 1080p frame with CUDA. RVM was built for exactly this job, and its defining property is recurrence: it carries hidden state between frames, so each matte is informed by the last. That’s what buys temporal stability — the difference between hair that breathes naturally and hair that sizzles — and it’s something no per-frame segmenter can offer, however accurate.

Two production notes that cost us real debugging time. First, resist the urge to post-process: our early pipeline binarized and despilled the alpha, which produced clean demos and then black boxes on real footage — RVM’s native soft alpha was right all along, and despill only belongs when you’re actually replacing the background. Second, watch your bake wall-clock. Running inference to completion and then encoding the result made bakes take a quarter of an hour; streaming masks into the encoder as they come off the GPU overlapped the two phases:

Delivery: the WebM detail that looked like an outage

The matte travels as VP9 in WebM — the mainstream codec path that carries a real alpha channel (H.264/MP4 doesn’t). One file, cached and keyed to the source, consumed by both the editor preview and the export renderer.

And it handed us our best war story. Clips would open with the behind-text effect dark for the first seconds, only on cold loads, only sometimes. The chain, once we found it: WebM writers put the seek index — the cues element — at the end of the file by default, because they can’t know byte offsets until encoding finishes. A player that wants to seek must therefore fetch the tail first, so it issues ranged requests against the far end of the object… and our storage layer, mid cold-start, answered enough of those with 503s that the video element gave up entirely. The effect “broke.” Nothing was broken.

The fix is the WebM cousin of MP4’s faststart: remux with the cues moved to the front, so one sequential read gets a player everything it needs to start and to seek. We also taught the player never to give up permanently on a 503 — throttling is weather, not a verdict.

Compositing: where the illusion is won

A perfect matte composited naively still looks pasted. The layer sandwich gets a set of occlusion treatments we call depth effects: where the body overlaps the letters, the text is slightly blurred, shrunk, and dimmed; the silhouette casts a soft contact shadow onto the letterforms; the cutout edge is feathered a couple of pixels. Each is subliminal alone — together they’re why the word reads as in the room. There’s also a layout rule doing quiet work: the hero word is placed so the head occludes only a target band of it — enough overlap to sell the depth, never so much that the word stops being readable.

All of it runs in one place. The editor preview and the export render the same composition — the preview embeds the export renderer rather than approximating it, so what you scrub is what encodes. That single decision deleted a whole class of “it looked different in the editor” bugs, and one final gotcha for anyone building similar: if your sources are HDR phone footage, make sure your renderer’s tone-mapping never touches the matte itself — tone-map a grayscale alpha and your exports come out looking like a photocopier accident.

Why not the obvious alternatives

• “Just use SAM.” Promptable segmenters are superb at stills and far too heavy to run per-frame in a browser tab — and they still hand you a mask, not an alpha matte, so hair is unsolved. Wrong shape on both axes.
• Depth estimation. Monocular depth gives you a beautiful gradient and no crisp person boundary; you end up thresholding it back into a worse matte.
• One pipeline instead of two. Run RVM only, and every preview waits on a GPU round-trip; run MediaPipe only, and exports ship binary edges. The two-path design isn’t an optimization — it’s the product working at all.

Quick answers