Background replacement went from a green screen and a lighting kit to a checkbox in about three years, and the results are still, mostly, obviously fake. Not because the segmentation is bad — it’s remarkably good — but because a convincing composite needs three things to agree, and the segmentation is only the first of them.
What the tool is actually computing
Replacing a background means separating the person from everything else, on every frame. That separation is a matte: for each pixel, how much of it belongs to the subject. Crucially it isn’t a yes/no mask — the edge of a person is genuinely part-subject, part-background, and hair is almost entirely that.
This is the same machinery that puts text behind a speaker, and the engineering behind it is in our matting deep dive. Two failure modes carry over directly:
- Halo. A rim of the original background clinging to the silhouette. Instantly legible as a cut-out.
- Crawl. The edge shimmering frame to frame because the matte is computed independently each time. Invisible on a still, obvious in motion — which is why checking a paused frame proves nothing.
The three tells, in the order people notice them
- Edges. Hair against a detailed background is the hard case, and it’s where the eye goes. A slightly soft, slightly halo’d edge is what separates “composited” from “filmed there”.
- Lighting. The subject is lit by their real room; the background implies a different light. Warm lamp on the face, cool daylight behind — the brain flags it as wrong long before anyone consciously notices why. This is the one that makes technically-perfect composites still feel off.
- Depth and motion. A pin-sharp, perfectly still background behind a handheld subject reads as a photograph the person is standing in front of. Real backgrounds are slightly out of focus and move with the camera.
Getting a better result
- Separate yourself from the wall. A metre of distance does more for edge quality than any setting. It gives the model a real depth cue and reduces spill.
- Avoid matching colours. A grey hoodie against a grey wall is the hardest possible input. Contrast between subject and background is free accuracy.
- Light the subject to match the target. If the background is a warm room, don’t sit under a cool ring light. This is the single biggest lever and the one nobody pulls.
- Blur the new background. A slight defocus does two things: it implies a lens, and it hides edge imperfections in the part of the frame you can’t fix.
- Keep hair simple. Loose flyaway hair against detail is the worst case for every matting model there is.
- Check it in motion, at speed. Edge crawl is invisible on a paused frame, which is exactly how it ships.
Cheaper things that usually work better
| Instead of replacing | What it costs | When |
|---|---|---|
| Blur the real background | Nothing — same matte, no compositing risk | The room is distracting rather than wrong |
| Darken or desaturate it | Nothing | You want attention on the face, not a new location |
| A graphic panel behind you | A design decision | Explainer content — it can carry information too |
| Text behind the subject | Same matte, no realism burden | You wanted production value, not a new room |
| Move, and film somewhere better | Ten minutes | Almost always the highest-quality option |
That last row is not a joke. A real room, even an imperfect one, has correct lighting, correct depth, correct motion and no edges to get wrong. Replacement is worth its cost when the new background is doing something — establishing a place, carrying a graphic, creating an image you couldn’t otherwise make. It is rarely worth it to hide a shelf.
Quick answers
Why do replaced video backgrounds look fake?
Three tells, in order of how often they give it away: edge quality (especially hair), lighting that doesn't match between subject and background, and a background that is too sharp or too still relative to the footage. Edges are what people consciously notice; lighting is what makes it feel wrong even when they can't say why.
Do I need a green screen?
Not any more for most talking-head footage — segmentation models compute a per-frame matte from ordinary video. A green screen still gives a cleaner edge on hair and fast motion, so it's worth it for high-stakes work, but it's no longer the price of entry.
What is despill and why does it matter?
Colour from the old background bleeding onto the edge of the subject — a green rim from a green screen, or a warm rim from a lamp behind you. It's only worth correcting when you're actually replacing the background; run it over the original background and you get a smudged edge instead of a clean one.
Should I replace the background or just blur it?
Blur, usually. It solves the actual problem — a distracting room — with none of the edge and lighting risk, and it doesn't ask the viewer to believe anything. Replace the background when the new one is doing real work, not to hide a messy shelf.
Use the matte for something with no realism burden
The same per-frame cutout that replaces a background can put your captions behind you — production value with nothing to get wrong.
Open Moonshot free