AI Image & Video

A Four-Step AI Talking-Head Pipeline: How to Build It, What Breaks, and Why We Deleted Eye Enhancement

An AI talking head is not one end-to-end model but four independent steps passed like a relay. Before adding a feature, ask what you can least afford to compromise in this pipeline. A good-looking effect is not a reason to keep it.

10/21/2026 ｜ 4 min read ｜ Runs locally in ComfyUI, no rented GPU or API. About 30 minutes to read, 1 to 2 hours to set up depending on your machine

The one idea that matters

An AI talking head is, in essence, four independent steps in a relay, not one model that does everything. Each step does one thing and hands its result to the next:

1. Lip-sync: one photo plus one audio clip becomes a video whose mouth follows the audio
2. Identity swap: transplant that motion onto the face or character you actually want
3. Face restoration: faces from AI frame generation are usually soft, so repair them frame by frame
4. Upscaling: raise the resolution so the result does not look cheap

Once you see it as a relay and not a black box, you know where to look when a step fails, and why adding a feature can break earlier results in unexpected places.

Steps

Step 1: Lip-sync (audio-driven mouth motion)

Use the open-source project InfiniteTalk (MeiGen-AI, Apache-2.0, commercial use allowed). Feed it a photo and an audio clip and it generates a video whose mouth matches the audio. It generates mouth, expression and head motion purely from sound, with no separate demo video needed.

Step 2: Identity swap (character replacement)

Step 1’s mouth motion is right, but the face may not be the character you want. Use a motion-transfer or character-replacement model (we use Z.ai’s SCAIL-2, Apache-2.0): the Step 1 video is the motion template and your target photo is the identity source, and the output has the new face with the same motion.

For longer videos, these models usually need “generate in segments, then join”, with each segment taking the last few frames of the previous one as its continuity reference. That takes some scripting; clicking through a UI by hand gets painful.

Step 3: Face restoration

After the identity swap, skin texture, stubble and wrinkles usually look soft. Run GFPGAN (an open-source face-restoration model) frame by frame and detail comes back clearly.

Step 4: Upscaling

Finally use Real-ESRGAN (an open-source upscaler) to raise the overall resolution. Our output spec is 1024x1792 vertical, suited to short-video talking-head material.

Traps you will hit

1. Old repositories break in chains on new Python. One package’s install script read its version with a dynamic exec() pattern that fails under Python 3.13’s internal optimizations, so the install errors out. Fix: download the source package, replace the line that reads the version with a fixed string, then install (assuming you trust the author). Rule of thumb: when a package has been stale for two years, expect to hit incompatibilities, usually more than one.

2. Reinstalling Python packages while ComfyUI is running can fail. Packages it is already using, such as opencv-python, fail with “file in use”. Fix: check whether the old version works (usually it does), skip it, and install only the dependencies that are truly missing.

3. A great-looking feature can damage something more important without you noticing (the main thread of this note). We wanted one more step for eye and micro-expression, and used LivePortrait. The eyes did look livelier, but a careful side-by-side with the original photo showed the face shape and eyebrows had changed too. It was not a micro-expression of the same person; it was a different face.

   Root cause: expression transfer compares the face in a driving video against your character’s face, computes a set of deformation parameters from the difference, and applies them. That difference inevitably carries the driving face’s features, and the stronger the expression you ask for, the more leaks through. One parameter controls both, so you cannot turn up expression without affecting face shape.

   How to decide whether to adopt a new feature: ask what you can least afford to compromise in this pipeline (for us, identity consistency). After adding the feature, do not stop at one metric improving. Compare input and output by eye and look specifically for whether that thing was sacrificed. Metrics lie; eyes are harder to fool.

Takeaway

The talking-head pipeline is a four-step relay (lip-sync, identity swap, face restoration, upscaling), each step wired to an open-source tool. Before adding a feature, be clear about what you cannot compromise. We deleted a feature we had working precisely because it touched what it should not.

Universal rescue: how to ask an AI when you are stuck

I am on [your OS, for example Windows 11] using [your tool, for example ComfyUI] to [your goal]. The situation is: [describe the behavior, or paste the full error text]. I have already tried: [list, or write “nothing yet”]. Assume I am a beginner. Tell me step by step what the likely causes are and give concrete steps I can follow directly.

For details that go stale quickly, such as version numbers and paths, ask the AI to look up the current way. Do not copy any tutorial text that is over six months old.