Video editing direction · 06 Nov to 19 Nov · Levon Khachatryan

H3 video editing: style transfer and character swap

Where we are on the two use cases we chose: turning a video into another style, and swapping the people in trending videos.

1 · Video style transfer

15 s clips in both directions, then long videos (3:46 and 4:41)

2 · Character swap on trending videos

A new person from one picture, the whole 15 s in one pass

Where we are

From a broad look at H3 editing to two use cases

1. H3's general video editing capabilities

Removal, general edits and character swap, 63 cases on base H3, FastH3 and Pika: h3-video-editing.pages.dev

2. Two use cases where H3 alone is weak and that are interesting for us

Character swap for trending videos, and video style transfer; strong and weak points of H3 and open methods: h3-video-editing-2.pages.dev

3. Now: a working solution for both

Style transfer works on 15 s clips in both directions and is being extended to long videos; character swap works for the whole 15 s in one pass.

Part 1 · Video style transfer

Style transfer on 15 s clips

One generation per clip, both directions: style to real, and real to style.

Model

FastH3 (8 steps) Ref2VA with the ControlNet-Union-2.0 for H3: scribble from the source video, strength 0.5

Conditions

One picture per character in the target look, plus the first frame in the target look; a timeline prompt of who does what when

Sound and lip sync

The source soundtrack is pinned during generation, so the output keeps the original sound and the lips follow the speech

Style transfer · 15 s

Style to real: 10 examples

Source | result. 3D, anime, puppet, pixel art, clay, 2D cartoon and comic clips turned into live-action footage.

What to see

For most clips the result is acceptable: real people and places, the same motion, timing and sound as the source.

Style transfer · 15 s

Real to style: 10 examples

Source | result. Live-action clips turned into 3D, comic, watercolour, 2D cartoon, oil, anime, ink wash, ukiyo-e and clay.

What to see

For most clips the target style holds for the whole 15 s and the people, motion and lip sync are kept.

Part 2 · Style transfer on long videos

Style to real on long videos

We are making the solution work on long videos, focusing on style to real first, on two kinds of cartoon.

Two kinds of cartoon

Naruto (3:46): realistic body shapes. The Simpsons (4:41): body shapes that are not realistic

One run per video

The whole video is generated at once as 14.4 s windows, joined by randomized blending (StreamingT2V)

Control and prompt

FastH3 with depth control on the first 3 of 8 steps; a time-accurate timeline prompt checked against the source

Style to real · long video

Naruto, 3:46

Source | result. Realistic character shapes. Depth control on the first 3 of 8 steps, the whole video in one run.

What to see

Real people and places that follow the source scenes and timing; in some places the join between two chunks is still visible.

Style to real · long video

The Simpsons, 4:41

Source | result. Character shapes that are not realistic. Same setup as Naruto.

What to see

The scenes, actions and people are followed and look real; joins between chunks are still visible for some chunks.

Long videos · what we see

Not bad on both cartoons; the joins need a better method

The depth-conditioned results are not bad on both cartoons

Real people and places that follow the source scenes and timing, on realistic (Naruto) and non-realistic (Simpsons) character shapes.

Joins between chunks are still visible for some chunks

We need something better than the original randomized blending from StreamingT2V; we are testing better joins on 42 s excerpts.

Open question

One solution for both kinds of cartoon, or two different solutions for realistic and non-realistic character shapes. Real to style on long videos comes after this.

Part 3 · Character swap on trending videos

Character swap on trending videos

Replace the people in a viral clip with new people from one picture each, keeping the motion, the moments and the sound.

Model

H3 with the two community Motion Transfer LoRAs: TARA and Guide Latents

One pass

The whole 15 s at 768p in one generation, one picture per new person, original sound

Clips

A dance trend, a movie scene and a music challenge; 8 examples on the next slides

Character swap · Dance trend

Three women become three grandmothers

Source | TARA | Guide Latents, and the three pictures.

What to see

Each grandmother replaces the right woman at her moment and follows the dance for the whole 15 s; TARA keeps the parking lot, Guide Latents redraws it.

Character swap · Dance trend

The same three women become three grandfathers

New pictures, same clip and roles.

What to see

Each grandfather appears in his own shot and keeps his look.

Character swap · Movie scene

A new couple, each person in two looks

Many short shots: the man in the house and in the field, the woman as a zombie and in the field.

What to see

Both new people appear in their shots, and the cuts and poses follow the source.

Character swap · Movie scene

A second couple, each person in two looks

Distinctive faces, so identity is easy to judge.

What to see

Identities hold in every shot with Guide Latents; TARA leaks the original woman in one short close-up.

Character swap · Music challenge

Two performers become a new couple

Woman on the left, man on the right.

What to see

The new couple performs every move of the source in sync with the song.

Character swap · Music challenge

Another couple

Same 15 s clip, new pictures.

What to see

The swap and the motion hold for the whole clip.

Character swap · Music challenge

A third couple

Same 15 s clip, new pictures.

What to see

The swap and the motion hold for the whole clip.

Character swap · Music challenge

A man and a dog

Same clip; the dog takes the place of the woman.

What to see

Even a dog standing upright performs the source moves.

Summary

Where we are

Both use cases have a working solution on 15 s clips; long videos are the current focus.

Style transfer, 15 s

Acceptable results in both directions, style to real and real to style

Style transfer, long videos

Not bad with depth control; joins between chunks still need a better method than randomized blending

Character swap

The Motion Transfer LoRAs swap the new people from start to end on trending videos