Video editing direction · 06 Nov to 19 Nov · Levon Khachatryan
H3 video editing: style transfer and character swap
Where we are on the two use cases we chose: turning a video into another style, and swapping the people in trending videos.
1 · Video style transfer
15 s clips in both directions, then long videos (3:46 and 4:41)
2 · Character swap on trending videos
A new person from one picture, the whole 15 s in one pass
Where we are
From a broad look at H3 editing to two use cases
1. H3's general video editing capabilities
Removal, general edits and character swap, 63 cases on base H3, FastH3 and Pika: h3-video-editing.pages.dev
2. Two use cases where H3 alone is weak and that are interesting for us
Character swap for trending videos, and video style transfer; strong and weak points of H3 and open methods: h3-video-editing-2.pages.dev
3. Now: a working solution for both
Style transfer works on 15 s clips in both directions and is being extended to long videos; character swap works for the whole 15 s in one pass.
Part 1 · Video style transfer
Style transfer on 15 s clips
One generation per clip, both directions: style to real, and real to style.
Model
FastH3 (8 steps) Ref2VA with the ControlNet-Union-2.0 for H3: scribble from the source video, strength 0.5
Conditions
One picture per character in the target look, plus the first frame in the target look; a timeline prompt of who does what when
Sound and lip sync
The source soundtrack is pinned during generation, so the output keeps the original sound and the lips follow the speech
Style transfer · 15 s
Style to real: 10 examples
Source | result. 3D, anime, puppet, pixel art, clay, 2D cartoon and comic clips turned into live-action footage.
What to see
For most clips the result is acceptable: real people and places, the same motion, timing and sound as the source.
Style transfer · 15 s
Real to style: 10 examples
Source | result. Live-action clips turned into 3D, comic, watercolour, 2D cartoon, oil, anime, ink wash, ukiyo-e and clay.
What to see
For most clips the target style holds for the whole 15 s and the people, motion and lip sync are kept.
Part 2 · Style transfer on long videos
Style to real on long videos
We are making the solution work on long videos, focusing on style to real first, on two kinds of cartoon.
Two kinds of cartoon
Naruto (3:46): realistic body shapes. The Simpsons (4:41): body shapes that are not realistic
One run per video
The whole video is generated at once as 14.4 s windows, joined by randomized blending (StreamingT2V)
Control and prompt
FastH3 with depth control on the first 3 of 8 steps; a time-accurate timeline prompt checked against the source
Style to real · long video
Naruto, 3:46
Source | result. Realistic character shapes. Depth control on the first 3 of 8 steps, the whole video in one run.
What to see
Real people and places that follow the source scenes and timing; in some places the join between two chunks is still visible.
Style to real · long video
The Simpsons, 4:41
Source | result. Character shapes that are not realistic. Same setup as Naruto.
What to see
The scenes, actions and people are followed and look real; joins between chunks are still visible for some chunks.
Long videos · what we see
Not bad on both cartoons; the joins need a better method
The depth-conditioned results are not bad on both cartoons
Real people and places that follow the source scenes and timing, on realistic (Naruto) and non-realistic (Simpsons) character shapes.
Joins between chunks are still visible for some chunks
We need something better than the original randomized blending from StreamingT2V; we are testing better joins on 42 s excerpts.
Open question
One solution for both kinds of cartoon, or two different solutions for realistic and non-realistic character shapes. Real to style on long videos comes after this.
Part 3 · Character swap on trending videos
Character swap on trending videos
Replace the people in a viral clip with new people from one picture each, keeping the motion, the moments and the sound.
Model
H3 with the two community Motion Transfer LoRAs: TARA and Guide Latents
One pass
The whole 15 s at 768p in one generation, one picture per new person, original sound
Clips
A dance trend, a movie scene and a music challenge; 8 examples on the next slides
Character swap · Dance trend
Three women become three grandmothers
Source | TARA | Guide Latents, and the three pictures.
What to see
Each grandmother replaces the right woman at her moment and follows the dance for the whole 15 s; TARA keeps the parking lot, Guide Latents redraws it.
Character swap · Dance trend
The same three women become three grandfathers
New pictures, same clip and roles.
What to see
Each grandfather appears in his own shot and keeps his look.
Character swap · Movie scene
A new couple, each person in two looks
Many short shots: the man in the house and in the field, the woman as a zombie and in the field.
What to see
Both new people appear in their shots, and the cuts and poses follow the source.
Character swap · Movie scene
A second couple, each person in two looks
Distinctive faces, so identity is easy to judge.
What to see
Identities hold in every shot with Guide Latents; TARA leaks the original woman in one short close-up.
Character swap · Music challenge
Two performers become a new couple
Woman on the left, man on the right.
What to see
The new couple performs every move of the source in sync with the song.
Character swap · Music challenge
Another couple
Same 15 s clip, new pictures.
What to see
The swap and the motion hold for the whole clip.
Character swap · Music challenge
A third couple
Same 15 s clip, new pictures.
What to see
The swap and the motion hold for the whole clip.
Character swap · Music challenge
A man and a dog
Same clip; the dog takes the place of the woman.
What to see
Even a dog standing upright performs the source moves.
Summary
Where we are
Both use cases have a working solution on 15 s clips; long videos are the current focus.
Style transfer, 15 s
Acceptable results in both directions, style to real and real to style
Style transfer, long videos
Not bad with depth control; joins between chunks still need a better method than randomized blending
Character swap
The Motion Transfer LoRAs swap the new people from start to end on trending videos