Decoupled Long Video Generation with Chain of Training-free Diffusion Experts

GitHub logo Code (coming soon)

Abstract

Video generation models hold substantial potential in areas such as filmmaking. However, current video diffusion models struggle with poor quality and slow speed due to extreme complexity of video generation task. In this paper, we propose ConFiner, an efficient video generation framework that decouples video generation into three easier subtasks: structure control, spatial refinement and temporal refinement. It can generate high-quality videos with chain of off-the-shelf diffusion model experts, each expert responsible for a decoupled subtask. During the refinement, we introduce coordinated denoising, which can merge multiple diffusion experts' capabilities into a single sampling. Furthermore, we design ConFiner-Long framework, which can generate long coherent video with three constraint strategies on ConFiner. Experimental results indicate that with only 20% of the inference cost, our ConFiner achieves higher average score than representative models like Wan2.1 and HunyuanVideo on Vbench. And ConFiner-Long can generate high-quality and coherent videos with infinite length.

Pipeline

Pipeline

ConFiner decouples the video generation process. Firstly, control expert generates a video structure. Subsequently, temporal and spatial experts perform the refinement of spatio-temporal details. V' and V'' represent videos on SE's noise scheduler and TE's scheduler, respectively. V_0(t) denotes the estimated pure video at timestep t. By adding consistency initialization, coherence guidance and staggered refinement to ConFiner, ConFiner-Long can generate coherent long videos.

Example Videos(Generated by Confiner Setting1 in less than 5s)

Off-road vehicle driving off-road in the desert.

The wizard is manipulating fire magic.

A group of penguins line up and march across the snowfield.

Flying cars go through a cyber city.

Clownfish swims among the colorful corals.

The coffee on the table is steaming.

Example Videos(Generated by Confiner Setting2)

A group of penguins line up and march across the snowfield.

Clownfish swims among the colorful corals.

The wizard is manipulating fire magic.

Comparisons with Previous Methods

StreamingT2V

ConFiner(Setting1)

Lavie

Prompt: "Off-road vehicle driving off-road in the desert."

StreamingT2V

ConFiner(Setting1)

Lavie

Prompt: "The wizard is manipulating fire magic."

StreamingT2V

ConFiner(Setting1)

Lavie

Prompt: "A group of penguins line up and march across the snowfield."

StreamingT2V

ConFiner(Setting1)

Lavie

Prompt: "The coffee on the table is steaming."

StreamingT2V

ConFiner(Setting1)

Lavie

Prompt: "Flying cars go through a cyber city."

StreamingT2V

ConFiner(Setting1)

Lavie

Prompt: "Clownfish swims among the colorful corals."

Experimental Results

Ablation Study on Three Strategies of ConFiner-Long

A clownfish in the seabed, with background of sand.

Comparison 1

All Strategies

Comparison 2

W/O Staggered Refinement

Comparison 3

W/O Coherence Guidance

Comparison 4

W/O Consistency Initialization