Video generation models hold substantial potential in areas such as filmmaking. However, current video diffusion models need high computational costs and produce suboptimal results due to high complexity of video generation task. In this paper, we propose ConFiner, an efficient high-quality video generation framework that decouples video generation into easier subtasks: structure control and spatial-temporal refinement. It can generate high-quality videos with chain of off-the-shelf diffusion model experts, each expert responsible for a decoupled subtask. During the refinement, we introduce coordinated denoising, which can merge multiple diffusion experts' capabilities into a single sampling. Furthermore, we design ConFiner-Long framework, which can generate long coherent video with three constraint strategies on Confiner. Experimental results indicate that with only 10% of the inference cost, our ConFiner surpasses representative models like Lavie and Modelscope across all objective and subjective metrics. And ConFiner-Long can generate high-quality and coherent videos with up to 600 frames.
ConFiner decouples the video generation process. Firstly, control expert generates a video structure. Subsequently, temporal and spatial experts perform the refinement of spatio-temporal details. Spatial and temporal experts work together in refinement stage with our coordinated denoising. By adding consistency initialization, coherence guidance and staggered refinement to ConFiner, ConFiner-Long can generate coherent long videos.
Off-road vehicle driving off-road in the desert.
The wizard is manipulating fire magic.
A group of penguins line up and march across the snowfield.
The coffee on the table is steaming.
Astronaut floating in space.
Two white swans gracefully swam in the serene lake.
StreamingT2V
ConFiner
Lavie
Prompt: "Off-road vehicle driving off-road in the desert."
StreamingT2V
ConFiner
Lavie
Prompt: "The wizard is manipulating fire magic."
StreamingT2V
ConFiner
Lavie
Prompt: "A group of penguins line up and march across the snowfield."
StreamingT2V
ConFiner
Lavie
Prompt: "The coffee on the table is steaming."
StreamingT2V
ConFiner
Lavie
Prompt: "Astronaut floating in space."
StreamingT2V
ConFiner
Lavie
Prompt: "Two white swans gracefully swam in the serene lake."
A clownfish in the seabed, with background of sand.
All Strategies
W/O Staggered Refinement
W/O Coherence Guidance
W/O Consistency Initialization