Paper Info

• CVPR 2024 Best Paper (2 out of 2719)
• Zhengqi Li, Richard Tucker, Noah Snavely and Aleksander Holynski from Google Research
• Link to Live Demo

Related Works

• Animating Pictures with Stochastic Motion Textures (Yung-Yu Chuang, et al. 2005)
• Image-space Modal Bases for Plausible Manipulation of Objects in Video (Abe Davis, Justin G. Chen and Frédo Durand. 2015)
• Visual Vibration Analysis (Abe Davis. 2016)

How Related?

• Abe Davis, Noah Snavely and Zhengqi Li are all from Cornell Graphics and Vision Group.

• A demo for Plausible Manipulation of Objects in Video :

• GID is some kind of deep learning version of the work above.

Framework

• Input: a single still image.
• M1: Generate spectral volume <== LDM ~ IFFT
• M2: Render seamless looping video <== Softmax splatting
• M3: Setup an interactive dynamic scene <== Davis' work

Optical Flow

• Motion of brightness parrern in the image.

Donguri wave illusion

Optical flow != Motion field

団栗(ドングリ) More details check this link

Quasi-periodic Motion

We only consider videos featuring oscillatory motions such as those of trees, flowers, or candle flames swaying in the breeze.

Spectral Volume

FFT of quasi-periodic optical flow.

Formulation:

• Video: for each pixel .
• Motion texture:
  • Displacement(optical flow): in time domain.
• Spectral volume:
  • in frequency domain.

M1: Generate Spectral Volume

• Employ latent diffusion model (LDM) as the backbone for the motion prediction module
• Techniques:
  • Frequency adaptive normalization
  • Frequency-coordinated denoising(multi channel output)

M2: Render Video

• Refer to: Niklaus, Simon, and Feng Liu. "Softmax splatting for video frame interpolation." Proceedings of the IEEE/CVF conference on CVPR. 2020.

M3: Interactive Dynamics

• Refer to: Davis, Abe, Justin G. Chen, and Frédo Durand. "Image-space modal bases for plausible manipulation of objects in video." ACM Transactions on Graphics (TOG) 34.6 (2015): 1-7.

• Spectral volume can approximate an image-space modal basis that is a projection of the vibration modes of the underlying scene and can be used to simulate the object’s response to a user-defined force.

where are complex modal coordinates.

Comparison Experiments(quantitative)

Comparison Experiments(qualitative)

Comparison Exps(with large video model)

• On a randomly selected 30 videos from the test set, we ask users "which video is more realistic?". Users report a 80.9% preference for our approach over others.

Ablation Experiments

Limitations

• Can fail to model nonoscillating motions or high-frequency vibrations
• May degrade in scenes with thin moving objects or objects with large displacements

THANKS