AI Video
From a sentence to a moving picture.
A video is a sequence of pictures, with sound. Creating one is the hardest generation task, because every frame must look good, and also fit the frames around it. In this lesson, you will learn how video models work, and where they struggle.
- Many frames that must agree
- Why video is so hard
- What the tools can do today
Diffusion, in space and time
Video models extend the diffusion idea. Instead of denoising one picture, they denoise a whole block of frames together, treating time as another dimension. The model must keep objects consistent, a person must keep the same face, and the same coat, from the first frame to the last. And the motion must obey the laws of physics, or at least look as if it does.
What video tools can do
Today's tools can do many things. Turn a sentence into a short clip. Animate a still image. Extend a clip, or change its style. Replace the background, or remove an object. And generate matching sound. Clips are usually only a few seconds, or tens of seconds long, but length, and quality, improve every few months.
Where video still struggles
There are still weaknesses. Hands, and fine details, sometimes look odd. Long clips can drift, so a character slowly changes. Physics is imperfect, objects may pass through each other. And text on screen can be garbled. Knowing these limits helps you plan shots, that work, short, simple, and clear, and then edit them together.
Common weaknesses
- Hands and fine details
- Characters drift over time
- Imperfect physics
- Garbled on-screen text
Working with the limits
- Keep shots short and simple
- Edit clips together
- Check every second
- Fix details by hand
Summary
Good. Video models denoise blocks of frames across time, keeping objects and motion consistent. They can create, extend, and edit clips, but still struggle with details, and long shots. Next, we see how all these abilities, text, images, audio, and video, merge, in multimodal models.
Key takeaways- Video = diffusion across frames and time
- Text-to-video, image-to-video, editing
- Limits: details, drift, physics