[Workflow] Stable Diffusion video deflickering and cleanup, WITHOUT DaVinci Resolve
Workflow:
Video generation is another entire video (comment+sub if interested) but hints are to use ControlNet against a fixed pose to keep the model "on topic", and then let it daydream about the accessories. You can use an SSIM metric to increase or decrease "step size" to try to keep the perceived motion per frame pretty steady.
This cleanup trick requires a frame interpolation engine. I see some good results with RIFE, but I've been enjoying FILM from Google. It's sort of like latent upscaling, but across time instead of space. - Create two interpolation functions:
-- interp_upsample([frame0, frame1, frame2...]): Return [frame0, frame 0.5, frame 1, frame 1.5...]
-- interp_smooth([frame0, frame1, frame2...]): Return [interp of frame 0 and frame 2, interp of frame 1 and frame 3, interp of frame 2 and frame 4...]
Play around to see what works for your video, but I've found the following works well:
imgs = load_frames()
imgs = interp_upsample(imgs)
imgs = interp_smooth(imgs)
imgs = interp_upsample(imgs)
imgs = interp_smooth(imgs)
imgs = interp_smooth(imgs)
And then interp_upsample(imgs) until you hit your desired framerate
If there's enough interest (like and comment, y'all), I can write up an explanation or make a video on why this particular sequence, but basically, you're trying to minimize large frame to frame jumps, and then do a temporal smoothing, and then zoom in and do it again. It's like numerical approximation techniques but in a crazy N dimensional latent space.
More projects at: www.RiskofReptiles.com