What AI thinks working out looks like [ControlNet + StableDiffusion]
MediaPipe, ControlNet, StableDiffusion, and OpenPose working together to make AI Generative Art. AI generated voice is from ElevenLabs, but good results can also be gotten via vits-svc. My workflow is posted below. This video does not use DaVinci Resolve to clean up the flickering, but you can. SDXL is right around the corner, can't wait to play with it! I'll get back to making ChatGPT Eldrich horror content soon, don't you worry.
I'm happy to answer any specific questions in the comments. And if you want to know more about any of the techniques, feel free to request videos. For this video, we utilize the following tricks:
ControlNet with MediaPipe's OpenPose can do sophisticated choreography. Create a reference video with the movement you want, and then use each frame to drive a ControlNet module in StableDiffusion.
Do temporal filtering on the pose in Python/MediaPipe (NOT Automatic1111 etc) to prevent deviant keyframes.
Mirror pose along the ground plane in Python/MediaPipe to manage reflections.
Create a second ControlNet module, and weakly influence the current frame's generation based on the pose of the previous frame. This gives the pose "inertia", and keeps Stable Diffusion on track while still allowing the pose to change over time.
Use a final Seg (Segmentation) ControlNet model to handle compositional elements. In this video, I'm using Person (to keep the mask of the man roughly the same frame to frame), Sky, and Beach to manage composition.
Having all the control elements diffusing around pose makes it harder to prevent flickering. I've found that rendering at a low framerate and then upsampling to a higher one can yield good results. DaVinci Resolve can also help.
Visit www.RiskOfReptiles.com for more eclectic projects. GANs, GPT4, ChatGPT, and more