Realtime Doodling with Stable Diffusion in Photoshop/Krita | Workflow Demo
Demonstration of a Stable Diffusion workflow I'm developing for Photoshop/Krita. There are still some parts that are rough around the edges, but the experience (especially using a tablet) is compelling! Left is the canvas I'm working on, right is the synthesis of the AI's changes. Brushes and brush effects make the experience feel less...MS Paint.
The basic flow is using ComfyPhotoSD and ComfyUI (or an equivalent Krita flow) with the Juggernaut XL checkpoint and an LCM Adaptor LoRA for realtime (well, on a RTX 4090 anyway, lol). Using a scribble ControlNet adapter to guide the AI, and a fixed set of styling prompts (smooth brushstrokes, watercolor on textured paper, sponge painting, etc) for text_positive_l prompt encoding. For text_positive_g, I have a DeepBooru interrogator trying to figure out what I'm scribbling (sakura tree, angel wings, etc). Keeping denoising low is important to keep the AI from just completely taking over. To merge the results, I advise copying the AI image to a new layer with 40-70% opacity, and then merging to the canvas layer or you'll get the dreaded "burn-in". Lower cfg scale or a second de-noising pass can also help with this.
I'll release some workflows once I factor out some proprietary code from a startup I'm assisting. Sub for those as they become available :)