Sketch-to-app : A fully offline webapp turn your hand written sketch ui into html css ui using Gemma
We used Google Gemma 3 (gemma3:4b) — a 4-billion parameter multimodal model served locally via Ollama. Gemma 3's vision capabilities allow it to directly "look at" a photo of a hand-drawn sketch and extract a structured understanding of the layout.
The Self-Healing Loop (Key Innovation)
Unlike typical AI code generation, we built a browser-in-the-loop validation system. After Gemma generates HTML:
Playwright loads it in a real Chromium browser.
Console errors and page crashes are captured.
A screenshot is taken and sent back to Gemma for comparison against the original sketch.
If there's a structural mismatch or runtime error, Gemma revises its own code.
This repeats up to 3 times.
This makes the output self-certifying through a real browser, not just token prediction.