DALL·E 2 Explained - model architecture, results and comparison
DALL·E 2 Explained - model architecture, results and comparison
Dalle-2 or unCLIP is an image generation model that leverages the diffusion model to generate images from text embeddings. Here is a video that explains the DALLE-2 paper. More specifically, the model architecture, the results and comparison to other state-of-the-art models like GLIDE.
Paper Title
Hierarchical Text-Conditional Image Generation with CLIP Latents
Paper Abstract
Contrastive models like CLIP have been shown to learn robust representations of images that capture both semantics and style. To leverage these representations for image generation, we propose a two-stage model: a prior that generates a CLIP image embedding given a text caption, and a decoder that generates an image conditioned on the image embedding. We show that explicitly generating image representations improves image diversity with minimal loss in photorealism and caption similarity. Our decoders conditioned on image representations can also produce variations of an image that preserve both its semantics and style, while varying the non-essential details absent from the image representation. Moreover, the joint embedding space of CLIP enables language-guided image manipulations in a zero-shot fashion. We use diffusion models for the decoder and experiment with both autoregressive and diffusion models for the prior, finding that the latter are computationally more efficient and produce higher-quality samples.
Paper Link
https://arxiv.org/abs/2204.06125
Website
https://openai.com/dall-e-2
Video Outline
0:00 - Introduction
1:05 - Method / Model of CLIP
1:53 - Method / Model of unCLIP
3:02 - Decoder Architecture
4:04 - Prior Architecture
5:59 - Image Manipulation
7:49 - Image Interpolation
8:26 - Languge Guided Manipulation
9:14 - Importance of the Prior
10:07 - Results / Human Evaluation
AI Bites
YouTube: / aibites
Twitter: / ai_bites
Patreon: / ai_bites
Github: https://github.com/ai-bites
Vision Transformers (ViT): • Vision Transformer (ViT) - An Image is Wor...
Data Efficient Image Transformer (DeiT): • DeiT - Data-efficient image transformers &...
📚 📚 📚 BOOKS I HAVE READ, REFER AND RECOMMEND 📚 📚 📚
📖 Deep Learning by Ian Goodfellow - https://amzn.to/3Wnyixv
📙 Pattern Recognition and Machine Learning by Christopher M. Bishop - https://amzn.to/3ZVnQQA
📗 Machine Learning: A Probabilistic Perspective by Kevin Murphy - https://amzn.to/3kAqThb
📘 Multiple View Geometry in Computer Vision by R Hartley and A Zisserman - https://amzn.to/3XKVOWi
Music: https://www.bensound.com
#machinelearning #deeplearning #aibites