PaperSummary03 : High-resolution image synthesis with latent diffusion models

The paper addresses the computational inefficiencies of traditional diffusion models for high-resolution image synthesis. Latent Diffusion Models (LDMs) operate in the latent space of pretrained autoencoders instead of the pixel space. The key difference from traditional methods include the separation of perceptual compression from generative modeling and the use of a convolutional architecture.

The key steps of algorithm and training are:

1. Latent space representation: A pretrained autoencoder compresses images into a lower-dimensional latent space while preserving perceptual fidelity.
2. Diffusion process in latent space: Diffusion models are trained to denoise latent representations, focusing on semantics rather than imperceptible details.
3. Cross attention conditioning: Conditioning inputs (such as text, semantic maps) are integrated via cross-attention, enabling multimodal generation.
4. Training is divided into two phases: autoencoder training for compression and subsequent training of the diffusion model in the latent space. This approach allows reuse across multiple tasks.

LDMs significantly lowers the computational costs for training and inference and achieves high-quality synthesis at resolutions up to the megapixel scales without extensive hardware demands. These models are versatile and successfully applied to diverse tasks such as text-to image synthesis, inpainting, super-resolution and unconditional generation.

References:

High-Resolution Image Synthesis with Latent Diffusion Models