Text to Image Models
Karlo

Karlo represents a text-conditional diffusion model that builds upon the unCLIP framework, incorporating prior, decoder, and super-resolution modules. The official repository includes an enhanced version of the standard super-resolution module designed to upscale images from 64px to 256px in just seven reverse steps.

The standard super-resolution (SR) module follows a two-step process. Initially, it undergoes training with the DDPM objective, enabling it to upscale 64px images to 256px within the first six denoising steps, utilizing a respacing technique. Subsequently, the SR module undergoes further fine-tuning using the VQ-GAN-style loss, enabling it to perform the final reverse step and recover high-frequency details. This approach has demonstrated exceptional effectiveness in upscaling low-resolution images using a minimal number of reverse steps.

For training purposes, all components are developed from scratch using a dataset of 115 million image-text pairs, including the COYO-100M, CC3M, and CC12M subsets. The Prior and Decoder modules leverage the ViT-L/14 model OpenAI's CLIP repository provides. Notably, in contrast to the original unCLIP implementation, the trainable transformer in the decoder is replaced with the text encoder in ViT-L/14, resulting in improved efficiency. Regarding the SR module, an initial training phase of 1 million steps is undertaken using the DDPM objective and followed by an additional 234,000 steps to fine-tune the module with the supplementary component.