Text to Image Models,
Kandinsky

Kandinsky 2.1 builds upon the advancements of Dall-E 2 and Latent diffusion, while also introducing novel concepts. It incorporates the CLIP model as both the text and image encoder, and utilizes a diffusion image prior to establish a mapping between the latent spaces of CLIP modalities. This innovative approach enhances the model's visual capabilities and opens up new possibilities for blending images and manipulating them based on text guidance.

Model Card

View All Models

100+ Technical Experts

50 Custom AI projects

4.8 Minimum Rating

About Model
Model Highlights
Training Details
Architecture
Model Tasks
Results
Misuse
Limitations/Bias
Other LLMs

About Model

Kandinsky 2.1 builds upon the advancements of Dall-E 2 and Latent diffusion while introducing novel concepts. It incorporates the CLIP model as both the text and image encoder and utilizes a diffusion image before establishing a mapping between the latent spaces of CLIP modalities. This innovative approach enhances the model's visual capabilities and opens up new possibilities for blending images and manipulating them based on text guidance.

In November 2022, Kandinsky 2.0 was introduced as a multi-lingual image synthesis model, capable of accepting prompts in more than 100 languages. However, the most recent version, Kandinsky 2.1, has significantly improved image quality and functionality. The updated architecture of Kandinsky 2.1 has led to remarkable enhancements in the quality of generated images. Users can now expect more visually appealing and realistic results than the previous version.

Additionally, Kandinsky 2.1 offers a valuable set of image manipulation features. This allows users to manipulate and modify the generated images in various ways, providing greater control and flexibility in their creative process. Although Kandinsky 2.1 continues to support English and Russian, it represents a substantial leap forward regarding image quality and the range of available image manipulation options.

Model Repository

HuggingFace

Author Blon

Model Highlights

In the generation process, Kandinsky relies on visual embeddings as an intermediate representation. These embeddings serve as a condensed and continuous form that captures the semantic meaning of the text description. By utilizing visual embeddings, the model gains the ability to perform advanced operations, including blending and precise control over the generated image. Visual embeddings offer a valuable advantage by enabling the model to manipulate and modify the image without solely relying on the original text description. This means that Kandinsky can perform more sophisticated image transformations and adjustments based on the visual embeddings, resulting in a more nuanced and refined output.

Kandinsky 2.1 uses CLIP model and diffusion image prior (mapping) between latent spaces of CLIP modalities. This approach increases the visual performance of the model and unveils new horizons in blending images and text-guided image manipulation. The model use use transformer with num_layers=20, num_heads=32 and hidden_size=2048.

Text encoder (XLM-Roberta-Large-Vit-L-14) - 560M
Diffusion Image Prior — 1B
CLIP image encoder (ViT-L/14) - 427M
Latent Diffusion U-Net - 1.22B
MoVQ encoder/decoder - 67M

Training Details

Kandinsky 2.1 was trained on a large-scale image-text dataset LAION HighRes and fine-tuned on our internal datasets. Kandinsky 2.0 was trained on a large 1B multilingual set, including samples that we used to train Kandinsky. In terms of diffusion architecture Kandinsky 2.0 implements UNet with 1.2B parameters. These encoders and multilingual training datasets unveil the real multilingual text-to-image generation experience.

The main Text2Image diffusion model was trained on the basis of 170M text-image pairs from the LAION HighRes dataset (an important condition was the presence of images with a resolution of at least 768x768). The use of 170M pairs is due to the fact that we kept the UNet diffusion block from Kandinsky 2.0, which allowed us not to train it from scratch. Further, at the stage of fine-tuning, a dataset of 2M very high-quality high-resolution images with descriptions (COYO, anime, landmarks_russia, and a number of others) was used separately collected from open sources.

Sample Codes

Kandinsky 2.1 text2image

Kandinsky 2.1 image fuse

Kandinsky 2.1 inpainting

from kandinsky2 import get_kandinsky2
from PIL import Image
import numpy as np

model = get_kandinsky2('cuda', task_type='inpainting', model_version='2.1', use_flash_attention=False)
init_image = Image.open('img.jpg')
mask = np.ones((768, 768), dtype=np.float32)
mask[:,:550] =  0
images = model.generate_inpainting(
    'man 4k photo', 
    init_image, 
    mask, 
    num_steps=150,
    batch_size=1, 
    guidance_scale=5,
    h=768, w=768,
    sampler='p_sampler', 
    prior_cf_scale=4,
    prior_steps="5"
)

Other LLMs

OPT

Meta AI first introduced OPT (Open Pre-trained Transformer) Language Model and released it in metaseq's repository on May 3rd, 2022

Galactica

Galactica is a large-scale language model developed by the research team at Meta Platforms, Inc.

LLaMA

Meta first introduced LLaMA in February 2023. LLaMA (Large Language Model Meta AI)

White Papers

Products

MENU

Kandinsky 2.1

Text to Image Models,
Kandinsky

100+ Technical Experts

50 Custom AI projects

4.8 Minimum Rating

About Model

Model Highlights

Training Details

Sample Codes

Kandinsky 2.1 text2image

Kandinsky 2.1 image fuse

Kandinsky 2.1 inpainting

Results

Other LLMs

OPT

Galactica

LLaMA

White Papers

Products

MENU

Text to Image Models,Kandinsky

100+ Technical Experts

50 Custom AI projects

4.8 Minimum Rating

About Model

Model Highlights

Training Details

Sample Codes

Kandinsky 2.1 text2image

Kandinsky 2.1 image fuse

Kandinsky 2.1 inpainting

Results

Other LLMs

OPT

Galactica

LLaMA

Text to Image Models,
Kandinsky