LLMs Explained,
Dolly

Model Card

View All Models

100+ Technical Experts

50 Custom AI projects

4.8 Minimum Rating

Introduction
Key Highlights
Training Details
Business Applications
Model Features
Model Tasks
Fine-tuning
Sample Codes
Limitations
Other LLMs

About Model

Databricks' Dolly is a commercially licensed large language model developed on the Databricks machine learning platform. It is specifically designed to excel at instruction-following tasks. Built upon pythia-12b, Dolly is trained using approximately 15,000 instruction/response fine-tuning records known as databricks-dolly-15k. These records were generated by Databricks employees and cover various capability domains outlined in the InstructGPT paper, including brainstorming, classification, closed QA, generation, information extraction, open QA, and summarization. While dolly-v2-12b may not be considered a state-of-the-art model, it demonstrates surprisingly high-quality instruction-following capabilities that go beyond the characteristics of its foundational model. This unique behavior makes Dolly a valuable tool for tasks that require accurate and effective execution of instructions.

Model Repository

Author Note

Key highlights

The dataset databricks-dolly-15k comprises 15,000 prompt/response pairs meticulously created by humans to facilitate the fine-tuning of large language models in instruction-following tasks. This dataset, released under the Creative Commons Attribution-ShareAlike 3.0 Unported License, allows unrestricted usage, modification, and extension, even for commercial purposes. This dataset holds the distinction of being the first open-source collection of human-generated instructions explicitly tailored to enable large language models to exhibit the interactive capabilities seen in ChatGPT. Over 5,000 Databricks employees contributed to the creation of databricks-dolly-15k during the period of March and April in 2023. The training records within the dataset are natural, expressive, and encompass a diverse range of behaviors, including brainstorming, content generation, information extraction, and summarization.

Training Details

Training data

Dolly is trained on ~15k instruction/response fine tuning records databricks-dolly-15k generated by Databricks employees in capability domains from the InstructGPT paper, including brainstorming, classification, closed QA, generation, information extraction, open QA and summarization.

Training Observations

The evaluation results indicate that dolly-v2-12b does not achieve state-of-the-art performance and, in certain evaluation benchmarks, it even falls short of dolly-v1-6b. The researchers hypothesize that this discrepancy can be attributed to the composition and size of the fine-tuning datasets used. However, a comprehensive understanding of the factors contributing to these variations necessitates additional investigation and analysis. Further studies are required to provide a more conclusive explanation for these observed differences.

Sample Codes

Running the model on a GPU

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

# load Dolly model and tokenizer
model_name = "databricks/dolly-6B-512"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# check if GPU is available
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Using device: {device}")

# move model to GPU if available
model = model.to(device)

# generate text using Dolly
input_text = "The quick brown fox jumps over the lazy dog"
input_ids = tokenizer.encode(input_text, return_tensors="pt").to(device)
output = model.generate(input_ids, max_length=50)

# print generated text
generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
print(generated_text)

Running the model on a CPU

import transformers
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load the Dolly model and tokenizer
model_name = "databrickslabs/dolly-6B-512"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# Set device to CPU
device = torch.device("cpu")
model.to(device)

# Generate text using Dolly
prompt = "Hello, Dolly!"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)
output = model.generate(input_ids, max_length=50, do_sample=True, top_p=0.95, top_k=50)
generated_text = tokenizer.decode(output[0], skip_special_tokens=True)

print(generated_text)

Model Limitations

Dolly-6b is not intended for commercial use but only for research purposes.
Dolly-6b is not a state-of-the-art generative language model and may not perform competitively with more modern model architectures or models subject to larger pretraining corpora.
Dolly-6b's main strength is its instruction-following capabilities, given that it is based on an open-source model that anyone can use.
Dolly is under active development, so any list of limitations may not be exhaustive.
Dolly-6b struggles with syntactically complex prompts, mathematical operations, factual errors, dates and times, open-ended question answering, hallucination, enumerating lists of a specific length, and stylistic mimicry.

Other LLMs

OPT

Meta AI first introduced OPT (Open Pre-trained Transformer) Language Model and released it in metaseq's repository on May 3rd, 2022

Galactica

Galactica is a large-scale language model developed by the research team at Meta Platforms, Inc.

LLaMA

Meta first introduced LLaMA in February 2023. LLaMA (Large Language Model Meta AI)

White Papers

Products

MENU

Dolly

LLMs Explained,
Dolly

100+ Technical Experts

50 Custom AI projects

4.8 Minimum Rating

About Model

Key highlights

Training Details

Training data

Training Observations

Sample Codes

Running the model on a GPU

Running the model on a CPU

Model Limitations

Other LLMs

OPT

Galactica

LLaMA

White Papers

Products

MENU

LLMs Explained,Dolly

100+ Technical Experts

50 Custom AI projects

4.8 Minimum Rating

About Model

Key highlights

Training Details

Training data

Training Observations

Sample Codes

Running the model on a GPU

Running the model on a CPU

Model Limitations

Other LLMs

OPT

Galactica

LLaMA

LLMs Explained,
Dolly