LLMs Explained,
StableLM

StableLM is an open-source large language model launched by Stability AI. The initial version of the model, with 3 billion and 7 billion parameters, has been released. The larger versions of 15 billion and 65 billion will be made available in the near future. This model is accessible to developers for both commercial and research use.

Model Details

View All Models

100+ Technical Experts

50 Custom AI projects

4.8 Minimum Rating

Model Details

StableLM-Tuned-Alpha models are auto-regressive language models based on the NeoX transformer architecture. Fine-tuned checkpoints (StableLM-Tuned-Alpha) are licensed under the Non-Commercial Creative Commons license, in line with the original non-commercial license specified by Stanford Alpaca.

Model Repository

HuggingFace

Author Note

Checkpoints

Model highlights

StableLM is trained using a novel experimental dataset derived from The Pile, which is significantly larger, containing 1.5 trillion content tokens. Despite its relatively smaller size of 3 to 7 billion parameters (in contrast to GPT-3's 175 billion parameters), StableLM demonstrates remarkably high performance in conversational and coding tasks. The vastness and diversity of the dataset contribute to the model's effectiveness in these domains, showcasing its ability to excel despite its parameter limitations.

As part of their proof-of-concept, the researchers conducted fine-tuning on the model using Stanford Alpaca's procedure. This involved utilizing a combination of five recently curated datasets specifically designed for conversational agents, namely Stanford's Alpaca, Nomic-AI's gpt4all, RyokoAI's ShareGPT52K datasets, Databricks labs' Dolly, and Anthropic's HH. The resulting models, referred to as StableLM-Tuned-Alpha, will be made publicly available by the researchers.

Training Details

Training data

Models are fine-tuned on a combination of five datasets: Alpaca, GPT4All Prompt Generations, Anthropic HH, DataBricks Dolly, closed QA, generation, information extraction, open QA and summarization; and ShareGPT Vicuna.

Training infrastructure

Models are learned via supervised fine-tuning on the aforementioned datasets, trained in mixed-precision (FP16), and optimized with AdamW. The publishers outline the hyperparameters as shown in the table below.

Model Types

StableLM has released two architecture variations with varying numbers of parameters. Here's a brief explanation of each of them:

Parameters	Batch Size	Learning Rate	Betas
3B	256	2e-5	(0.9, 0.99)
7B	128	2e-5	(0.9, 0.99)

Model Features

To be updated soon

Model Tasks

To be updated soon

Limitations

Bias

While the inclusion of the mentioned datasets assists in guiding the base language models towards more socially acceptable text distributions, it is important to note that fine-tuning alone cannot completely eliminate biases and toxic elements. Therefore, users are encouraged to remain vigilant regarding potential issues that may arise in the generated responses. It is crucial to exercise human judgment and refrain from regarding model outputs as infallible or definitive sources of truth. Responsible usage is strongly advised.

Other LLMs

OPT

Meta AI first introduced OPT (Open Pre-trained Transformer) Language Model and released it in metaseq's repository on May 3rd, 2022

Galactica

Galactica is a large-scale language model developed by the research team at Meta Platforms, Inc.

LLaMA

Meta first introduced LLaMA in February 2023. LLaMA (Large Language Model Meta AI)

Use Case	Description
Customer Service	Model can be used to understand customer queries and respond to them accurately and efficiently, reducing the need for human intervention.

White Papers

Products

MENU

StableLM – WIP

LLMs Explained,
StableLM

100+ Technical Experts

50 Custom AI projects

4.8 Minimum Rating

Model Details

Model highlights

Training Details

Training data

Training infrastructure

Model Types

Business Applications

Model Features

To be updated soon

Model Tasks

To be updated soon

Fine-tuning

To be updated soon

Benchmark Results

Limitations

Bias

Other LLMs

OPT

Galactica

LLaMA

White Papers

Products

MENU

LLMs Explained, StableLM

100+ Technical Experts

50 Custom AI projects

4.8 Minimum Rating

Model Details

Model highlights

Training Details

Training data

Training infrastructure

Model Types

Business Applications

Model Features

To be updated soon

Model Tasks

To be updated soon

Fine-tuning

To be updated soon

Benchmark Results

Limitations

Bias

Other LLMs

OPT

Galactica

LLaMA

LLMs Explained,
StableLM