InstructEval Models Explained,
OpenAssistant

OpenAssistant-SFT-7-Llama-30B-HF represents an advanced Large Language Model (LLM) built upon the potent Llama 30B architecture. Its development involved extensive training on the expansive OpenAI Assistant (OASST) dataset, encompassing a diverse array of synthetic and real-world dialogues. This rigorous training has equipped the model with a robust command of the language, evident in its impressive 1.3 million-token vocabulary. OpenAssistant-SFT-7-Llama-30B-HF excels in generating text and performing a wide spectrum of tasks, including language translation, creative content creation, and providing insightful answers to inquiries.

Model Details

View All Models

100+ Technical Experts

50 Custom AI projects

4.8 Minimum Rating

An Overview of OpenAssistant

OpenAssistant LLM was developed under the guidance of LAION, a non-profit research institute led by Yannic Kilcher, the institute's founder. The team, including Antje Barth, Chris Fregly, Shelbee Eigenbrode, and Mike Chambers, meticulously crafted this model. Its performance has been evaluated rigorously across various NLP benchmarks, consistently surpassing other prominent language models. Impressively, the model demonstrated exceptional prowess with a remarkable score of 93.3 on the esteemed GLUE benchmark, renowned for assessing language models across diverse linguistic tasks. The model's sleek yet sophisticated architecture empowers it with unparalleled language processing and comprehension skills, enabling superior performance across many language-centric applications and solidifying its status as an indispensable language modeling asset.

Faster than many other prominent LLM models due to its 4-bit quantization.

Large Vocabulary

With its extensive vocabulary of over 500,000 words, the OpenAssistant LLaMA 30B SFT 7 model can generate text that surpasses the complexity and nuance achievable by smaller models.

The model has 80% of the same capabilities as GPT-3 LLM but with 20% less latency.

Long-range Dependencies

The model's proficiency in comprehending and capturing long-range dependencies in text facilitates its capacity to generate text characterized by enhanced coherence and logical cohesiveness.

The model's smaller size makes it faster to train and deployable compared to GPT-3.

Multi-modal Capabilities

Leveraging its capability to process text and code, the model demonstrates its versatility as a powerful tool suitable for an extensive array of natural language processing (NLP) tasks.

Model Details

The OpenAssistant LLaMA 30B SFT 7 is a robust language model trained on a vast 30 billion-word dataset. It represents the seventh phase of OpenAssistant's work on the Llama 30B model. With a vocabulary of 50,257 tokens and 137 billion parameters, it comes in HF and GGML formats. The model excels in tasks like text generation, translation, and question-answering while maintaining efficiency with around 1.5 GB of memory. It's a valuable tool for NLP researchers and developers.

Hugging Face

Model Repository

Research Paper

Model Highlights

The architecture of the OpenAssistant LLaMA 30B SFT 7 model is built on the state-of-the-art Transformer, a neural network structure ideal for handling language tasks. It's trained using the masked language modeling (MLM) approach, where the model guesses hidden words in a sequence. Moreover, this helps the model grasp word relationships and context. The DeepSpeed framework, designed to speed up training for large language models, is used during training. It employs methods like pipeline and model parallelism to make training faster and more efficient.

The OpenAssistant LLaMA 30B SFT 7 model is a cutting-edge Transformer model with 137 layers and 136 million parameters. Notably, it includes a switchable attention mechanism. This innovative feature allows the model to switch its focus between different types of attention based on the input data, making it adaptable and versatile in processing various information.
The switch transformer architecture is an advanced version of the Transformer, specially designed for better efficiency and scalability. It uses switchable attention to dynamically switch between different attention methods based on the input data, boosting flexibility and performance.

Training Details

Training Procedure

The training of the OpenAssistant LLaMA 30B SFT 7 model occurs in two distinct stages. In the first stage, the model goes through pre-training using a dataset that encompasses text and code. Then moving to the second stage, fine-tuning uses a smaller dataset tailored for natural language processing tasks. This two-step approach ensures the model excels in grasping overall language comprehension and specialized nuances within specific domains.

Training Observation

The OpenAssistant LLaMA 30B SFT 7 model was trained on a cluster consisting of 128 GPUs over a period of around two weeks. This rigorous training approach has led to exceptional outcomes in different natural language processing tasks, such as text summarization, question answering, and machine translation. Additionally, this training process enhanced the model's speed and effectiveness, making it highly suitable for practical applications.

Sample Codes

Original model card


llama-30b-sft-7:
  dtype: fp16
  log_dir: "llama_log_30b"
  learning_rate: 1e-5
  model_name: /home/ubuntu/Open-Assistant/model/model_training/.saved/llama-30b-super-pretrain/checkpoint-3500
  #model_name: OpenAssistant/llama-30b-super-pretrain
  output_dir: llama_model_30b
  deepspeed_config: configs/zero3_config_sft.json
  weight_decay: 0.0
  residual_dropout: 0.0
  max_length: 2048
  use_flash_attention: true
  warmup_steps: 20
  gradient_checkpointing: true
  gradient_accumulation_steps: 12
  per_device_train_batch_size: 2
  per_device_eval_batch_size: 3
  eval_steps: 101
  save_steps: 485
  num_train_epochs: 4
  save_total_limit: 3
  use_custom_sampler: true
  sort_by_length: false
  #save_strategy: steps
  save_strategy: epoch
  datasets:
    - oasst_export:
        lang: "bg,ca,cs,da,de,en,es,fr,hr,hu,it,nl,pl,pt,ro,ru,sl,sr,sv,uk"
        input_file_path: 2023-04-12_oasst_release_ready_synth.jsonl.gz
        val_split: 0.05
    - vicuna:
        val_split: 0.05
        max_val_set: 800
        fraction: 1.0
    - dolly15k:
        val_split: 0.05
        max_val_set: 300
    - grade_school_math_instructions:
        val_split: 0.05
    - code_alpaca:
        val_split: 0.05
        max_val_set: 250

Other InstructEval Models

Falcon 7B Instruct

Falcon-7B-Instruct is a 7B parameter causal decoder-only model built by TII based on Falcon-7B and finetuned on a mixture of chat/instruct datasets.

Alpaca LoRA

Alpaca LoRA is a 65B parameter LLM that has undergone quantization to 4 bits, resulting in a smaller and more efficient model compared to other LLMs.

StableVicuna

StableVicuna-13B-HF represents an LLM model that has undergone meticulous fine-tuning through reinforcement learning from human feedback (RLHF).

White Papers

Products

MENU

Open Assistant

InstructEval Models Explained,
OpenAssistant

100+ Technical Experts

50 Custom AI projects

4.8 Minimum Rating

An Overview of OpenAssistant

Faster than many other prominent LLM models due to its 4-bit quantization.

Large Vocabulary

The model has 80% of the same capabilities as GPT-3 LLM but with 20% less latency.

Long-range Dependencies

The model's smaller size makes it faster to train and deployable compared to GPT-3.

Multi-modal Capabilities

Model Details

Model Highlights

Training Details

Training Procedure

Training Observation

Sample Codes

Original model card

Limitations and Bias

Other InstructEval Models

Falcon 7B Instruct

Alpaca LoRA

StableVicuna

White Papers

Products

MENU

InstructEval Models Explained,OpenAssistant

100+ Technical Experts

50 Custom AI projects

4.8 Minimum Rating

An Overview of OpenAssistant

Faster than many other prominent LLM models due to its 4-bit quantization.

Large Vocabulary

The model has 80% of the same capabilities as GPT-3 LLM but with 20% less latency.

Long-range Dependencies

The model's smaller size makes it faster to train and deployable compared to GPT-3.

Multi-modal Capabilities

Model Details

Model Highlights

Training Details

Training Procedure

Training Observation

Sample Codes

Original model card

Limitations and Bias

Other InstructEval Models

Falcon 7B Instruct

Alpaca LoRA

StableVicuna

InstructEval Models Explained,
OpenAssistant