metadata

datasets:
  - sciq
  - metaeval/ScienceQA_text_only
  - GAIR/lima
  - Open-Orca/OpenOrca
  - openbookqa
language:
  - en
tags:
  - upstage
  - llama
  - instruct
  - instruction
pipeline_tag: text-generation

LLaMa-30b-instruct model card

Model Developers

Upstage

Backbone Model

LLaMA

Variations

It has different model parameter sizes and sequence lengths: 30B/1024, 30B/2048, 65B/1024.

Input

Models solely process textual input.

Output

Models solely generate textual output

License

This model is under a Non-commercial Bespoke License and governed by the Meta license. You should only use this repository if you have been granted access to the model by filling out this form, but have either lost your copy of the weights or encountered issues converting them to the Transformers format.

Where to send comments

Instructions on how to provide feedback or comments on a model can be found by opening an issue in the Hugging Face community's model repository.

Dataset Details

Used Datasets

Hardware and Software

Hardware

We utilized an A100 for training our model.

Training Factors

We fine-tuned this model using a combination of the DeepSpeed library and the HuggingFace trainer.

Evaluation Results

Overview

We conducted a performance evaluation based on the tasks being evaluated on the Open LLM Leaderboard.
We evaluated our model on four benchmark datasets, which include ARC-Challenge, HellaSwag, MMLU, and TruthfulQA.
We used the lm-evaluation-harness repository, specifically commit b281b0921b636bc36ad05c0b0b0763bd6dd43463.
We can reproduce the evaluation environments using the command below:

Main Results

Model	Average	ARC	HellaSwag	MMLU	TruthfulQA
llama-65b-instruct (Ours, Local Reproduction)	69.4	67.6	86.5	64.9	58.8
llama-30b-instruct-2048 (Ours, Open LLM Leaderboard)	67.0	64.9	84.9	61.9	56.3
falcon-40b-instruct	63.4	61.6	84.3	55.4	52.5
llama-30b-instruct (Ours, Open LLM Leaderboard)	63.2	56.7	84.0	59.0	53.1
llama-65b	62.1	57.6	84.3	63.4	43.0

Scripts

Prepare evaluation environments:

# clone the repository
git clone https://github.com/EleutherAI/lm-evaluation-harness.git

# check out the specific commit
git checkout b281b0921b636bc36ad05c0b0b0763bd6dd43463

# change to the repository directory
cd lm-evaluation-harness

Ethical Issues

Ethical Considerations

There were no ethical issues involved, as we did not include the benchmark test set or the training set in the model's training process.

Contact Us

Why Upstage LLM?

Upstage's LLM research has yielded remarkable results. Our 30B model size outperforms all models worldwide with less than 65B, establishing itself as the leading performer. Recognizing the immense potential for private LLM adoption within companies, we invite you to effortlessly implement a private LLM and fine-tune it with your own data. For a seamless and tailored solution, please don't hesitate to reach out to us (click here to mail).