Model Card for Model ID

lainshower/Llama3-8b-alpaca

Model Details

Full Fine-tuned Llama3-8B Alpaca.

Training with (BF16) Mixed Precision For Stability.

This is One-Epoch Training Model due to Overfitting Problem > Click here Llama3-8B-Alpaca-3EPOCHS For the 3-Epochs Training Model.

A detailed description will be given later.

Direct Use

[Templates]

You can use the following standard templates for inference the Llama3 Alpaca model:


PROMPT_DICT = {
    "prompt_input": (
        "Below is an instruction that describes a task, paired with an input that provides further context. "
        "Write a response that appropriately completes the request.\n\n"
        "### Instruction:\n{instruction}\n\n### Input:\n{input}\n\n### Response:"
    ),
    "prompt_no_input": (
        "Below is an instruction that describes a task. "
        "Write a response that appropriately completes the request.\n\n"
        "### Instruction:\n{instruction}\n\n### Response:"
    ),
}

[Code]

[Model Loading]


### We recommend using Float32 when running inference on the models.
model = LlamaForCausalLM.from_pretrained("lainshower/Llama3-8b-alpaca")
tokenizer = AutoTokenizer.from_pretrained("lainshower/Llama3-8b-alpaca")

[Template]


PROMPT_DICT = {
    "prompt_input": (
        "Below is an instruction that describes a task, paired with an input that provides further context. "
        "Write a response that appropriately completes the request.\n\n"
        "### Instruction:\n{instruction}\n\n### Input:\n{input}\n\n### Response:"
    ),
    "prompt_no_input": (
        "Below is an instruction that describes a task. "
        "Write a response that appropriately completes the request.\n\n"
        "### Instruction:\n{instruction}\n\n### Response:"
    ),
}

ann = {}
ann['instruction'] = '''You are presented with the quiz "What causes weather changes on Earth? " But you don't know the answer, so you turn to your teacher to ask for hints. He says that "the Earth being tilted on its rotating axis causes seasons" and "weather changes from season to season". So, what's the best answer to the question? Choose your answer from: (a). the sun's energy (b). The tilt in its rotating axis. (c). high temperature (d). Weather in space (e). Vertical movement (f). Greenhouse gases (g). Spinning backwards (h). wind and erosion Answer:'''
prompt = PROMPT_DICT["prompt_no_input"].format_map(ann)
'''
Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
"What causes weather changes on Earth? " But you don't know the answer, so you turn to your teacher to ask for hints. He says that "the Earth being tilted on its rotating axis causes seasons" and "weather changes from season to season". So, what's the best answer to the question? Choose your answer from: (a). the sun's energy (b). The tilt in its rotating axis. (c). high temperature (d). Weather in space (e). Vertical movement (f). Greenhouse gases (g). Spinning backwards (h). wind and erosion Answer:

### Response:
'''

[Generation]


input_ids = token.batch_encode_plus([prompt], return_tensors="pt", padding=False)
total_sequences = model.generate(input_ids=input_ids['input_ids'].cuda(), attention_mask=input_ids['attention_mask'].cuda(), max_length=490, do_sample=True, top_p=0.9) 
print(token.decode(total_sequences[0], skip_special_tokens=True)))


### **LLAMA3-8B-ALPACA**

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
"What causes weather changes on Earth? " But you don't know the answer, so you turn to your teacher to ask for hints. He says that "the Earth being tilted on its rotating axis causes seasons" and "weather changes from season to season". So, what's the best answer to the question? Choose your answer from: (a). the sun's energy (b). The tilt in its rotating axis. (c). high temperature (d). Weather in space (e). Vertical movement (f). Greenhouse gases (g). Spinning backwards (h). wind and erosion Answer:

### Response:Weather changes from season to season due to the Earth being tilted on its rotating axis.


### **LLAMA3-8B ** (with same generation setting)

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
"What causes weather changes on Earth? " But you don't know the answer, so you turn to your teacher to ask for hints. He says that "the Earth being tilted on its rotating axis causes seasons" and "weather changes from season to season". So, what's the best answer to the question? Choose your answer from: (a). the sun's energy (b). The tilt in its rotating axis. (c). high temperature (d). Weather in space (e). Vertical movement (f). Greenhouse gases (g). Spinning backwards (h). wind and erosion Answer:

### Response:

(No Answer)


### **Gold Response**
(b). The tilt in its rotating axis.

Datum is Sourced From Open-Orca Dataset

Training Details

Training Data

We use stanford_alpaca for Instruct Tuning Llama-8B.

Training Hyperparameters

Learning Rates : 2e-5
Training Procedures : Mixed Precision (bfloat16)
Context Length: 512
This is One-Epoch Training Model due to Overfitting Problem > Click here Llama3-8B-Alpaca-3EPOCHS For the 3-Epochs Training Model.

Training Graph

Citation


@misc{lainshower-llama3-alpaca,
    title = {Llama3-8b-alpaca: A Full Fine-tuned Llama3-8B Model with Alpaca Training},
    author = {Lainshower},
    year = {2024},
    publisher = {Hugging Face},
    howpublished = {\url{https://huggingface.co/lainshower/Llama3-8b-alpaca}},
    note = {One-Epoch Training Model with BF16 Mixed Precision}
}

lainshower
/

Llama3-8b-alpaca