Create README.md

README.md

@@ -0,0 +1,213 @@
+---
+language:
+ - en
+tags:
+ - llama
+ - llm
+ - fine-tuning
+ - fill-in-the-middle
+ - instruction-following
+license: apache-2.0
+datasets:
+ - mlabonne/FineTome-100k
+ - mlfoundations/dclm-baseline-1.0-parquet
+ - wikimedia/wikipedia
+ - bigcode/starcoderdata
+pipeline_tag: text-generation
+---
+
+# Custom LLM with Full Fine-Tuning
+
+## Model Overview
+
+This project implements a custom-trained language model based on the Meta-Llama-3.1-8B architecture. Unlike the previous version which used a high-rank adapter, this model employs full fine-tuning for enhanced learning capacity across a variety of tasks.
+
+- **Developer:** Eric Florenzano
+- **Model Type:** Large Language Model (LLM)
+- **Language(s):** English, with a focus on Python for code-related tasks
+- **License:** Apache-2.0
+- **Base Model:** meta-llama/Meta-Llama-3.1-8B
+
+## Unique Training Approach
+
+This model is trained directly on a mixture of high-quality datasets for general text and code completion tasks, as well as instruction-following. Key features include:
+
+- **Full Fine-Tuning:** Unlike the previous LoRA approach, this version uses full fine-tuning to update all model parameters.
+- **Diverse Dataset Mixture:** Combines pretraining and instruction datasets for comprehensive language understanding.
+- **Multi-Format Instruction Tuning:** Alternates between ChatML and Llama Chat templates for flexible instruction-following.
+- **Contextual Data Prefixing:** Uses source information to address data imbalance during training.
+- **Fill-in-the-Middle (FIM) Training:** Incorporates FIM tasks for enhanced context understanding.
+
+## Training Data
+
+The model is trained on a blend of high-quality data sources:
+
+- **FineTome-100k:** High-quality instruction-tuned data for general language tasks.
+- **dclm-baseline-1.0-parquet:** Apple's pretraining corpus for text completion/prediction.
+- **English, Spanish, and French Wikipedia:** For broad language understanding.
+- **Starcoder:** High-quality Python-focused code dataset for code completion tasks.
+
+## Training Procedure
+
+### Setup
+
+```bash
+pip install -U transformers accelerate trl wandb wheel packaging peft bitsandbytes liger-kernel flash_attn
+```
+
+## Key Features
+
+1. **Full Fine-Tuning:** Updates all model parameters for comprehensive learning.
+2. **8-bit AdamW Optimizer:** Uses `adamw_bnb_8bit` for memory-efficient training.
+3. **Flash Attention 2:** Implements `flash_attention_2` for faster training.
+4. **Gradient Checkpointing:** Enables training with limited GPU memory.
+5. **Liger and Packing:** Utilizes `use_liger=true` and `packing=true` for efficient data handling.
+6. **BFloat16 Precision:** Uses `bfloat16` for balanced precision and performance.
+
+## Advanced Training Techniques
+
+This model incorporates several advanced training techniques to enhance its capabilities:
+
+### 1. Fill-in-the-Middle (FIM) Capability
+
+FIM allows the model to complete text when given both a prefix and a suffix, making it particularly useful for tasks like code completion, text infilling, and context-aware generation.
+
+#### Using FIM with the Model
+
+To use the FIM capability, structure your input with special tokens:
+
+- `<|fim_start|>`: Marks the start of the FIM input
+- `<|fim_marker|>`: Separates the prefix from the suffix
+- `<|fim_gen|>`: Indicates where the generated content should begin
+- `<|fim_end|>`: Marks the end of the FIM input
+
+Example FIM input:
+```
+<|fim_start|>{prefix}<|fim_marker|>{suffix}<|fim_gen|>
+```
+
+The model will generate content to replace `<|fim_gen|>`, filling in the middle between the prefix and suffix.
+
+### 2. Reverse Prediction and Instruction Backtranslation
+
+This technique enhances the model's context understanding by training it to predict previous parts of a conversation or text. It's also known as instruction backtranslation.
+
+#### How it works:
+1. The model is given a snippet of conversation or text.
+2. It's then tasked with predicting what came before this snippet.
+3. This process helps the model understand context, conversation flow, and logical progression of ideas.
+
+#### Benefits:
+- Improved context understanding
+- Enhanced ability to maintain coherent, contextually appropriate conversations
+- Better grasp of cause-and-effect relationships in text
+
+#### Example use case:
+Input:
+```
+Human: Thank you for the information about Paris. Can you recommend some popular tourist attractions there?
+```
+Task: Predict the previous exchange in this conversation.
+
+Possible model output:
+```
+Human: What's the capital of France?
+Assistant: The capital of France is Paris. It's known as the "City of Light" and is famous for its art, culture, and historic landmarks.
+Human: Thank you for the information about Paris. Can you recommend some popular tourist attractions there?
+```
+
+### 3. Meta-FIM
+
+Meta-FIM applies the Fill-in-the-Middle technique to larger chunks of text, including entire conversations or documents. This improves the model's ability to handle complex, nested contexts.
+
+#### Benefits:
+- Enhanced understanding of long-range dependencies in text
+- Improved ability to maintain coherence across longer contexts
+- Better performance on tasks requiring integration of information from multiple parts of a document or conversation
+
+#### Example:
+```
+<|fim_start|>Human: What's the weather like today?
+Assistant: I'm sorry, but I don't have access to real-time weather information. Could you please provide your location?<|fim_marker|>Human: Thank you for the information about Paris. Can you recommend some popular tourist attractions there?<|fim_gen|>Human: I'm in Paris, France.
+Assistant: Ah, Paris! While I can't provide real-time weather information, I can tell you that Paris generally has a temperate climate. May I suggest checking a local weather website or app for the most up-to-date information?
+Human: That's a good idea, thanks. While we're on the topic of Paris, can you tell me about some famous landmarks?
+Assistant: Certainly! Paris is known for its iconic landmarks. Here are a few famous ones:
+1. Eiffel Tower
+2. Louvre Museum
+3. Notre-Dame Cathedral
+4. Arc de Triomphe
+5. Sacré-Cœur Basilica<|fim_end|>
+```
+
+In this example, the model needs to understand and generate a coherent conversation that fits between the given start and end points.
+
+## Evaluation
+
+| Tasks |Version| Filter |n-shot| Metric | |Value | |Stderr|
+|-----------------|-------|----------------|-----:|-----------|---|-----:|---|------|
+|tinyBenchmarks | N/A| | | | | | | |
+| - tinyArc | 0|none | 25|acc_norm |↑ |0.5791|± | N/A|
+| - tinyGSM8k | 0|flexible-extract| 5|exact_match|↑ |0.5536|± | N/A|
+| | |strict-match | 5|exact_match|↑ |0.5536|± | N/A|
+| - tinyHellaswag | 0|none | 10|acc_norm |↑ |0.8391|± | N/A|
+| - tinyMMLU | 0|none | 0|acc_norm |↑ |0.6377|± | N/A|
+| - tinyTruthfulQA| 0|none | 0|acc |↑ |0.4914|± | N/A|
+| - tinyWinogrande| 0|none | 5|acc_norm |↑ |0.7608|± | N/A|
+
+### Training Command
+
+```bash
+python sft_14.py \
+ --run_name="llama3.1-8b-continued3" \
+ --model_name_or_path="meta-llama/Meta-Llama-3.1-8B" \
+ --dataset_name="mlfoundations/dclm-baseline-1.0-parquet,mlabonne/FineTome-100k" \
+ --report_to="wandb" \
+ --optim="adamw_bnb_8bit" \
+ --lr_scheduler_type="cosine" \
+ --max_steps=100000 \
+ --max_seq_length=64000 \
+ --learning_rate=0.00001 \
+ --attn_implementation="flash_attention_2" \
+ --save_strategy="steps" \
+ --save_steps 50 \
+ --save_total_limit=10 \
+ --per_device_train_batch_size=1 \
+ --per_device_eval_batch_size=1 \
+ --gradient_accumulation_steps=8 \
+ --logging_steps=1 \
+ --num_train_epochs=1 \
+ --push_to_hub \
+ --hub_model_id="ericflo/Llama-3.1-8B-ContinuedTraining3-FFT" \
+ --hub_strategy="all_checkpoints" \
+ --gradient_checkpointing \
+ --use_liger=true \
+ --packing=true \
+ --torch_dtype="bfloat16" \
+ --output_dir="continuedtraining3_output"
+```
+
+## Intended Uses
+
+This model is designed for:
+
+- Text Completion and Generation
+- Code Completion (especially Python)
+- Instruction Following
+- General Language Understanding
+- Context-Aware Text Infilling (using FIM)
+
+## Limitations and Biases
+
+- The model may exhibit biases present in the training data.
+- It lacks real-time knowledge beyond its training data.
+- Should not be used for critical decision-making without human oversight.
+
+## Technical Specifications
+
+- **Base Model:** meta-llama/Meta-Llama-3.1-8B
+- **Training Approach:** Full Fine-Tuning
+- **Library:** Hugging Face Transformers and TRL
+
+## Contact
+
+For inquiries about this model, please contact Eric Florenzano through the [model repository](https://huggingface.co/ericflo/Llama-3.1-8B-ContinuedTraining3-FFT).