|
--- |
|
metrics: |
|
- accuracy |
|
- code_eval |
|
--- |
|
# Model Card for Evaluate360M |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
Evaluate360M is a lightweight large language model optimized for reasoning tasks. It is designed to run efficiently on low-end commercial hardware, such as mobile phones, while maintaining strong performance in logical reasoning and general-purpose applications. |
|
|
|
- **Developed by:** [More Information Needed] |
|
- **Funded by [optional]:** [More Information Needed] |
|
- **Shared by [optional]:** [More Information Needed] |
|
- **Model type:** Transformer-based decoder model |
|
- **Language(s) (NLP):** English |
|
- **License:** [More Information Needed] |
|
- **Finetuned from model [optional]:** `HuggingFaceTB/SmolLM2-360M-Instruct` |
|
|
|
### Model Sources |
|
|
|
- **Repository:** [More Information Needed] |
|
- **Paper [optional]:** [More Information Needed] |
|
- **Demo [optional]:** [More Information Needed] |
|
|
|
## Uses |
|
|
|
### Direct Use |
|
Evaluate360M is intended for general-purpose reasoning tasks and can be used in applications that require lightweight LLMs, such as: |
|
- Mobile-based AI assistants |
|
- Low-power embedded systems |
|
- Edge computing applications |
|
|
|
### Downstream Use |
|
It can be further fine-tuned for specific domains, including code generation, summarization, or dialogue systems. |
|
|
|
### Out-of-Scope Use |
|
- Not optimized for handling very large context windows |
|
- Not designed for generating high-fidelity creative text, such as poetry or fiction |
|
|
|
## Bias, Risks, and Limitations |
|
|
|
### Limitations |
|
- Struggles with handling large context windows. |
|
- Not evaluated for potential biases yet. |
|
|
|
### Recommendations |
|
Users should be aware of the model’s limitations in context length and should evaluate its performance for their specific use cases. |
|
|
|
## How to Get Started with the Model |
|
|
|
```python |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
model_name = "evaluate360m" |
|
tokenizer = AutoTokenizer.from_pretrained(model_name) |
|
model = AutoModelForCausalLM.from_pretrained(model_name) |
|
|
|
inputs = tokenizer("What is the capital of France?", return_tensors="pt") |
|
outputs = model.generate(**inputs) |
|
print(tokenizer.decode(outputs[0])) |
|
``` |
|
|
|
## Training Details |
|
|
|
### Training Data |
|
- **Dataset:** `HuggingFaceH4/Bespoke-Stratos-17k` |
|
- **Preprocessing:** Token packing enabled (`--packing`), sequence length up to 2048 tokens |
|
|
|
### Training Procedure |
|
- **Optimizer & Precision:** |
|
- `bf16` mixed precision |
|
- `gradient_accumulation_steps = 8` |
|
- Gradient checkpointing enabled |
|
- **Hyperparameters:** |
|
- Learning rate: `2e-5` |
|
- Epochs: `3` |
|
- Batch size: `4` (per device, both training and evaluation) |
|
- **Evaluation & Saving:** |
|
- Evaluation every `500` steps |
|
- Model checkpoint saved every `1000` steps, keeping a max of `2` checkpoints |
|
|
|
### Compute Infrastructure |
|
- **Hardware Used:** A100 GPU |
|
- **Training Time:** 6 hours |
|
|
|
## Evaluation |
|
|
|
- **Benchmarks:** No evaluation conducted yet. |
|
- **Metrics:** Not available yet. |
|
|
|
## Environmental Impact |
|
|
|
- **Hardware Type:** A100 GPU |
|
- **Hours Used:** 6 hours |
|
- **Cloud Provider:** [More Information Needed] |
|
- **Compute Region:** [More Information Needed] |
|
- **Carbon Emitted:** [More Information Needed] |
|
|
|
## Technical Specifications |
|
|
|
### Model Architecture |
|
- Similar to SmolLM2-360M |
|
- Inspired by MobileLLM |
|
- Uses **Grouped-Query Attention (GQA)** |
|
- Prioritizes depth over width |
|
|
|
## Citation [optional] |
|
|
|
**BibTeX:** |
|
[More Information Needed] |
|
|
|
**APA:** |
|
[More Information Needed] |
|
|
|
## More Information |
|
|
|
[More Information Needed] |
|
|
|
## Model Card Authors [optional] |
|
|
|
[More Information Needed] |
|
|
|
## Model Card Contact |
|
|
|
[More Information Needed] |
