README.md

---
language:
- en
base_model:
- meta-llama/Llama-3.1-70B-Instruct
tags:
- finance
- Llama3.1
---
# Llama-3.1-Omni-FinAI-70B Model Card

## Model Overview (Built with Llama)
Llama-3.1-Omni-FinAI-70B is a pre-trained large language model optimized for finance-specific fine-tuning applications. Based on the LLaMA 3.1 70B architecture, this model was pre-trained on 143 billion tokens of high-quality financial texts. Llama-3.1-Omni-FinAI-70B provides a foundation for further fine-tuning in specialized financial analysis tasks.

## Model Details
- **Base Model**: Llama-3.1-70B-Instruct
- **Training Data**: 
  - SEC 10-K, 10-Q, and 8-K filings
  - Reuters News data (RCV1, TRC2)
  - Finance-specific papers from Arxiv
  - Financial discussions from Reddit
  - Wikipedia
- **Primary Use Case**: Pre-training for finance-specific fine-tuning, allowing users to leverage Llama-3.1-Omni-FinAI-70B's foundational financial language understanding.

## Use Cases
Llama-3.1-Omni-FinAI-70B is designed as a base model for finance-specific fine-tuning tasks, supporting applications such as:
- Sentiment Analysis
- Stock Movement Prediction
- QA Instruction
- Summarization
- Predictive Financial Analysis

## Training Process
Llama-3.1-Omni-FinAI-70B was trained using the NVIDIA NeMo framework on 64 H100 GPUs, utilizing a diverse dataset that ensures robust performance for fine-tuning in finance-related applications.

## Limitations
This model is pre-trained for finance-specific fine-tuning tasks and may require additional fine-tuning for specialized applications. Due to its large size, substantial computational resources are recommended for deployment.

## License
This model is licensed under the Llama 3.1 Community License.

## Citation
If you use the Llama-3.1-Omni-FinAI-70B model, please cite as follows:
> Chiu, I-Chan and Hung, Mao-Wei and Chen, Zih-Ching and Chiu, Jun-wei and Lin, Yang-Hsien and Lee, Cheng-Kuang and Huang, Eddie TC and See, Simon, Omni-FinAI: Unlocking Financial Disclosure Insights (October 30, 2024). Available at SSRN: https://ssrn.com/abstract=5004298