metadata

license: mit
language:
  - en
pipeline_tag: text-generation
tags:
  - llama-3
  - astronomy
  - astrophysics
  - arxiv
inference: false
base_model:
  - meta-llama/Llama-3-8b-hf

AstroLLaMA-3-8B-Base_Summary

AstroLLaMA-3-8B-Base_Summary is a specialized base language model for astronomy, developed by fine-tuning Meta's LLaMA-3-8b architecture on summarized astronomical literature. This model was developed by the AstroMLab team. It is designed for next token prediction tasks and is not an instruct/chat model.

Model Details

Base Architecture: LLaMA-3-8b
Training Data: Summarized content from arXiv's astro-ph category papers
Data Processing:
1. Optical character recognition (OCR) on PDF files using the Nougat tool
2. Summarization of OCR'd text using Qwen-2-8B and LLaMA-3.1-8B, reducing content to about 1,000-4,000 tokens per paper
Fine-tuning Method: Continual Pre-Training (CPT) using the LMFlow framework
Training Details:
- Learning rate: 2 × 10⁻⁵
- Total batch size: 96
- Maximum token length: 512
- Warmup ratio: 0.03
- No gradient accumulation
- BF16 format
- Cosine decay schedule for learning rate reduction
- Training duration: 1 epoch
Primary Use: Next token prediction for astronomy-related text generation and analysis
Reference: Pan et al. 2024 [Link to be added]

Generating text from a prompt

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load the model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("AstroMLab/astrollama-3-8b-base_summary")
model = AutoModelForCausalLM.from_pretrained("AstroMLab/astrollama-3-8b-base_summary", device_map="auto")

# Create the pipeline with explicit truncation
from transformers import pipeline
generator = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    device_map="auto",
    truncation=True,
    max_length=512
)

# Example prompt from an astronomy paper
prompt = "In this letter, we report the discovery of the highest redshift, " \
    "heavily obscured, radio-loud QSO candidate selected using JWST NIRCam/MIRI, " \
    "mid-IR, sub-mm, and radio imaging in the COSMOS-Web field. "

# Set seed for reproducibility
torch.manual_seed(42)

# Generate text
generated_text = generator(prompt, do_sample=True)
print(generated_text[0]['generated_text'])

Model Improvements and Performance

A key innovation in this model is the use of summarized content for training, which has led to improved performance compared to the AIC (Abstract, Introduction, Conclusion) version. The summarization process allows for the inclusion of more comprehensive information from each paper while maintaining a manageable token count.

Here's a performance comparison chart based upon the astronomical benchmarking Q&A as described in Ting et al. 2024, and Pan et al. 2024:

Model	Score (%)
LLaMA-3.1-8B	73.7
AstroLLaMA-3-8B-Base_Summary (AstroMLab)	72.3
LLaMA-3-8B	72.0
AstroLLaMA-3-8B-Base_AIC	72.3
Gemma-2-9B	71.5
Qwen-2.5-7B	70.4
Yi-1.5-9B	68.4
InternLM-2.5-7B	64.0
Mistral-7B-v0.3	63.9
ChatGLM3-6B	50.4

As shown, AstroLLaMA-3-8B-Base_Summary performs competitively, nearly matching the performance of the base LLaMA-3.1-8B model and outperforming the AIC version. This improvement demonstrates the importance of information density in the training data.

Notably, the instruct version of this model shows even more significant improvements, highlighting the effectiveness of the summarization approach in capturing and retaining key astronomical concepts. For detailed performance analysis of the instruct version, please refer to Pan et al. 2024.

While AstroLLaMA-3-8B performs competitively among models in its class, it does not surpass the performance of the base LLaMA-3-8B model. This underscores the challenges in developing specialized models and the need for more diverse and comprehensive training data.

It's worth noting that the AstroLLaMA-3-8B-Plus which we will release in the next model release addresses these limitations by expanding beyond astro-ph data.

Ethical Considerations

While this model is designed for scientific use, users should be mindful of potential misuse, such as generating misleading scientific content. Always verify model outputs against peer-reviewed sources for critical applications.

Citation

If you use this model in your research, please cite:

[Citation for Pan et al. 2024 to be added]