metadata

license: apache-2.0
pipeline_tag: text-generation
language:
  - en
tags:
  - pretrained
datasets:
  - Skylion007/openwebtext
  - c4
  - wikimedia/wikipedia
  - tiiuae/falcon-refinedweb
  - izumi-lab/open-text-books
  - togethercomputer/RedPajama-Data-V2
  - databricks/databricks-dolly-15k
  - euclaise/reddit-instruct-curated
  - CohereForAI/aya_dataset
widget:
  - text: |-
      Let me tell you a short story:
      Once upon a time,
  - text: |-
      Chapter 1: About Quantum Computing

      Quantum computing has the potential to
  - text: Hi! I'm
  - text: The story takes place in
  - text: Reducing waste generation is essential to
  - text: The best way to improve your health is
inference:
  parameters:
    max_new_tokens: 64
    do_sample: true
    temperature: 0.72
    top_p: 0.73
    top_k: 50
    repetition_penalty: 1.176

Minueza-32M-Base

Summary

Minueza-32M-Base is a foundation model with 32 million parameters trained from scratch.

The following versions of this model are available:

Minueza-32M-Base: The base model, trained from scratch on a large corpus of text in English.
Minueza-32M-Chat: A version of the base model fine-tuned on conversational datasets.

Intended Uses

This model was created with the following objectives in mind:

Run on mobile web browsers via Transformers.js.
Run fast on machines without GPU.
Serve as a base for fine-tunes using ChatML format, hence the two additional special tokens (<|im_start|> and <|im_end|>) with <|im_end|> as default EOS token.
- ChatML works great for both instruction and chat models, so if all fine-tunes are made following the ChatML pattern, other users might benefit from the easiness of creating merges.

Datasets

The model was trained on a subset of each of the following non-synthetic datasets:

The subsets were interleaved to form the final training corpus of approximately 650 million tokens.

Usage

This is a pre-trained foundation model. For your task, you will likely want to perform application-specific fine-tuning.

Also note that this model was trained on internet text data, which may contain biases, offensive or inappropriate content, and may produce incorrect or irrelevant responses. No evaluation has been conducted, so use with care.

Having that said, here's how you can run it:

from transformers import pipeline

generate = pipeline("text-generation", "Felladrin/Minueza-32M-Base")

prompt = "The best way to improve your health is"

output = generate(
    prompt,
    max_new_tokens=256,
    do_sample=True,
    temperature=0.72,
    top_p=0.73,
    top_k=50,
    repetition_penalty=1.176,
)

print(output[0]["generated_text"])

Model Architecture

Trained on a context window of 2048 tokens, this is a transformer model with the Mistral architecture, which includes Grouped-Query Attention, Sliding-Window Attention, and Byte-fallback BPE tokenizer.

Configuration	Value
max_position_embeddings	2048
hidden_size	312
intermediate_size	1092
num_attention_heads	12
num_hidden_layers	10
num_key_value_heads	4
vocab_size	32002

Training

Hyperparameter	Value
learning_rate	5e-05
train_batch_size	1
eval_batch_size	1
seed	42
gradient_accumulation_steps	8
total_train_batch_size	8
optimizer	Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type	linear
num_epochs	1.0

Framework	Version
Transformers	4.38.0.dev0
Pytorch	2.1.2
Datasets	2.16.1
Tokenizers	0.15.1

License

This model is licensed under the Apache License 2.0.