metadata

license: other
license_name: rwkv-decepticon
license_link: LICENSE
datasets:
  - roneneldan/TinyStories
language:
  - en

Dataset

This model was trained using the TinyStories dataset, specifically with the GPT-4 version.

The Model

The name "Decepticon" stems from the model's unique architecture, which combines elements of both Transformer and RNN architechtures. This fusion creates a deceptive yet beneficial design.

The model features a context length of 1024, but in theory, it can be extended indefinitely through fine-tuning.

rwkv-decepticon-char-20m.pth is to be used with vocab.json. This is a character level model.
rwkv-decepticon-70m.pth (coming soon) is to be used with 20B_tokenizer.json.
rwkv-decepticon-170m.pth (coming soon) is trained on a small subset of the SlimPajama dataset (6gb). This also uses the 20B_tokenizer.json file.

I would like to train a 7B parameter model but lack the compute required. If you would like to sponsor some compute, please contact me.

Thank you to the creators of RWKV who made all of this possible. Their repo is here: https://github.com/BlinkDL/RWKV-LM