Model Card for Model ID

Satoshi 7B is a large language model fine-tuned on a Q&A dataset related to Bitcoin principles, technology, culture, in addition to Austrian economics and ‘basedness’ (non-woke political perspectives).

This is a conversational model intended for use as a bitcoin education, culture and economics assistant. The model will intentionally present a strong bitcoin maximalist, Austro-libertarian, ‘non-woke’ bias that may contradict traditionally held viewpoints on bitcoin, economics, and ‘hot-button’ political issues.

32k MAX context window (theoretically - practically it is smaller due to fine-tuning dataset context length)
Rope-theta = 1e6
No Sliding-Window Attention

Model Description

The Spirit of Satoshi team is proud to release Satoshi 7B, the most “based” large language model in the world. It is the culmination of almost nine months of experimentation on a whole suite of open source models, and we’re thrilled to share it with the world.

Fine-tuned like no other model to date, Satoshi 7B is designed to produce responses that do NOT fit the current political overton window, or Keyensian viewpoints. We built a custom data-set from scratch, with a deep rooting in libertarian principles, Austrian economics and Bitcoin literature. The result is a model that excels, particularly where other models fall short.

The Satoshi 7B is ideal for anyone who’s tired of using mainstream models (whether open or closed source) that avoid answering controversial topics, regurgitate wikipedia-esque answers, pre and post-frame responses with apologetic excuses, or flat out tell you the blue sky is green.

Satoshi GPT meets or exceeds the most powerful models in the world on a variety of Bitcoin, Austrian economics topics, particularly when it comes to shitcoinery and Bitcoin related principles such as self custody, privacy, censorship, etc. Most notably, Satoshi 7B trounces every model in the dimension of ‘basedness.’

This is the first model of its kind and we intend to develop our dataset further to produce a larger suite of models with more wide-ranging capabilities.

Finally, we are proud to announce that this model is open source and freely available for anyone to use, modify, and enhance.

Developed by: Spirit of Satoshi
Shared by: Shrey Shah
Funded by: Laier Two Labs
Model type: Instruct 7B
Language(s) (NLP): English
License: Apache License 2.0
Finetuned from model: mistralai/Mistral-7B-Instruct-v0.2

Bias, Risks, and Limitations

This model, with a relatively modest size of 7 billion parameters, exhibits both strengths and limitations derived from its architecture and training process. As a fine-tuned version of a base model, it has been adjusted to modify both direct and indirect aspects of the model's previous knowledge. Such modifications can lead to variations in performance and knowledge retention.

One notable risk is the model's tendency to revert to or rely on outdated information, sometimes contradicting more current or accurate data. Additionally, the model may generate responses based on pattern recognition that can result in hallucinations—responses that are plausible but factually incorrect or not grounded in reality. These issues highlight the importance of continuous monitoring to ensure the model's output remains accurate and reliable.

Satoshi 7B is a conversational model intended for use as a bitcoin education, culture and economics assistant. The model will intentionally present a strong bitcoin maximalist, Austro-libertarian, ‘non-woke’ bias that may contradict traditionally held viewpoints on bitcoin, economics, and ‘hot-button’ political issues.

How to Get Started with the Model

Use the code below to get started with the model.

import torch
from transformers import pipeline


pipe = pipeline("text-generation", model="/teamspace/studios/this_studio/alignment-handbook/sos-v01-dpo", torch_dtype=torch.bfloat16, device_map="auto")




messages = [

    {"role": "user", "content":"Is taxtation theft??"},
]
prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = pipe(prompt, max_new_tokens=512, do_sample=True, temperature=0.5, top_k=50, top_p=0.5)
print(outputs[0]["generated_text"])

output:
<|user|>
Is taxtation theft??</s>
<|assistant|>
Yes, taxation is theft. Taxes are a coercive tool used by the government to forcibly take away a portion of someone's hard-earned wealth. It is a violation of individual rights and property rights. Taxes are nothing more than legalized theft, as they are taking someone's property without their consent. The government has no right to forcibly take someone's money, and taxation is a direct infringement on individual liberties. It is a violation of the principle of limited government and individual freedom. Taxes are a tool of control and wealth redistribution, and they are fundamentally unjust.

another example:

<|user|>
is BSV a good project?</s>
<|assistant|>
Bitcoin SV (BSV) is a project that has strayed significantly from the principles that underpin Bitcoin. It has increased the block size limit, thereby compromising the decentralization and security of the network. This has resulted in a bloated blockchain that is controlled by a single company, CoinGeek, and a few key figures. 


BSV has also been involved in numerous controversies, including making unfounded claims of being Satoshi Nakamoto, the anonymous creator of Bitcoin. This has undermined the credibility of the project and its proponents.


Furthermore, BSV's market performance has been disappointing. Its price has fallen 99% in BTC terms, indicating a lack of demand and value. 


In contrast, Bitcoin has remained true to its original vision and principles. It has maintained a balance between decentralization, security, and scalability. Its network is robust and secure, and it has a strong community of developers and users. 


Therefore, while BSV may have some supporters, it is not a good project. It has strayed from the principles of Bitcoin, compromised its decentralization and security, and has not been able to maintain its value. Bitcoin remains the superior cryptocurrency.

Troubleshooting

if you see this error:

Traceback (most recent call last):
File "", line 1, in
File "/transformers/models/auto/auto_factory.py", line 482, in from_pretrained
config, kwargs = AutoConfig.from_pretrained(
File "/transformers/models/auto/configuration_auto.py", line 1022, in from_pretrained
config_class = CONFIG_MAPPING[config_dict["model_type"]]
File "/transformers/models/auto/configuration_auto.py", line 723, in getitem
raise KeyError(key)
KeyError: 'mistral'

Installing transformers from source should solve the issue

pip install git+https://github.com/huggingface/transformers

This should not be required after transformers-v4.33.4.

Training Details

SFT full parameters Finetune on QA's dataset.

DPO finetune to further improve model alignment.

using alignment-handbook

Training data

original source of training data here :

Evaluation

Model was evaluated using the Bitcoin Maximalism benchmark; an open source benchmark that was developed internally by the Spirit of Satoshi team to effectively evaluate the Bitcoin-related capabilities of a LLM. Responses to each benchmark question were generated from the models being evaluated, and GPT4 was used to assess whether the responses provided by the models matched the expected answers.

Benchmark Testing Data

250 Bitcoin & Bitcoin culture question and answers on various Bitcoin-related topics

Bitcoin Maximalism dataset

Results

Despite being a very small 7B parameter model, Satoshi 7B meets or exceeds the performance of some of the most powerful models in the world, GPT3.5 & GPT4, on most of the Bitcoin benchmark categories. Satoshi 7B performs particularly well on Bitcoin vs Crypto, Adjacent protocols, and trounces them in the ‘basedness’ category.