File size: 1,650 Bytes
7a9ace6 cccf815 7a9ace6 cccf815 7a9ace6 cccf815 512186e 39379db be0946a 39379db ff9ea24 944ae40 dd2f5c9 6f726bb 4ccf1d2 3957ef8 a6b6a11 5f88540 1e12a22 aee797f 512186e 0616e70 a6b6a11 30457d0 ab45047 34d8092 ab45047 30457d0 ab45047 30457d0 ab45047 34d8092 30457d0 1067d68 06dc12e 1e12a22 aee797f 1e12a22 aee797f 1e12a22 aee797f 1e12a22 06dc12e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 |
---
language:
- en
- zh
- de
- fr
- es
- pt
- ru
- it
- ja
- ko
- vi
- ar
tags:
- pytorch
- text-generation
- causal-lm
- rwkv
license: apache-2.0
datasets:
- EleutherAI/pile
- togethercomputer/RedPajama-Data-1T
---
# RWKV-4 World
## Model Description
RWKV-4 trained on 100+ world languages (70% English, 15% multilang, 15% code).
Some_Pile + Some_RedPajama + Some_OSCAR + All_Wikipedia + All_ChatGPT_Data_I_can_find
How to use:
* use latest rwkv pip package (0.7.4+)
* use https://github.com/BlinkDL/ChatRWKV/blob/main/v2/benchmark_world.py to test it
* larger models are stronger even though not fully trained yet
The differences between World & Raven:
* set pipeline = PIPELINE(model, "rwkv_vocab_v20230424") instead of 20B_tokenizer.json (EXACTLY AS WRITTEN HERE. "rwkv_vocab_v20230424" is included in rwkv 0.7.4+)
* use Question/Answer or User/AI or Human/Bot for chat. **DO NOT USE Bob/Alice or Q/A**
For 0.1/0.4/1.5B models, use **fp32** for first layer (will overflow in fp16 at this moment - fixable in future), or bf16 if you have 30xx/40xx GPUs. Example strategy: cuda fp32 *1 -> cuda fp16
NOTE: the new greedy tokenizer (https://github.com/BlinkDL/ChatRWKV/blob/main/tokenizer/rwkv_tokenizer.py) will tokenize '\n\n' as one single token instead of ['\n','\n']
QA prompt (replace \n\n in xxx to \n):
```
Question: xxx
Answer:
```
and
```
Instruction: xxx
Input: xxx
Response:
```
A good chat prompt (replace \n\n in xxx to \n):
```
User: hi
Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it.
User: xxx
Assistant:
``` |