Add files using upload-large-folder tool

README.md

@@ -0,0 +1,208 @@
+---
+license: apache-2.0
+---
+### Huggingface RWKV v6 7B Model
+
+> HF compatible model for v6-7B.
+
+![v6 Bird](./imgs/v6.jpg)
+
+
+> **! Important Note !**
+>
+> The following is the HF transformers implementation of the v6 7B model. This is meant to be used with the huggingface transformers
+>
+>
+
+
+## Quickstart with the hugging face transformer library
+
+```
+model = AutoModelForCausalLM.from_pretrained("zhiyuan8/RWKV-v6-7B-World-v3-GGUF", trust_remote_code=True).to(torch.float32)
+tokenizer = AutoTokenizer.from_pretrained("zhiyuan8/RWKV-v6-7B-World-v3-GGUF", trust_remote_code=True)
+```
+
+
+#### Running on CPU via HF transformers
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+def generate_prompt(instruction, input=""):
+    instruction = instruction.strip().replace('\r\n','\n').replace('\n\n','\n')
+    input = input.strip().replace('\r\n','\n').replace('\n\n','\n')
+    if input:
+        return f"""Instruction: {instruction}
+
+Input: {input}
+
+Response:"""
+    else:
+        return f"""User: hi
+
+Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it.
+
+User: {instruction}
+
+Assistant:"""
+
+
+model = AutoModelForCausalLM.from_pretrained("zhiyuan8/RWKV-v6-7B-World-v3-GGUF", trust_remote_code=True).to(torch.float32)
+tokenizer = AutoTokenizer.from_pretrained("zhiyuan8/RWKV-v6-7B-World-v3-HF", trust_remote_code=True)
+
+text = "请介绍北京的旅游景点"
+prompt = generate_prompt(text)
+
+inputs = tokenizer(prompt, return_tensors="pt")
+output = model.generate(inputs["input_ids"], max_new_tokens=333, do_sample=True, temperature=1.0, top_p=0.3, top_k=0, )
+print(tokenizer.decode(output[0].tolist(), skip_special_tokens=True))
+```
+
+output:
+
+```shell
+User: hi
+
+Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it.
+
+User: 请介绍北京的旅游景点
+
+Assistant: 北京是中国的首都，拥有众多的旅游景点，以下是其中一些著名的景点：
+1. 故宫：位于北京市中心，是明清两代的皇宫，内有大量的文物和艺术品。
+2. 天安门广场：是中国最著名的广场之一，是中国人民政治协商会议的旧址，也是中国人民政治协商会议的中心。
+3. 颐和园：是中国古代皇家园林之一，有着悠久的历史和丰富的文化内涵。
+4. 长城：是中国古代的一道长城，全长约万里，是中国最著名的旅游景点之一。
+5. 北京大学：是中国著名的高等教育机构之一，有着悠久的历史和丰富的文化内涵。
+6. 北京动物园：是中国最大的动物园之一，有着丰富的动物资源和丰富的文化内涵。
+7. 故宫博物院：是中国最著名的博物馆之一，收藏了大量的文物和艺术品，是中国最重要的文化遗产之一。
+8. 天坛：是中国古代皇家
+```
+
+#### Running on GPU via HF transformers
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+def generate_prompt(instruction, input=""):
+    instruction = instruction.strip().replace('\r\n','\n').replace('\n\n','\n')
+    input = input.strip().replace('\r\n','\n').replace('\n\n','\n')
+    if input:
+        return f"""Instruction: {instruction}
+
+Input: {input}
+
+Response:"""
+    else:
+        return f"""User: hi
+
+Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it.
+
+User: {instruction}
+
+Assistant:"""
+
+
+model = AutoModelForCausalLM.from_pretrained("zhiyuan8/RWKV-v6-7B-World-v3-HF", trust_remote_code=True, torch_dtype=torch.float16).to(0)
+tokenizer = AutoTokenizer.from_pretrained("zhiyuan8/RWKV-v6-7B-World-v3-HF", trust_remote_code=True)
+
+text = "介绍一下大熊猫"
+prompt = generate_prompt(text)
+
+inputs = tokenizer(prompt, return_tensors="pt").to(0)
+output = model.generate(inputs["input_ids"], max_new_tokens=128, do_sample=True, temperature=1.0, top_p=0.3, top_k=0, )
+print(tokenizer.decode(output[0].tolist(), skip_special_tokens=True))
+```
+
+output:
+
+```shell
+User: hi
+
+Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it.
+
+User: 介绍一下大熊猫
+
+Assistant: 大熊猫是一种中国特有的哺乳动物，也是中国的国宝之一。它们的外貌特征是圆形的黑白相间的身体，有着黑色的毛发和白色的耳朵。大熊猫的食物主要是竹子，它们会在竹林中寻找竹子，并且会将竹子放在竹笼中进行储存。大熊猫的寿命约为20至30年，但由于栖息地的丧失和人类活动的
+```
+
+#### Batch Inference
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+def generate_prompt(instruction, input=""):
+    instruction = instruction.strip().replace('\r\n', '\n').replace('\n\n', '\n')
+    input = input.strip().replace('\r\n', '\n').replace('\n\n', '\n')
+    if input:
+        return f"""Instruction: {instruction}
+
+Input: {input}
+
+Response:"""
+    else:
+        return f"""User: hi
+
+Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it.
+
+User: {instruction}
+
+Assistant:"""
+
+model = AutoModelForCausalLM.from_pretrained("zhiyuan8/RWKV-v6-7B-World-v3-HF", trust_remote_code=True).to(torch.float32)
+tokenizer = AutoTokenizer.from_pretrained("zhiyuan8/RWKV-v6-7B-World-v3-HF", trust_remote_code=True)
+
+texts = ["请介绍北京的旅游景点", "介绍一下大熊猫", "乌兰察布"]
+prompts = [generate_prompt(text) for text in texts]
+
+inputs = tokenizer(prompts, return_tensors="pt", padding=True)
+outputs = model.generate(inputs["input_ids"], max_new_tokens=128, do_sample=True, temperature=1.0, top_p=0.3, top_k=0, )
+
+for output in outputs:
+    print(tokenizer.decode(output.tolist(), skip_special_tokens=True))
+
+```
+
+output:
+
+```shell
+User: hi
+
+Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it.
+
+User: 请介绍北京的旅游景点
+
+Assistant: 北京是中国的首都，拥有丰富的旅游资源和历史文化遗产。以下是一些北京的旅游景点：
+1. 故宫：位于北京市中心，是明清两代的皇宫，是中国最大的古代宫殿建筑群之一。
+2. 天安门广场：位于北京市中心，是中国最著名的城市广场之一，也是中国最大的城市广场。
+3. 颐和
+User: hi
+
+Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it.
+
+User: 介绍一下大熊猫
+
+Assistant: 大熊猫是一种生活在中国中部地区的哺乳动物，也是中国的国宝之一。它们的外貌特征是圆形的黑白相间的身体，有着黑色的毛发和圆圆的眼睛。大熊猫是一种濒危物种，目前只有在野外的几个保护区才能看到它们的身影。大熊猫的食物主要是竹子，它们会在竹子上寻找食物，并且可以通
+User: hi
+
+Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it.
+
+User: 乌兰察布
+
+Assistant: 乌兰察布是中国新疆维吾尔自治区的一个县级市，位于新疆维吾尔自治区中部，是新疆的第二大城市。乌兰察布市是新疆的第一大城市，也是新疆的重要城市之一。乌兰察布市是新疆的经济中心，也是新疆的重要交通枢纽之一。乌兰察布市的人口约为2.5万人，其中汉族占绝大多数。乌
+```
+
+## Links
+- [Our wiki](https://wiki.rwkv.com)
+- [Recursal.AI Cloud Platform](https://recursal.ai)
+- [Featherless Inference](https://featherless.ai/models/RWKV/v6-14B)
+- [Blog article, detailing our model launch](https://blog.rwkv.com/p/rwkv-v6-v6-14b-is-here)
+
+## Acknowledgement
+We are grateful for the help and support from the following key groups:
+
+- [Recursal.ai](https://recursal.ai) team for financing the GPU resources, and managing the training of this foundation model - you can run the v6 line of RWKV models on their cloud / on-premise platform today.
+- EleutherAI for their support, especially in the v5/v6 Eagle/v6 paper
+- Linux Foundation AI & Data group for supporting and hosting the RWKV project

added_tokens.json

@@ -0,0 +1,3 @@
+{
+  "<s>": 0
+}

config.json

@@ -0,0 +1,25 @@
+{
+  "architectures": [
+    "Rwkv6ForCausalLM"
+  ],
+  "auto_map": {
+      "AutoConfig": "configuration_rwkv6.Rwkv6Config",
+      "AutoModelForCausalLM": "modeling_rwkv6.Rwkv6ForCausalLM"
+  },
+  "attention_hidden_size": 4096,
+  "bos_token_id": 0,
+  "eos_token_id": 0,
+  "head_size": 64,
+  "head_size_divisor": 8,
+  "hidden_size": 4096,
+  "intermediate_size": null,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "rwkv6",
+  "num_attention_heads": 64,
+  "num_hidden_layers": 32,
+  "rescale_every": 6,
+  "tie_word_embeddings": false,
+  "transformers_version": "4.34.0",
+  "use_cache": true,
+  "vocab_size": 65536
+}

configuration_rwkv6.py

@@ -0,0 +1,118 @@
+# coding=utf-8
+# Copyright 2023 The OpenAI Team Authors and HuggingFace Inc. team.
+# Copyright (c) 2018, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" RWKV configuration"""
+
+from transformers.configuration_utils import PretrainedConfig
+from transformers.utils import logging
+
+
+logger = logging.get_logger(__name__)
+
+RWKV6_PRETRAINED_CONFIG_ARCHIVE_MAP = {}
+
+
+class Rwkv6Config(PretrainedConfig):
+    """
+    This is the configuration class to store the configuration of a [`Rwkv6Model`]. It is used to instantiate a RWKV6
+    model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
+    defaults will yield a similar configuration to that of the RWVK-4
+    [RWKV/rwkv-5-world-1b5](https://huggingface.co/RWKV/rwkv-5-world-1b5) architecture.
+
+    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
+    documentation from [`PretrainedConfig`] for more information.
+
+
+    Args:
+        vocab_size (`int`, *optional*, defaults to 65536):
+            Vocabulary size of the RWKV6 model. Defines the number of different tokens that can be represented by the
+            `inputs_ids` passed when calling [`Rwkv6Model`].
+        hidden_size (`int`, *optional*, defaults to 768):
+            Dimensionality of the embeddings and hidden states.
+        num_hidden_layers (`int`, *optional*, defaults to 24):
+            Number of hidden layers in the model.
+        attention_hidden_size (`int`, *optional*):
+            Dimensionality of the attention hidden states. Will default to `hidden_size` if unset.
+        num_attention_heads (`int`, *optional*, defaults to 64):
+            The attention heads to use in rwkv6 self_attention module.
+        head_size (`int`, *optional*, defaults to 64): head_size of rwkv6 self_attention module.
+        intermediate_size (`int`, *optional*):
+            Dimensionality of the inner feed-forward layers. Will default to 4 times `hidden_size` if unset.
+        layer_norm_epsilon (`float`, *optional*, defaults to 1e-05):
+            The epsilon to use in the layer normalization layers.
+        bos_token_id (`int`, *optional*, defaults to 0):
+            The id of the beginning of sentence token in the vocabulary. Defaults to 0.
+        eos_token_id (`int`, *optional*, defaults to 0):
+            The id of the end of sentence token in the vocabulary. Defaults to 0.
+        rescale_every (`int`, *optional*, defaults to 6):
+            At inference, the hidden states (and weights of the correponding output layers) are divided by 2 every
+            `rescale_every` layer. If set to 0 or a negative number, no rescale is done.
+        tie_word_embeddings (`bool`, *optional*, defaults to `False`):
+            Whether or not to tie the word embeddings with the input token embeddings.
+        use_cache (`bool`, *optional*, defaults to `True`):
+            Whether or not the model should return the last state.
+
+
+    Example:
+
+    ```python
+    >>> from transformers import Rwkv6Config, Rwkv6Model
+
+    >>> # Initializing a Rwkv6 configuration
+    >>> configuration = Rwkv6Config()
+
+    >>> # Initializing a model (with random weights) from the configuration
+    >>> model = Rwkv6Model(configuration)
+
+    >>> # Accessing the model configuration
+    >>> configuration = model.config
+    ```"""
+
+    model_type = "rwkv6"
+
+    def __init__(
+        self,
+        vocab_size=65536,
+        hidden_size=768,
+        num_hidden_layers=24,
+        attention_hidden_size=None,
+        head_size=64,
+        head_size_divisor=8,
+        intermediate_size=None,
+        layer_norm_epsilon=1e-5,
+        bos_token_id=0,
+        eos_token_id=0,
+        rescale_every=6,
+        tie_word_embeddings=False,
+        use_cache=True,
+        **kwargs,
+    ):
+        self.vocab_size = vocab_size
+        self.hidden_size = hidden_size
+        self.num_hidden_layers = num_hidden_layers
+        self.attention_hidden_size = attention_hidden_size if attention_hidden_size is not None else hidden_size
+        self.head_size = head_size
+        self.head_size_divisor = head_size_divisor
+        self.intermediate_size = None
+        self.layer_norm_epsilon = layer_norm_epsilon
+        self.rescale_every = rescale_every
+        self.use_cache = use_cache
+
+        self.bos_token_id = bos_token_id
+        self.eos_token_id = eos_token_id
+
+        super().__init__(
+            tie_word_embeddings=tie_word_embeddings, bos_token_id=bos_token_id, eos_token_id=eos_token_id, **kwargs
+        )

generation_config.json

@@ -0,0 +1,12 @@
+{
+    "chat_format": "chatml",
+    "eos_token_id": 0,
+    "pad_token_id": 0,
+    "max_window_size": 4096,
+    "max_new_tokens": 4096,
+    "do_sample": true,
+    "top_k": 0,
+    "top_p": 0.1,
+    "repetition_penalty": 1.0,
+    "transformers_version": "4.31.1"
+}

hf_rwkv_tokenizer.py

@@ -0,0 +1,279 @@
+# coding=utf-8
+# Copyright 2024 The HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Tokenization classes for RWKV6."""
+
+import os
+import re
+from typing import TYPE_CHECKING, List, Optional, Tuple
+
+from transformers.tokenization_utils import AddedToken, PreTrainedTokenizer
+from transformers.utils import logging
+
+
+if TYPE_CHECKING:
+    pass
+
+logger = logging.get_logger(__name__)
+
+
+VOCAB_FILES_NAMES = {
+    "vocab_file": "rwkv_vocab_v20230424.txt",
+}
+
+class TRIE:
+    __slots__ = tuple("ch,to,values,front".split(","))
+    to: list
+    values: set
+
+    def __init__(self, front=None, ch=None):
+        self.ch = ch
+        self.to = [None for ch in range(256)]
+        self.values = set()
+        self.front = front
+
+    def __repr__(self):
+        fr = self
+        ret = []
+        while fr != None:
+            if fr.ch != None:
+                ret.append(fr.ch)
+            fr = fr.front
+        return "<TRIE %s %s>" % (ret[::-1], self.values)
+
+    def add(self, key: bytes, idx: int = 0, val=None):
+        if idx == len(key):
+            if val is None:
+                val = key
+            self.values.add(val)
+            return self
+        ch = key[idx]
+        if self.to[ch] is None:
+            self.to[ch] = TRIE(front=self, ch=ch)
+        return self.to[ch].add(key, idx=idx + 1, val=val)
+
+    def find_longest(self, key: bytes, idx: int = 0):
+        u: TRIE = self
+        ch: int = key[idx]
+
+        while u.to[ch] is not None:
+            u = u.to[ch]
+            idx += 1
+            if u.values:
+                ret = idx, u, u.values
+            if idx == len(key):
+                break
+            ch = key[idx]
+        return ret
+
+
+class RWKV_TOKENIZER:
+    def __init__(self, file_name):
+        self.idx2token = {}
+        sorted = []  # must be already sorted
+        with open(file_name, "r", encoding="utf-8") as f:
+            lines = f.readlines()
+        for l in lines:
+            idx = int(l[: l.index(" ")])
+            x = eval(l[l.index(" ") : l.rindex(" ")])
+            x = x.encode("utf-8") if isinstance(x, str) else x
+            assert isinstance(x, bytes)
+
+            assert len(x) == int(l[l.rindex(" ") :])
+            sorted += [x]
+            self.idx2token[idx] = x
+
+        self.token2idx = {}
+        for k, v in self.idx2token.items():
+            self.token2idx[v] = int(k)
+
+        self.root = TRIE()
+        for t, i in self.token2idx.items():
+            _ = self.root.add(t, val=(t, i))
+
+    def encodeBytes(self, src: bytes):
+        idx: int = 0
+        tokens = []
+        while idx < len(src):
+            _idx: int = idx
+            idx, _, values = self.root.find_longest(src, idx)
+            assert idx != _idx
+            _, token = next(iter(values))
+            tokens.append(token)
+        return tokens
+
+    def decodeBytes(self, tokens):
+        return b"".join(map(lambda i: self.idx2token[i], tokens))
+
+    def encode(self, src):
+        if isinstance(src, str):
+            return [self.encodeBytes(src.encode("utf-8"))]
+        elif isinstance(src, list):
+            return [self.encodeBytes(s.encode("utf-8")) for s in src]
+
+    def decode(self, tokens):
+        return [self.decodeBytes(batch).decode("utf-8") for batch in tokens]
+        # try:
+        #     return self.decodeBytes(tokens).decode('utf-8')
+        # except:
+        #     return '\ufffd' # bad utf-8
+
+    def printTokens(self, tokens):
+        for i in tokens:
+            s = self.idx2token[i]
+            try:
+                s = s.decode("utf-8")
+            except:
+                pass
+            print(f"{repr(s)}{i}", end=" ")
+        print()
+
+
+class Rwkv6Tokenizer(PreTrainedTokenizer):
+    vocab_files_names = VOCAB_FILES_NAMES
+    model_input_names = ["input_ids", "attention_mask"]
+
+    def __init__(
+        self, vocab_file, bos_token="<s>", eos_token="<s>", unk_token="<s>", **kwargs
+    ):
+        if not os.path.isfile(vocab_file):
+            raise ValueError(
+                f"Can't find a vocabulary file at path '{vocab_file}'. To load the vocabulary from a Google pretrained"
+                " model use `tokenizer = BertTokenizer.from_pretrained(PRETRAINED_MODEL_NAME)`"
+            )
+
+        with open(vocab_file, "r", encoding="utf-8") as reader:
+            tokens = reader.readlines()
+
+        if "add_bos_token" in kwargs:
+            self.add_bos_token = kwargs["add_bos_token"]
+        else:
+            self.add_bos_token = False
+        self.trie_tokenizer = RWKV_TOKENIZER(vocab_file)
+        vocab = self.trie_tokenizer.token2idx
+        self.encoder = vocab
+        self.decoder = {v: k for k, v in vocab.items()}
+        self._added_tokens_decoder = {0: AddedToken(str(bos_token))}
+        super().__init__(
+            bos_token=bos_token, eos_token=eos_token, unk_token=unk_token, **kwargs
+        )
+
+    @property
+    def vocab_size(self):
+        return len(self.encoder)
+
+    def get_vocab(self):
+        vocab = {str(self.convert_ids_to_tokens(i)): i for i in range(self.vocab_size)}
+        vocab.update(self.added_tokens_encoder)
+        return vocab
+
+    def _tokenize(self, text, split_special_tokens=False):
+        # return self.wordpiece_tokenizer.tokenize(text.encode("utf-8"))
+        return self.trie_tokenizer.encode(text)[0]
+
+    def _convert_token_to_id(self, token):
+        return token
+
+    def _convert_id_to_token(self, index):
+        """Converts an index (integer) in a token (byte) using the vocab."""
+        token = self.decoder.get(index, self.unk_token)
+        if isinstance(token, (bytes)):
+            token = token.decode("utf-8", errors="replace")
+        return token
+
+    def convert_tokens_to_string(self, tokens):
+        """Converts a sequence of tokens (bytes) in a single string. Additional tokens are encoded to bytes"""
+        out_string = b"".join(
+            [k.encode(errors="replace") if isinstance(k, str) else k for k in tokens]
+        ).decode("utf-8")
+        return out_string
+
+    def save_vocabulary(
+        self, save_directory: str, filename_prefix: Optional[str] = None
+    ) -> Tuple[str]:
+        index = 0
+        if os.path.isdir(save_directory):
+            vocab_file = os.path.join(
+                save_directory,
+                (filename_prefix + "-" if filename_prefix else "") + "vocab.txt",
+            )
+        else:
+            vocab_file = (
+                filename_prefix + "-" if filename_prefix else ""
+            ) + save_directory
+        with open(vocab_file, "w", encoding="utf-8") as writer:
+            for token, token_index in sorted(
+                self.encoder.items(), key=lambda kv: kv[1]
+            ):
+                if index != token_index:
+                    logger.warning(
+                        f"Saving vocabulary to {vocab_file}: vocabulary indices are not consecutive."
+                        " Please check that the vocabulary is not corrupted!"
+                    )
+                    index = token_index
+                writer.write(str(token) + "\n")
+                index += 1
+        return (vocab_file,)
+
+    def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None):
+        if self.add_bos_token:
+            bos_token_ids = [self.bos_token_id]
+        else:
+            bos_token_ids = []
+
+        output = bos_token_ids + token_ids_0
+
+        if token_ids_1 is None:
+            return output
+
+        return output + bos_token_ids + token_ids_1
+
+    def get_special_tokens_mask(
+        self,
+        token_ids_0: List[int],
+        token_ids_1: Optional[List[int]] = None,
+        already_has_special_tokens: bool = False,
+    ) -> List[int]:
+        """
+        Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding
+        special tokens using the tokenizer `prepare_for_model` or `encode_plus` methods.
+
+        Args:
+            token_ids_0 (`List[int]`):
+                List of IDs.
+            token_ids_1 (`List[int]`, *optional*):
+                Optional second list of IDs for sequence pairs.
+            already_has_special_tokens (`bool`, *optional*, defaults to `False`):
+                Whether or not the token list is already formatted with special tokens for the model.
+
+        Returns:
+            `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
+        """
+        if already_has_special_tokens:
+            return super().get_special_tokens_mask(
+                token_ids_0=token_ids_0,
+                token_ids_1=token_ids_1,
+                already_has_special_tokens=True,
+            )
+
+        if not self.add_bos_token:
+            return super().get_special_tokens_mask(
+                token_ids_0=token_ids_0,
+                token_ids_1=token_ids_1,
+                already_has_special_tokens=False,
+            )
+
+        if token_ids_1 is None:
+            return [1] + ([0] * len(token_ids_0))
+        return [1] + ([0] * len(token_ids_0)) + [1] + ([0] * len(token_ids_1))

modeling_rwkv6.py

@@ -0,0 +1,801 @@
+# coding=utf-8
+# Copyright 2024 The RWKV team and HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""PyTorch RWKV6 World model."""
+
+from dataclasses import dataclass
+from typing import List, Optional, Tuple, Union
+
+import torch
+import torch.nn.functional as F
+import torch.utils.checkpoint
+from torch import nn
+from torch.nn import CrossEntropyLoss
+
+from transformers.generation import GenerationMixin
+from transformers.modeling_utils import PreTrainedModel
+from transformers.utils import (
+    ModelOutput,
+    add_code_sample_docstrings,
+    add_start_docstrings,
+    add_start_docstrings_to_model_forward,
+    logging,
+)
+
+from .configuration_rwkv6 import Rwkv6Config
+
+
+def check_dependencies():
+    missing_deps = []
+
+    try:
+        import triton  # noqa: F401
+    except ImportError:
+        missing_deps.append("triton>=3.0.0")
+
+    try:
+        import rwkvfla  # noqa: F401
+    except ImportError:
+        missing_deps.append("rwkv-fla")
+
+    if missing_deps:
+        install_instructions = """
+Required dependencies are missing. Please install them using:
+
+{}
+
+""".strip()
+        install_commands = "\n".join(f"pip install {dep}" for dep in missing_deps)
+        print(install_instructions.format(install_commands))
+        return False
+    else:
+        return True
+
+
+if check_dependencies():
+    # flake8: noqa: E402
+    from rwkvfla.ops.rwkv6.chunk import chunk_rwkv6  # pylint: disable=C0411
+    from rwkvfla.ops.rwkv6.fused_recurrent import fused_recurrent_rwkv6  # pylint: disable=C0411
+    from rwkvfla.ops.rwkv6.recurrent_naive import native_recurrent_rwkv6  # pylint: disable=C0411
+else:
+    from .wkv6 import native_recurrent_rwkv6
+
+    chunk_rwkv6 = native_recurrent_rwkv6
+    fused_recurrent_rwkv6 = native_recurrent_rwkv6
+
+
+logger = logging.get_logger(__name__)
+
+_CHECKPOINT_FOR_DOC = "RWKV/rwkv-6-world-1b6"
+_CONFIG_FOR_DOC = "Rwkv6Config"
+
+
+def rwkv6_linear_attention(
+    training,
+    receptance,
+    key,
+    value,
+    time_decay,
+    time_first,
+    state,
+):
+    one_token = key.size(1) == 1
+    batch, seq_length, _ = receptance.shape
+    num_heads, head_size = time_first.shape
+    # pylint: disable=line-too-long
+    key = key.float().view(batch, seq_length, num_heads, head_size).transpose(1, 2)  # B, T, H, K -> B, H, T, K
+    value = value.float().view(batch, seq_length, num_heads, head_size).transpose(1, 2)  # B, T, H, K - > B, H, T, V
+    receptance = receptance.float().view(batch, seq_length, num_heads, head_size).transpose(1, 2)  # B, H, T, K
+    time_decay = (
+        -torch.exp(time_decay.float()).view(batch, seq_length, num_heads, head_size).permute(0, 2, 1, 3)
+    )  # B, T, H, K -> B, H, T, K
+    time_first = time_first.float().reshape(num_heads, head_size)  # H, K
+    if receptance.device.type == "cpu":
+        out, state = native_recurrent_rwkv6(
+            receptance, key, value, time_decay, time_first, scale=1.0, initial_state=state, output_final_state=True
+        )
+    elif one_token:
+        out, state = fused_recurrent_rwkv6(
+            receptance, key, value, time_decay, time_first, scale=1.0, initial_state=state, output_final_state=True
+        )
+    else:
+        out, state = chunk_rwkv6(
+            receptance, key, value, time_decay, time_first, scale=1.0, initial_state=state, output_final_state=True
+        )
+    return out.transpose(1, 2), state
+
+
+class Rwkv6SelfAttention(nn.Module):
+    def __init__(self, config, layer_id=0):
+        super().__init__()
+        self.config = config
+        self.layer_id = layer_id
+        hidden_size = config.hidden_size
+        attention_hidden_size = config.attention_hidden_size
+        self.attention_hidden_size = attention_hidden_size
+        head_size = config.head_size
+        num_heads = attention_hidden_size // head_size
+
+        self.time_maa_x = nn.Parameter(torch.empty(1, 1, hidden_size))
+        self.time_maa_w = nn.Parameter(torch.empty(1, 1, hidden_size))
+        self.time_maa_k = nn.Parameter(torch.empty(1, 1, hidden_size))
+        self.time_maa_v = nn.Parameter(torch.empty(1, 1, hidden_size))
+        self.time_maa_r = nn.Parameter(torch.empty(1, 1, hidden_size))
+        self.time_maa_g = nn.Parameter(torch.empty(1, 1, hidden_size))
+
+        time_mix_extra_dim = 32  # generate TIME_MIX for w,k,v,r,g
+        if hidden_size == 4096:  # 7b
+            time_mix_extra_dim = 64
+        self.time_maa_w1 = nn.Parameter(torch.empty(hidden_size, time_mix_extra_dim * 5))
+        self.time_maa_w2 = nn.Parameter(torch.empty(5, time_mix_extra_dim, hidden_size))
+
+        self.time_decay = nn.Parameter(torch.empty(1, 1, attention_hidden_size))
+
+        time_decay_extra_dim = 64
+        if hidden_size == 4096:  # 7b
+            time_decay_extra_dim = 128
+        self.time_decay_w1 = nn.Parameter(torch.empty(hidden_size, time_decay_extra_dim))
+        self.time_decay_w2 = nn.Parameter(torch.empty(time_decay_extra_dim, attention_hidden_size))
+
+        self.time_faaaa = nn.Parameter(torch.empty(num_heads, config.head_size))
+
+        self.time_shift = nn.ZeroPad2d((0, 0, 1, -1))
+        self.receptance = nn.Linear(hidden_size, attention_hidden_size, bias=False)
+        self.key = nn.Linear(hidden_size, attention_hidden_size, bias=False)
+        self.value = nn.Linear(hidden_size, attention_hidden_size, bias=False)
+        self.gate = nn.Linear(hidden_size, attention_hidden_size, bias=False)
+        self.output = nn.Linear(attention_hidden_size, hidden_size, bias=False)
+        self.ln_x = nn.GroupNorm(num_heads, hidden_size, eps=(1e-5) * (config.head_size_divisor**2))
+
+    def extract_key_value(self, hidden, state=None):
+        # Mix hidden with the previous timestep to produce key, value,
+        # receptance
+        if hidden.size(1) == 1 and state is not None:
+            shifted = state[0][:, :, self.layer_id]
+        else:
+            shifted = self.time_shift(hidden)
+            if state is not None:
+                shifted[:, 0] = state[0][:, :, self.layer_id]
+        if len(shifted.size()) == 2:
+            shifted = shifted.unsqueeze(1)
+
+        x = hidden
+
+        B, T, C = hidden.shape
+
+        xx = shifted - x
+
+        xxx = x + xx * self.time_maa_x
+        xxx = torch.tanh(xxx @ self.time_maa_w1).view(B * T, 5, -1).transpose(0, 1)
+        xxx = torch.bmm(xxx, self.time_maa_w2).view(5, B, T, -1)
+        mw, mk, mv, mr, mg = xxx.unbind(dim=0)
+
+        time_decay = x + xx * (self.time_maa_w + mw)
+        key = x + xx * (self.time_maa_k + mk)
+        value = x + xx * (self.time_maa_v + mv)
+        receptance = x + xx * (self.time_maa_r + mr)
+        gate = x + xx * (self.time_maa_g + mg)
+
+        receptance = self.receptance(receptance)
+        key = self.key(key)
+        value = self.value(value)
+        gate = F.silu(self.gate(gate))
+
+        time_decay = torch.tanh(time_decay @ self.time_decay_w1) @ self.time_decay_w2
+        time_decay = self.time_decay + time_decay
+
+        if state is not None:
+            state[0][:, :, self.layer_id] = hidden[:, -1]
+
+        return receptance, key, value, gate, time_decay, state
+
+    def forward(self, hidden, state=None, use_cache=False, seq_mode=True):
+        receptance, key, value, gate, time_decay, state = self.extract_key_value(hidden, state=state)
+
+        B, T, _ = receptance.shape
+        H, S = self.time_faaaa.shape
+
+        layer_state = state[1][:, :, :, :, self.layer_id] if state is not None else None
+        out, layer_state = rwkv6_linear_attention(
+            self.training,
+            receptance,
+            key,
+            value,
+            time_decay,
+            self.time_faaaa,
+            layer_state,
+        )
+
+        if layer_state is not None:
+            state[1][:, :, :, :, self.layer_id] = layer_state
+
+        out = out.reshape(B * T, H * S)
+        out = F.group_norm(
+            out,
+            num_groups=H,
+            weight=self.ln_x.weight.to(out.dtype),
+            bias=self.ln_x.bias.to(out.dtype),
+            eps=self.ln_x.eps,
+        ).reshape(B, T, H * S)
+        out = out.to(dtype=hidden.dtype) * gate
+        out = self.output(out)
+        return out, state
+
+
+class Rwkv6FeedForward(nn.Module):
+    def __init__(self, config, layer_id=0):
+        super().__init__()
+        self.config = config
+        self.layer_id = layer_id
+        hidden_size = config.hidden_size
+        # https://github.com/BlinkDL/RWKV-LM/blob/3db37a72356b736966ddd377268f02b80963af3f/RWKV-v4neo/train.py#L168
+        intermediate_size = (
+            config.intermediate_size
+            if config.intermediate_size is not None
+            else int((config.hidden_size * 3.5) // 32 * 32)
+        )
+
+        self.time_shift = nn.ZeroPad2d((0, 0, 1, -1))
+        self.time_maa_k = nn.Parameter(torch.empty(1, 1, hidden_size))
+        self.time_maa_r = nn.Parameter(torch.empty(1, 1, hidden_size))
+
+        self.key = nn.Linear(hidden_size, intermediate_size, bias=False)
+        self.receptance = nn.Linear(hidden_size, hidden_size, bias=False)
+        self.value = nn.Linear(intermediate_size, hidden_size, bias=False)
+
+    def forward(self, hidden, state=None):
+        if hidden.size(1) == 1 and state is not None:
+            shifted = state[2][:, :, self.layer_id]
+        else:
+            shifted = self.time_shift(hidden)
+            if state is not None:
+                shifted[:, 0] = state[2][:, :, self.layer_id]
+        if len(shifted.size()) == 2:
+            shifted = shifted.unsqueeze(1)
+
+        delta_hidden_to_shifted = shifted - hidden
+        key = hidden + delta_hidden_to_shifted * self.time_maa_k
+        receptance = hidden + delta_hidden_to_shifted * self.time_maa_r
+
+        key = torch.square(torch.relu(self.key(key)))
+        value = self.value(key)
+        receptance = torch.sigmoid(self.receptance(receptance))
+
+        if state is not None:
+            state[2][:, :, self.layer_id] = hidden[:, -1]
+
+        return receptance * value, state
+
+
+class Rwkv6Block(nn.Module):
+    def __init__(self, config, layer_id):
+        super().__init__()
+        self.config = config
+        self.layer_id = layer_id
+
+        if layer_id == 0:
+            self.pre_ln = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_epsilon)
+
+        self.ln1 = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_epsilon)
+        self.ln2 = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_epsilon)
+
+        self.attention = Rwkv6SelfAttention(config, layer_id)
+        self.feed_forward = Rwkv6FeedForward(config, layer_id)
+
+    def forward(self, hidden, state=None, use_cache=False, output_attentions=False, seq_mode=True):
+        if self.layer_id == 0:
+            hidden = self.pre_ln(hidden)
+        attention, state = self.attention(self.ln1(hidden), state=state, use_cache=use_cache, seq_mode=seq_mode)
+        hidden = hidden + attention
+
+        feed_forward, state = self.feed_forward(self.ln2(hidden), state=state)
+        hidden = hidden + feed_forward
+
+        outputs = (hidden, state)
+        if output_attentions:
+            outputs += (attention,)
+        else:
+            outputs += (None,)
+
+        return outputs
+
+
+class Rwkv6PreTrainedModel(PreTrainedModel):
+    # pylint: disable=line-too-long
+    """
+    An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained
+    models.
+    """
+
+    config_class = Rwkv6Config
+    base_model_prefix = "rwkv6"
+    _no_split_modules = ["Rwkv6Block"]
+    _keep_in_fp32_modules = ["time_decay", "time_first"]
+    supports_gradient_checkpointing = True
+
+    def _init_weights(self, module):
+        """Initialize the weights."""
+        if isinstance(module, Rwkv6SelfAttention):
+            layer_id = module.layer_id
+            num_hidden_layers = module.config.num_hidden_layers
+            hidden_size = module.config.hidden_size
+            attention_hidden_size = module.attention_hidden_size
+            head_size = module.config.head_size
+            num_heads = attention_hidden_size // head_size
+
+            ratio_0_to_1 = layer_id / (num_hidden_layers - 1)  # 0 to 1
+            ratio_1_to_almost0 = 1.0 - (layer_id / num_hidden_layers)  # 1 to ~0
+
+            time_weight = torch.tensor(
+                [i / hidden_size for i in range(hidden_size)],
+                dtype=module.time_maa_k.dtype,
+                device=module.time_maa_k.device,
+            )
+            time_weight = time_weight[None, None, :]
+
+            decay_speed = [
+                -6.0 + 5.0 * (h / (attention_hidden_size - 1)) ** (0.7 + 1.3 * ratio_0_to_1)
+                for h in range(attention_hidden_size)
+            ]
+            decay_speed = torch.tensor(decay_speed, dtype=module.time_decay.dtype, device=module.time_decay.device)
+            tmp = torch.tensor(
+                [
+                    (1.0 - (i / (attention_hidden_size - 1.0))) * ratio_0_to_1 + 0.1 * ((i + 1) % 3 - 1)
+                    for i in range(attention_hidden_size)
+                ],
+                dtype=module.time_faaaa.dtype,
+                device=module.time_faaaa.device,
+            )
+
+            with torch.no_grad():
+                module.time_maa_x.data = 1.0 - torch.pow(time_weight, ratio_1_to_almost0)
+                module.time_maa_w.data = 1.0 - torch.pow(time_weight, ratio_1_to_almost0)
+                module.time_maa_k.data = 1.0 - torch.pow(time_weight, ratio_1_to_almost0)
+                module.time_maa_v.data = 1.0 - (torch.pow(time_weight, ratio_1_to_almost0) + 0.3 * ratio_0_to_1)
+                module.time_maa_r.data = 1.0 - torch.pow(time_weight, 0.5 * ratio_1_to_almost0)
+                module.time_maa_g.data = 1.0 - torch.pow(time_weight, 0.5 * ratio_1_to_almost0)
+
+                TIME_MIX_EXTRA_DIM = 32  # generate TIME_MIX for w,k,v,r,g
+                module.time_maa_w1.data = torch.zeros(
+                    hidden_size,
+                    TIME_MIX_EXTRA_DIM * 5,
+                    dtype=module.time_maa_w1.dtype,
+                    device=module.time_maa_w1.device,
+                ).uniform_(-1e-4, 1e-4)
+                module.time_maa_w2.data = torch.zeros(
+                    5,
+                    TIME_MIX_EXTRA_DIM,
+                    hidden_size,
+                    dtype=module.time_maa_w2.dtype,
+                    device=module.time_maa_w2.device,
+                ).uniform_(-1e-4, 1e-4)
+
+                TIME_DECAY_EXTRA_DIM = 64
+                module.time_decay_w1.data = torch.zeros(
+                    hidden_size,
+                    TIME_DECAY_EXTRA_DIM,
+                    dtype=module.time_decay_w1.dtype,
+                    device=module.time_decay_w1.device,
+                ).uniform_(-1e-4, 1e-4)
+                module.time_decay_w2.data = torch.zeros(
+                    TIME_DECAY_EXTRA_DIM,
+                    attention_hidden_size,
+                    dtype=module.time_decay_w2.dtype,
+                    device=module.time_decay_w2.device,
+                ).uniform_(-1e-4, 1e-4)
+
+                module.time_decay.data = decay_speed.reshape(num_heads, head_size)
+                module.time_faaaa.data = tmp.reshape(num_heads, head_size)
+
+        elif isinstance(module, Rwkv6FeedForward):
+            layer_id = module.layer_id
+            num_hidden_layers = module.config.num_hidden_layers
+            hidden_size = module.config.hidden_size
+
+            ratio_1_to_almost0 = 1.0 - (layer_id / num_hidden_layers)  # 1 to ~0
+
+            time_weight = torch.tensor(
+                [i / hidden_size for i in range(hidden_size)],
+                dtype=module.time_maa_k.dtype,
+                device=module.time_maa_k.device,
+            )
+            time_weight = time_weight[None, None, :]
+
+            with torch.no_grad():
+                module.time_maa_k.data = 1.0 - torch.pow(time_weight, ratio_1_to_almost0)
+                module.time_maa_r.data = 1.0 - torch.pow(time_weight, ratio_1_to_almost0)
+
+
+@dataclass
+class Rwkv6Output(ModelOutput):
+    # pylint: disable=line-too-long
+    """
+    Class for the RWKV model outputs.
+
+    Args:
+        last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`):
+            Sequence of hidden-states at the output of the last layer of the model.
+        state (list of five `torch.FloatTensor` of shape `(batch_size, hidden_size, num_hidden_layers)`):
+            The state of the model at the last time step. Can be used in a forward method with the next `input_ids` to
+            avoid providing the old `input_ids`.
+        hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`):
+            Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, +
+            one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of
+            the model at the output of each layer plus the optional initial embedding outputs.
+        attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`):
+            Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length,
+            sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in
+            the self-attention heads.
+    """
+
+    last_hidden_state: torch.FloatTensor = None
+    state: Optional[List[torch.FloatTensor]] = None
+    hidden_states: Optional[Tuple[torch.FloatTensor]] = None
+    attentions: Optional[Tuple[torch.FloatTensor]] = None
+
+
+@dataclass
+class Rwkv6CausalLMOutput(ModelOutput):
+    """
+    Base class for causal language model (or autoregressive) outputs.
+
+    Args:
+        loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided):
+            Language modeling loss (for next-token prediction).
+        logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`):
+            Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
+        state (list of five `torch.FloatTensor` of shape `(batch_size, hidden_size, num_hidden_layers)`):
+            The state of the model at the last time step. Can be used in a forward method with the next `input_ids` to
+            avoid providing the old `input_ids`.
+        hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`):
+            Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, +
+            one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of
+            the model at the output of each layer plus the optional initial embedding outputs.
+        attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`):
+            Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length,
+            sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in
+            the self-attention heads.
+    """
+
+    loss: Optional[torch.FloatTensor] = None
+    logits: torch.FloatTensor = None
+    state: Optional[List[torch.FloatTensor]] = None
+    hidden_states: Optional[Tuple[torch.FloatTensor]] = None
+    attentions: Optional[Tuple[torch.FloatTensor]] = None
+
+
+RWKV6_START_DOCSTRING = r"""
+    This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the
+    library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
+    etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module)
+    subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to
+    general usage and behavior.
+
+    Parameters:
+        config ([`Rwkv6Config`]): Model configuration class with all the parameters of the model.
+            Initializing with a config file does not load the weights associated with the model, only the
+            configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights.
+"""
+
+RWKV6_INPUTS_DOCSTRING = r"""
+    Args:
+        input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`):
+            `input_ids_length` = `sequence_length` if `past_key_values` is `None` else
+            `past_key_values[0][0].shape[-2]` (`sequence_length` of input past key value states). Indices of input
+            sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their
+            past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See
+            [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input
+            IDs?](../glossary#input-ids)
+        inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*):
+            Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This
+            is useful if you want more control over how to convert `input_ids` indices into associated vectors than the
+            model's internal embedding lookup matrix.
+        state (tuple of five `torch.FloatTensor` of shape `(batch_size, hidden_size, num_hidden_layers)`, *optional*):
+            If passed along, the model uses the previous state in all the blocks (which will give the output for the
+            `input_ids` provided as if the model add `state_input_ids + input_ids` as context).
+        use_cache (`bool`, *optional*):
+            If set to `True`, the last state is returned and can be used to quickly generate the next logits.
+        output_attentions (`bool`, *optional*):
+            Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned
+            tensors for more detail.
+        output_hidden_states (`bool`, *optional*):
+            Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for
+            more detail.
+        return_dict (`bool`, *optional*):
+            Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
+"""
+
+
+@add_start_docstrings(
+    "The bare RWKV6 Model transformer outputting raw hidden-states without any specific head on top.",
+    RWKV6_START_DOCSTRING,
+)
+class Rwkv6Model(Rwkv6PreTrainedModel):
+    def __init__(self, config):
+        super().__init__(config)
+
+        self.embeddings = nn.Embedding(config.vocab_size, config.hidden_size)
+        self.blocks = nn.ModuleList([Rwkv6Block(config, layer_id=idx) for idx in range(config.num_hidden_layers)])
+        self.ln_out = nn.LayerNorm(config.hidden_size)
+
+        self.layers_are_rescaled = False
+        self.gradient_checkpointing = False
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.embeddings
+
+    def set_input_embeddings(self, new_embeddings):
+        self.embeddings = new_embeddings
+
+    @add_start_docstrings_to_model_forward(RWKV6_INPUTS_DOCSTRING)
+    @add_code_sample_docstrings(
+        checkpoint=_CHECKPOINT_FOR_DOC,
+        output_type=Rwkv6Output,
+        config_class=_CONFIG_FOR_DOC,
+    )
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.LongTensor] = None,  # noqa
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        state: Optional[List[torch.FloatTensor]] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+    ) -> Union[Tuple, Rwkv6Output]:
+        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
+        output_hidden_states = (
+            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        )
+        # training is supportable with the Triton kernel
+        # rwkv6 only support inference in huggingface.
+        use_cache = use_cache if use_cache is not None else self.config.use_cache
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        if self.training == self.layers_are_rescaled and (
+            self.embeddings.weight.dtype == torch.float16 or self.embeddings.weight.dtype == torch.bfloat16
+        ):
+            self._rescale_layers()
+
+        if input_ids is not None and inputs_embeds is not None:
+            raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
+        elif input_ids is None and inputs_embeds is None:
+            raise ValueError("You have to specify either input_ids or inputs_embeds")
+
+        if inputs_embeds is None:
+            inputs_embeds = self.embeddings(input_ids)
+
+        if state is None:
+            state = []
+            head_size = self.config.head_size
+            num_heads = self.config.attention_hidden_size // head_size
+            state_attn_x = torch.zeros(
+                (inputs_embeds.size(0), self.config.hidden_size, self.config.num_hidden_layers),
+                dtype=inputs_embeds.dtype,
+                requires_grad=False,
+                device=inputs_embeds.device,
+            ).contiguous()
+            state_attn_kv = torch.zeros(
+                (
+                    inputs_embeds.size(0),
+                    num_heads,
+                    head_size,
+                    head_size,
+                    self.config.num_hidden_layers,
+                ),
+                dtype=torch.float32,
+                requires_grad=False,
+                device=inputs_embeds.device,
+            ).contiguous()
+            state_ffn_x = torch.zeros(
+                (inputs_embeds.size(0), self.config.hidden_size, self.config.num_hidden_layers),
+                dtype=inputs_embeds.dtype,
+                requires_grad=False,
+                device=inputs_embeds.device,
+            ).contiguous()
+            state.append(state_attn_x)
+            state.append(state_attn_kv)
+            state.append(state_ffn_x)
+
+        seq_mode = inputs_embeds.shape[1] > 1
+        hidden_states = inputs_embeds
+
+        all_self_attentions = () if output_attentions else None
+        all_hidden_states = () if output_hidden_states else None
+        for idx, block in enumerate(self.blocks):
+            hidden_states, state, attentions = block(
+                hidden_states, state=state, use_cache=use_cache, output_attentions=output_attentions, seq_mode=seq_mode
+            )
+            if (
+                self.layers_are_rescaled
+                and self.config.rescale_every > 0
+                and (idx + 1) % self.config.rescale_every == 0
+            ):
+                hidden_states = hidden_states / 2
+
+            if output_hidden_states:
+                all_hidden_states = all_hidden_states + (hidden_states,)
+
+            if output_attentions:
+                all_self_attentions = all_self_attentions + (attentions,)
+
+        hidden_states = self.ln_out(hidden_states)
+
+        if output_hidden_states:
+            all_hidden_states = all_hidden_states + (hidden_states,)
+
+        if not return_dict:
+            return (hidden_states, state, all_hidden_states, all_self_attentions)
+
+        return Rwkv6Output(
+            last_hidden_state=hidden_states,
+            state=state,
+            hidden_states=all_hidden_states,  # None
+            attentions=all_self_attentions,  # None
+        )
+
+    def _rescale_layers(self):
+        # Layers should be rescaled for inference only.
+        if self.layers_are_rescaled == (not self.training):
+            return
+        if self.config.rescale_every > 0:
+            with torch.no_grad():
+                for block_id, block in enumerate(self.blocks):
+                    if self.training:
+                        block.attention.output.weight.mul_(2 ** int(block_id // self.config.rescale_every))
+                        block.feed_forward.value.weight.mul_(2 ** int(block_id // self.config.rescale_every))
+                    else:
+                        # Deal with quantization statistics
+                        if hasattr(block.attention.output.weight, "SCB"):
+                            block.attention.output.weight.SCB.div_(2 ** int(block_id // self.config.rescale_every))
+                            block.feed_forward.value.weight.SCB.div_(2 ** int(block_id // self.config.rescale_every))
+                        elif hasattr(block.attention.output.weight, "quant_state"):
+                            self._bnb_4bit_dequantize_and_rescale(block.attention.output, block_id)
+                            self._bnb_4bit_dequantize_and_rescale(block.feed_forward.value, block_id)
+                        else:
+                            block.attention.output.weight.div_(2 ** int(block_id // self.config.rescale_every))
+                            block.feed_forward.value.weight.div_(2 ** int(block_id // self.config.rescale_every))
+
+        self.layers_are_rescaled = not self.training
+
+    def _bnb_4bit_dequantize_and_rescale(self, target_layer, block_id):
+        r"""
+        Perform the dequantization and rescaling of the weights of a given layer. After that operation the layer will
+        be quantized again.
+        """
+        try:
+            import bitsandbytes as bnb
+        except ImportError:
+            raise ImportError("Please install bitsandbytes to use this method.")
+
+        dequant_weights = bnb.functional.dequantize_4bit(target_layer.weight.data, target_layer.weight.quant_state)
+
+        dequant_weights.div_(2 ** int(block_id // self.config.rescale_every))
+
+        # re-quantize the model:
+        # we need to put it first on CPU then back to the device
+        # this will create an overhead :/
+        # We set requires_grad=False as we cannot compute gradients on top of 4bit parameters anyway and to avoid
+        # bugs with bnb
+        quant_weight = bnb.nn.Params4bit(dequant_weights.to("cpu"), requires_grad=False).to(dequant_weights.device)
+        setattr(target_layer, "weight", quant_weight)
+
+
+# copied from HuggingFace
+# https://github.com/huggingface/transformers/blob/main/src/transformers/models/rwkv/modeling_rwkv.py
+@add_start_docstrings(
+    """
+    The RWKV6 Model transformer with a language modeling head on top (linear layer with weights tied to the input
+    embeddings).
+    """,
+    RWKV6_START_DOCSTRING,
+)
+class Rwkv6ForCausalLM(Rwkv6PreTrainedModel, GenerationMixin):
+    _tied_weights_keys = ["head.weight"]
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.rwkv = Rwkv6Model(config)
+        self.head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def get_output_embeddings(self):
+        return self.head
+
+    def set_output_embeddings(self, new_embeddings):
+        self.head = new_embeddings
+
+    def prepare_inputs_for_generation(self, input_ids, state=None, inputs_embeds=None, **kwargs):
+        # only last token for inputs_ids if the state is passed along.
+        if state is not None:
+            input_ids = input_ids[:, -1].unsqueeze(-1)
+
+        # if `inputs_embeds` are passed, we only want to use them in the 1st
+        # generation step
+        if inputs_embeds is not None and state is None:
+            model_inputs = {"inputs_embeds": inputs_embeds}
+        else:
+            model_inputs = {"input_ids": input_ids}
+
+        model_inputs["state"] = state
+        return model_inputs
+
+    @add_start_docstrings_to_model_forward(RWKV6_INPUTS_DOCSTRING)
+    @add_code_sample_docstrings(
+        checkpoint=_CHECKPOINT_FOR_DOC,
+        output_type=Rwkv6CausalLMOutput,
+        config_class=_CONFIG_FOR_DOC,
+    )
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.LongTensor] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        state: Optional[List[torch.FloatTensor]] = None,
+        labels: Optional[torch.LongTensor] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+    ) -> Union[Tuple, Rwkv6CausalLMOutput]:
+        # pylint: disable=line-too-long
+        r"""
+        labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
+            Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set
+            `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100`
+            are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]`
+        """
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        outputs = self.rwkv(
+            input_ids,
+            inputs_embeds=inputs_embeds,
+            state=state,
+            use_cache=use_cache,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        )
+        hidden_states = outputs[0]
+
+        logits = self.head(hidden_states)
+
+        loss = None
+        if labels is not None:
+            # move labels to correct device to enable model parallelism
+            labels = labels.to(logits.device)
+            # Shift so that tokens < n predict n
+            shift_logits = logits[..., :-1, :].contiguous()
+            shift_labels = labels[..., 1:].contiguous()
+            # Flatten the tokens
+            loss_fct = CrossEntropyLoss()
+            loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))
+
+        if not return_dict:
+            output = (logits,) + outputs[1:]
+            return ((loss,) + output) if loss is not None else output
+
+        return Rwkv6CausalLMOutput(
+            loss=loss,
+            logits=logits,
+            state=outputs.state,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5984baf1510cdf659b287166c2205fdaa10097ab7f4e4cdc57b640f14a0a965e
+size 15271603150

special_tokens_map.json

@@ -0,0 +1,5 @@
+{
+  "bos_token": "<s>",
+  "eos_token": "<s>",
+  "unk_token": "<s>"
+}

tokenizer_config.json

@@ -0,0 +1,12 @@
+{
+  "name_or_path": "rwkv-6-tokenizer",
+  "add_prefix_space": false,
+  "tokenizer_class": "Rwkv6Tokenizer",
+  "use_fast": false,
+  "auto_map": {
+    "AutoTokenizer": [
+      "hf_rwkv_tokenizer.Rwkv6Tokenizer",
+      null
+      ]
+  }
+}

wkv6.py

@@ -0,0 +1,209 @@
+# -*- coding: utf-8 -*-
+
+from typing import Optional, Tuple
+
+import torch
+
+
+def naive_recurrent_rwkv6(
+    q: torch.Tensor,
+    k: torch.Tensor,
+    v: torch.Tensor,
+    w: torch.Tensor,
+    u: torch.Tensor,
+    scale: float = 1.0,
+    initial_state: Optional[torch.Tensor] = None,
+    output_final_state: bool = False,
+    u_2d: bool = False,
+):
+    torch_dtype = q.dtype if q.dtype in [torch.float16, torch.float32, torch.float64] else torch.float32
+    orig_dtype = q.dtype
+    B, H, T, K, V = q.shape[0], q.shape[1], q.shape[2], q.shape[3], v.shape[-1]
+    q, k, v, w, u = (x.to(dtype=torch_dtype) for x in (q, k, v, w, u))
+    h = torch.zeros(B, H, K, V, dtype=torch_dtype, device=q.device)
+    o = torch.zeros_like(v)
+
+    if scale == -1.0:
+        scale = K**-0.5
+
+    if initial_state is not None:
+        h += initial_state.to(dtype=torch_dtype)
+
+    w = w.exp()
+
+    if u_2d:
+        u_expand = u[None, ..., None]
+    else:
+        u_expand = u[..., None]
+
+    for i in range(T):
+        q_i = q[:, :, i, :] * scale
+        k_i = k[:, :, i] * scale
+        v_i = v[:, :, i, :]
+        w_i = w[:, :, i]
+        kv_i = k_i[..., None] * v_i[..., None, :]
+        o_i = (h + u_expand * kv_i) * q_i[..., None]
+        o[:, :, i] = o_i.sum(-2)
+        h = h * w_i[..., None] + kv_i
+
+    ht = h if output_final_state else None
+    return o.to(orig_dtype), ht
+
+
+def naive_recurrent_rwkv6_bwd(
+    q: torch.Tensor,
+    k: torch.Tensor,
+    v: torch.Tensor,
+    w: torch.Tensor,
+    u: torch.Tensor,
+    o: torch.Tensor,
+    do: torch.Tensor,
+    dh_t: Optional[torch.Tensor] = None,
+    initial_state: Optional[torch.Tensor] = None,
+    scale: float = 1.0,
+    u_2d: bool = False,
+):
+    torch_type = torch.float32 if q.dtype != torch.float16 else torch.float16
+    q, k, v, w, u, o, do = (x.to(dtype=torch_type) for x in (q, k, v, w, u, o, do))
+    B, H, T, K, V = q.shape[0], q.shape[1], q.shape[2], q.shape[3], v.shape[-1]
+    h = torch.zeros(B, H, K, V, dtype=torch_type, device=q.device)
+    dq = torch.zeros_like(q)
+    dq_aux = torch.zeros_like(q)
+
+    if initial_state is not None:
+        h += initial_state
+
+    if scale == -1.0:
+        scale = K**-0.5
+
+    w = w.exp()
+    if u_2d:
+        u_expand = u[None, ..., None]
+        sum_dims = [0, -1]
+    else:
+        u_expand = u[..., None]
+        sum_dims = [-1]
+
+    for i in range(T):
+        k_i = k[:, :, i] * scale
+        v_i = v[:, :, i]
+        w_i = w[:, :, i]
+        kv_i = k_i[..., None] * v_i[..., None, :]
+        h_i = h + u_expand * kv_i
+        dq_i = (do[:, :, i, None, :] * h_i).sum(-1)
+        dq_aux_i = (do[:, :, i, None, :] * h).sum(-1)
+        dq[:, :, i] = dq_i * scale
+        dq_aux[:, :, i] = dq_aux_i
+        h = h * w_i[..., None] + kv_i
+
+    du = torch.zeros_like(u)
+    dh = torch.zeros_like(h)
+    if dh_t is not None:
+        dh += dh_t
+    dk = torch.zeros_like(k)
+    dk_aux = torch.zeros_like(k)
+    dv = torch.zeros_like(v)
+
+    for i in range(T - 1, -1, -1):
+        q_i = q[:, :, i] * scale
+        k_i = k[:, :, i] * scale
+        v_i = v[:, :, i]
+
+        d_kv_i = do[:, :, i, None, :] * q_i[..., None]
+        du += (d_kv_i * k_i[..., None] * v_i[..., None, :]).sum(sum_dims)
+
+        dk_i = (dh * v_i[..., None, :]).sum(-1)
+        dk_aux[:, :, i] = dk_i
+        dk_i += (d_kv_i * u_expand * v_i[..., None, :]).sum(-1)
+
+        dv_i = (d_kv_i * u_expand * k_i[..., None]).sum(-2)
+        dv_i += (dh * k_i[..., None]).sum(-2)
+
+        dk[:, :, i] = dk_i * scale
+        dv[:, :, i] = dv_i
+        dh = dh * w[:, :, i, :, None] + d_kv_i
+
+    # dw = q * dq_aux - k * dk_aux
+    dw = torch.zeros_like(w)
+    for i in range(T - 2, -1, -1):
+        dw[:, :, i] = (
+            dw[:, :, i + 1] + dq_aux[:, :, i + 1] * q[:, :, i + 1] * scale - dk_aux[:, :, i] * k[:, :, i] * scale
+        )
+
+    return dq, dk, dv, dw, du, dh
+
+
+class NativeRecurrentRWKV6Function(torch.autograd.Function):
+    @staticmethod
+    def forward(
+        ctx,
+        q,
+        k,
+        v,
+        w,
+        u,
+        scale,
+        initial_state,
+        output_final_state: bool = False,
+        u_2d: bool = False,
+        training: bool = True,
+    ):
+        o, ht = naive_recurrent_rwkv6(q, k, v, w, u, scale, initial_state, output_final_state, u_2d)
+        if initial_state is not None:
+            initial_state = initial_state.clone()
+        if training:
+            ctx.save_for_backward(q, k, v, w, u, o, initial_state)
+            ctx.u_2d = u_2d
+            ctx.scale = scale
+        return o, ht
+
+    @staticmethod
+    def backward(ctx, do, dht):
+        q, k, v, w, u, o, initial_state = ctx.saved_tensors
+        dq, dk, dv, dw, du, dh = naive_recurrent_rwkv6_bwd(
+            q, k, v, w, u, o, do, dht, initial_state, ctx.scale, ctx.u_2d
+        )
+        dh = dh if initial_state is not None else None
+        return dq, dk, dv, dw, du, None, dh, None, None, None
+
+
+def native_recurrent_rwkv6(
+    r: torch.Tensor,
+    k: torch.Tensor,
+    v: torch.Tensor,
+    w: torch.Tensor,
+    u: torch.Tensor,
+    scale: float = 1.0,
+    initial_state: torch.Tensor = None,
+    output_final_state: bool = False,
+    training: bool = True,
+    causal: bool = True,
+) -> Tuple[torch.Tensor, torch.Tensor]:
+    r"""
+    Args:
+        r (torch.Tensor):
+            reception of shape `(B, H, T, K)`. Alias: q, query in linear attention.
+        k (torch.Tensor):
+            keys of shape `(B, H, T, K)`
+        v (torch.Tensor):
+            values of shape `(B, H, T, V)`
+        w (torch.Tensor):
+            data-dependent decays of shape `(B, H, T, K)` in log space! Alias: g.
+        u (torch.Tensor):
+            bonus of shape `(H, K)` or `(B, H, K)` for each head.
+        scale (Optional[int]):
+            Scale factor for the RWKV6 attention scores.
+            If not provided, it will default to `1 / sqrt(K)`. Default: `None`.
+        initial_state (Optional[torch.Tensor]):
+            Initial state of shape `(B, H, K, V)`. Default: `None`.
+        output_final_state (Optional[bool]):
+            Whether to output the final state of shape `(B, H, K, V)`. Default: `False`.
+    """
+    if scale == -1.0:
+        scale = r.shape[-1] ** -0.5
+    u_2d = True if u.dim() == 2 else False
+    o, final_state = NativeRecurrentRWKV6Function.apply(
+        r, k, v, w, u, scale, initial_state, output_final_state, u_2d, training
+    )
+
+    return o, final_state