update README

README.md

@@ -14,6 +14,8 @@ Qwen2 is the new series of Qwen large language models. For Qwen2, we release a n
 
 Compared with the state-of-the-art opensource language models, including the previous released Qwen1.5, Qwen2 has generally surpassed most opensource models and demonstrated competitiveness against proprietary models across a series of benchmarks targeting for language understanding, language generation, multilingual capability, coding, mathematics, reasoning, etc.
 
+Qwen2-72B-Instruct supports a context length of up to 131,072 tokens, enabling the processing of extensive inputs. Please refer to [this section](#processing-long-texts) for detailed instructions on how to deploy Qwen2 for handling long texts.
+
 For more details, please refer to our [blog post](https://qwenlm.github.io/blog/qwen2/) and [GitHub repo](https://github.com/QwenLM/Qwen2).
 <br>
 
@@ -68,6 +70,57 @@ generated_ids = [
 response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
 ```
 
+### Processing Long Texts
+
+To handle extensive inputs exceeding 32,768 tokens, we utilize [YARN](https://arxiv.org/abs/2309.00071), a technique for enhancing model length extrapolation, ensuring optimal performance on lengthy texts.
+
+For deployment, we recommend using vLLM. You can enable long-context capabilities, follow these steps:
+
+1. **Install vLLM**: Ensure you have the latest version from the main branch of [vLLM](https://github.com/vllm-project/vllm).
+
+2. **Configure Model Settings**: After downloading the model weights, modify the `config.json` file by including the below snippet:
+     ```json5
+        {
+        "architectures": [
+            "Qwen2ForCausalLM"
+        ],
+        // ...
+        "vocab_size": 152064,
+
+        // adding the following snippets
+        "rope_scaling": {
+            "factor": 4.0,
+            "original_max_position_embeddings": 32768,
+            "type": "yarn"
+        }
+        }
+     ```
+    This snippet enable YARN to support longer contexts.
+
+3. **Model Deployment**: Utilize vLLM to deploy your model. For instance, you can set up an openAI-like server using the command:
+
+    ```bash
+    python -m vllm.entrypoints.openai.api_server --served-model-name Qwen2-72B-Instruct --model path/to/weights
+    ```
+
+    Then you can access the Chat API by:
+
+    ```bash
+    curl http://localhost:8000/v1/chat/completions \
+        -H "Content-Type: application/json" \
+        -d '{
+        "model": "Qwen2-72B-Instruct",
+        "messages": [
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Your Long Input Here."}
+        ]
+        }'
+    ```
+
+    For further usage instructions of vLLM, please refer to [our repository](https://github.com/QwenLM/Qwen2).
+
+**Note**: Presently, vLLM only supports static YARN, which means the scaling factor remains constant regardless of input length, potentially impacting performance on shorter texts. We advise adding the `rope_scaling` configuration only when processing long contexts is required.
+
 ## Citation
 
 If you find our work helpful, feel free to give us a cite.

config.json

@@ -17,7 +17,7 @@
   "num_key_value_heads": 8,
   "rms_norm_eps": 1e-06,
   "rope_theta": 1000000.0,
-  "sliding_window": 32768,
+  "sliding_window": 131072,
   "tie_word_embeddings": false,
   "torch_dtype": "bfloat16",
   "transformers_version": "4.40.1",