xverse
/

XVERSE-65B-Chat-GPTQ-Int8

@@ -5,30 +5,23 @@ inference: false
 ---
-# XVERSE-65B-Chat-GPTQ-Int4
 ## 更新信息
-**[2024/03/25]** 发布XVERSE-65B-Chat-GPTQ-Int4量化模型，支持vLLM推理xverse-65b量化模型。
-**[2023/12/08]** 发布 **XVERSE-65B-2** 底座模型，该模型在前一版本的基础上进行了 **Continual Pre-Training**，训练总 token 量达到 **3.2** 万亿；模型各方面的能力均得到提升，尤其是数学和代码能力，在 GSM8K 上提升 **20**%，HumanEval 上提升 **41**%。
-**[2023/11/29]** 更新模型架构及更多底座数据的相关信息。
-**[2023/11/24]** 更新预训练数据的相关信息。
-**[2023/11/06]** 发布 65B 尺寸的 XVERSE-65B 底座模型。
 ## Update Information
-**[2024/03/25] ** Release the XVERSE-65B-Chat-GPTQ-Int4 quantification model, supporting vLLM inference for the xverse-65b quantification model.
-**[2023/12/08]** Released the **XVERSE-65B-2** base model. This model builds upon its predecessor through **Continual Pre-Training**, reaching a total training volume of **3.2** trillion tokens. It exhibits enhancements in all capabilities, particularly in mathematics and coding skills, with a **20%** improvement on the GSM8K benchmark and a **41%** increase on HumanEval.
-**[2023/11/29]** Update model architecture and additional pre-training data information.
-**[2023/11/24]** Update the related information of the pre-training data.
-**[2023/11/06]** Released the XVERSE-65B base model.
 ## 模型介绍
@@ -70,19 +63,25 @@ We advise you to clone [`vllm`](https://github.com/vllm-project/vllm.git) and in
 ## 使用方法
-我们演示了如何使用 `vllm` 来运行XVERSE-65B-Chat-GPTQ-Int4量化模型:
 ```python
 from vllm import LLM, SamplingParams
-model_dir = "xverse/XVERSE-65B-Chat-GPTQ-Int4/"
 # Create an LLM.
 llm = LLM(model_dir,
           trust_remote_code=True)
 # Create a sampling params object.
-sampling_params = SamplingParams(temperature=0.5, top_p=0.85, max_tokens=2048, repetition_penalty=1.1)
 # Generate texts from the prompts. The output is a list of RequestOutput objects
 # that contain the prompt, generated text, and other information.
@@ -98,19 +97,26 @@ for output in outputs:
 ## Usage
-We demonstrated how to use 'vllm' to run the XVERSE-65B-Chat GPTQ Int4 quantization model:
 ```python
 from vllm import LLM, SamplingParams
-model_dir = "xverse/XVERSE-65B-Chat-GPTQ-Int4/"
 # Create an LLM.
 llm = LLM(model_dir,
           trust_remote_code=True)
 # Create a sampling params object.
-sampling_params = SamplingParams(temperature=0.5, top_p=0.85, max_tokens=2048, repetition_penalty=1.1)
 # Generate texts from the prompts. The output is a list of RequestOutput objects
 # that contain the prompt, generated text, and other information.

 ---
+# XVERSE-65B-Chat-GPTQ-Int8
 ## 更新信息
+- **[2024/03/25]** 发布XVERSE-65B-Chat-GPTQ-Int8量化模型，支持vLLM推理xverse-65b量化模型。
+- **[2023/12/08]** 发布 **XVERSE-65B-2** 底座模型，该模型在前一版本的基础上进行了 **Continual Pre-Training**，训练总 token 量达到 **3.2** 万亿；模型各方面的能力均得到提升，尤其是数学和代码能力，在 GSM8K 上提升 **20**%，HumanEval 上提升 **41**%。
+- **[2023/11/29]** 更新模型架构及更多底座数据的相关信息。
+- **[2023/11/24]** 更新预训练数据的相关信息。
+- **[2023/11/06]** 发布 65B 尺寸的 XVERSE-65B 底座模型。
 ## Update Information
+- **[2024/03/25]** Release the XVERSE-65B-Chat-GPTQ-Int8 quantification model, supporting vLLM inference for the xverse-65b quantification model.
+- **[2023/12/08]** Released the **XVERSE-65B-2** base model. This model builds upon its predecessor through **Continual Pre-Training**, reaching a total training volume of **3.2** trillion tokens. It exhibits enhancements in all capabilities, particularly in mathematics and coding skills, with a **20%** improvement on the GSM8K benchmark and a **41%** increase on HumanEval.
+- **[2023/11/29]** Update model architecture and additional pre-training data information.
+- **[2023/11/24]** Update the related information of the pre-training data.
+- **[2023/11/06]** Released the XVERSE-65B base model.
 ## 模型介绍
 ## 使用方法
+由于上传的safetensors文件大小超出50GB的最大文件限制，因此我们将safetensors文件切分为3个，因此您可以将它们连接起来以获得整个文件：
+```bash
+cat gptq_model-8bit-128g.safetensors.* > gptq_model-8bit-128g.safetensors
+```
+我们演示了如何使用 `vllm` 来运行XVERSE-65B-Chat-GPTQ-Int8量化模型:
 ```python
 from vllm import LLM, SamplingParams
+model_dir = "xverse/XVERSE-65B-Chat-GPTQ-Int8/"
 # Create an LLM.
 llm = LLM(model_dir,
           trust_remote_code=True)
 # Create a sampling params object.
+sampling_params = SamplingParams(temperature=0.85, top_p=0.85, max_tokens=2048, repetition_penalty=1.1)
 # Generate texts from the prompts. The output is a list of RequestOutput objects
 # that contain the prompt, generated text, and other information.
 ## Usage
+Due to the uploaded safetensors file size exceeding the maximum file limit of 50GB,
+we have divided the safetensors file into three parts, so you can connect them together to obtain the entire file:
+```bash
+cat gptq_model-8bit-128g.safetensors.* > gptq_model-8bit-128g.safetensors
+```
+We demonstrated how to use 'vllm' to run the XVERSE-65B-Chat-GPTQ-Int8 quantization model:
 ```python
 from vllm import LLM, SamplingParams
+model_dir = "xverse/XVERSE-65B-Chat-GPTQ-Int8/"
 # Create an LLM.
 llm = LLM(model_dir,
           trust_remote_code=True)
 # Create a sampling params object.
+sampling_params = SamplingParams(temperature=0.85, top_p=0.85, max_tokens=2048, repetition_penalty=1.1)
 # Generate texts from the prompts. The output is a list of RequestOutput objects
 # that contain the prompt, generated text, and other information.