metadata

license: apache-2.0
language:
  - zh
  - en
library_name: transformers
tags:
  - qihoo360
  - 奇虎360
  - zhinao
  - 360Zhinao
  - pretrain

中文｜ English

360Zhinao2 (360智脑)

🤗 HuggingFace | 🤖 ModelScope ｜ 💬 WeChat (微信)

Feel free to visit 360Zhinao's official website https://ai.360.com for more experience.

Introduction

🎉🎉🎉 We released the 360Zhinao2 model series:

360Zhinao2-7B-Base
360Zhinao2-7B-Chat-4K
360Zhinao2-7B-Chat-32K
360Zhinao2-7B-Chat-360K

Notable features of our 360Zhinao models are:

Base Model: Using popular two-stage training method, In the first stage we totally train 10T tokens with a cosine learning rate schedule. In the second stage we increase the proportion of high-quality data and totally train 100B tokens, with the learning rate decaying directly to 0. The total training data for 360Zhinao2-7B amounts to 10.1T tokens.
Chat Models: Powerful chat capabilities and three context lengths of 4K, 32K and 360K.

News and Updates

[2024.11.18] 🔥🔥🔥We release 360Zhinao2-7B, providing access to both the Base model and Chat models with text lengths of 4K, 32K, and 360K.
[2024.05.23] We released two models, 360Zhinao-search and 360Zhinao-1.8B-Reranking, which ranked first respectively in the Retrieval and Reranking tasks of C-MTEB Leaderboard .
[2024.05.20] We extended llama3 and released llama3-8B-360Zhinao-360k-Instruct🤗
[2024.04.12] We released 360Zhinao-7B v1.0, including the base model and three chat models with context lengths 4K, 32K and 360K. Technical report is on arXiv.

Download URL
Model Evaluation
Quickstart
Model Inference
Model Finetune
License

Download URL

Size	Model	BF16	Int4
7B	360Zhinao2-7B-Base	🤖 🤗
7B	360Zhinao2-7B-Chat-4K	🤖 🤗	🤖 🤗
7B	360Zhinao2-7B-Chat-32K	🤖 🤗	🤖 🤗
7B	360Zhinao2-7B-Chat-360K	🤖 🤗	🤖 🤗

Model Evaluation

Base Model

We used the open-source tool OpenCompass to evaluate the model and compared it with open-source models under 10B from the past six months. The 360Zhinao2-7B model is competive. The 360Zhinao2-7B model performs well on Chinese benchmarks such as CEval, C3 and LCSTS. The average socres of Chinese benchmarks is No 1. It also ranks No 1 on Math which is a challenging competition math dataset. The 360Zhinao2-7B model has advantages in Chinese benchmark and challenging competition math.

Type	Datasets	language	glm4-9b	Qwen2.5-7B	internlm2.5-7b	Yi1.5-9B	gemma2-9b	Llama3.1-8B	360Zhinao2-7B
Exam	ceval	zh	75.83	81.41	77.71	73.51	56.36	51.67	83.04
	mmlu	en	75.5	75.5	71.55	71.43	72.22	66.75	67.84
	cmmlu	zh	74.24	81.79	78.77	74.2	58.89	52.49	73.8
	ARC-c	en	94.92	80	85.08	87.46	77.63	80.68	87.12
	ARC-e	en	98.41	84.83	95.24	94.53	78.84	89.77	92.77
Language	WiC	en	51.57	52.82	50.78	50.63	50.47	50	49.84
Language	WSC	en	68.27	68.27	69.23	66.35	68.27	67.31	65.38
Knowledge	BoolQ	en	81.8	83.88	89.51	84.46	85.6	82.2	88.29
Knowledge	commonsense_qa	en	71.17	73.22	68.55	71.58	68.47	71.25	69.78
Understanding	C3	zh	91.51	92	93.04	85.86	81.64	83.51	93.26
	race-middle	en	91.99	91.02	92.06	91.16	88.09	81.69	90.46
	race-high	en	90.71	87.91	90.08	88.34	82.08	78.73	86.74
	lcsts	zh	18.29	15.82	15.96	16.49	10.62	17.29	18.61
	eprstmt-dev	zh	91.88	86.88	91.25	91.88	48.12	83.12	90
	lambada	en	71.67	71.14	69.98	70.64	75.43	74.23	72.56
Reasoning	hellaswag	en	70.25	72.76	70.38	71.55	66.83	74.65	71.49
	siqa	en	81.73	72.52	78.97	76.2	58.96	64.18	77.12
	bbh	en	73.68	54.63	59.43	67.86	68.45	59.9	46.54
Code	humaneval	en	69.51	75	60.37	26.22	5.49	27.44	60.98
Code	mbpp	en	60	60	43.6	56.8	51.2	42.6	54
Math	math	en	26.86	38	27.14	27.06	28.52	15.32	38.34
Math	gsm8k	en	78.54	79.76	52.54	71.11	73.09	56.25	75.51
Overall	avg_zh		70.35	71.58	71.35	68.39	51.13	57.62	71.74
Overall	avg_all		73.11	71.78	69.60	68.88	61.60	62.32	70.61

Quickstart

We provide simple examples illustrating the use of 360Zhinao2-7B-Base and 360Zhinao2-7B-Chat on 🤖ModelScope and 🤗Transformers.

Dependency Installation

python >= 3.8
pytorch >= 2.0
transformers >= 4.37.2
CUDA >= 11.4

pip install -r requirements.txt

Optionally, we recommend installing Flash-Attention 2 to improve performance and reduce memory footprint.

flash-attn >= 2.3.6

FLASH_ATTENTION_FORCE_BUILD=TRUE pip install flash-attn==2.3.6

🤗 Transformers

Demonstration of Base Model Inference

from transformers import AutoTokenizer, AutoModelForCausalLM
from transformers.generation import GenerationConfig

MODEL_NAME_OR_PATH = "qihoo360/360Zhinao2-7B-Base"

tokenizer = AutoTokenizer.from_pretrained(
    MODEL_NAME_OR_PATH, 
    trust_remote_code=True)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME_OR_PATH,
    device_map="auto",
    trust_remote_code=True)

generation_config = GenerationConfig.from_pretrained(
    MODEL_NAME_OR_PATH,
    trust_remote_code=True)

inputs = tokenizer('中国二十四节气\n1. 立春\n2. 雨水\n3. 惊蛰\n4. 春分\n5. 清明\n', return_tensors='pt')
inputs = inputs.to(model.device)

pred = model.generate(input_ids=inputs["input_ids"], generation_config=generation_config)
print("outputs:\n", tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))

Demonstration of Chat Model Inference

from transformers import AutoTokenizer, AutoModelForCausalLM
from transformers.generation import GenerationConfig

MODEL_NAME_OR_PATH = "qihoo360/360Zhinao2-7B-Chat-4K"

tokenizer = AutoTokenizer.from_pretrained(
    MODEL_NAME_OR_PATH, 
    trust_remote_code=True)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME_OR_PATH,
    device_map="auto",
    trust_remote_code=True)

generation_config = GenerationConfig.from_pretrained(
    MODEL_NAME_OR_PATH,
    trust_remote_code=True)

messages = []
#round-1
messages.append({"role": "user", "content": "介绍一下刘德华"})
response = model.chat(tokenizer=tokenizer, messages=messages, generation_config=generation_config)
messages.append({"role": "assistant", "content": response})
print(messages)

#round-2
messages.append({"role": "user", "content": "他有什么代表作？"})
response = model.chat(tokenizer=tokenizer, messages=messages, generation_config=generation_config)
messages.append({"role": "assistant", "content": response})
print(messages)

🤖 ModelScope

Demonstration of Base Model Inference

from modelscope import AutoModelForCausalLM, AutoTokenizer
from modelscope import GenerationConfig

MODEL_NAME_OR_PATH = "qihoo360/360Zhinao2-7B-Base"

tokenizer = AutoTokenizer.from_pretrained(
    MODEL_NAME_OR_PATH, 
    trust_remote_code=True)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME_OR_PATH,
    device_map="auto",
    trust_remote_code=True)

generation_config = GenerationConfig.from_pretrained(
    MODEL_NAME_OR_PATH,
    trust_remote_code=True)

inputs = tokenizer('中国二十四节气\n1. 立春\n2. 雨水\n3. 惊蛰\n4. 春分\n5. 清明\n', return_tensors='pt')
inputs = inputs.to(model.device)

pred = model.generate(input_ids=inputs["input_ids"], generation_config=generation_config)
print("outputs:\n", tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))

Demonstration of Chat Model Inference

from modelscope import AutoModelForCausalLM, AutoTokenizer
from modelscope import GenerationConfig

MODEL_NAME_OR_PATH = "qihoo360/360Zhinao2-7B-Chat-4K"

tokenizer = AutoTokenizer.from_pretrained(
    MODEL_NAME_OR_PATH, 
    trust_remote_code=True)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME_OR_PATH,
    device_map="auto",
    trust_remote_code=True)

generation_config = GenerationConfig.from_pretrained(
    MODEL_NAME_OR_PATH,
    trust_remote_code=True)

messages = []
#round-1
messages.append({"role": "user", "content": "介绍一下刘德华"})
response = model.chat(tokenizer=tokenizer, messages=messages, generation_config=generation_config)
messages.append({"role": "assistant", "content": response})
print(messages)

#round-2
messages.append({"role": "user", "content": "他有什么代表作？"})
response = model.chat(tokenizer=tokenizer, messages=messages, generation_config=generation_config)
messages.append({"role": "assistant", "content": response})
print(messages)

CLI Demo

Use terminal for command-line interface:

python cli_demo.py

Note: for Mac users, device = 'mps' is not supported yet.

Web Demo

streamlit run web_demo.py

API Demo

Launch api:

python openai_api.py

Then request with parameters:

curl 'http://localhost:8360/v1/chat/completions' \
-H 'Content-Type: application/json' \
-d '{
    "max_new_tokens": 200,
    "do_sample": true,
    "top_k": 0,
    "top_p": 0.8,
    "temperature": 1.0,
    "repetition_penalty": 1.0,
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "你好"}
    ]
}'

Model Inference

Quantization

We provide quantization schemes based on AutoGPTQ and release the Int4 quantization models.

Deployment

vLLM Installation

We recommend using vLLM==0.3.3.

If you are using CUDA 12.1 and PyTorch 2.1, you can install vLLM directly with:

pip install vllm==0.3.3

Otherwise, please refer to the official vLLM Installation Instructions.

After installation, perform the following steps:

Copy vllm/zhinao.py into vllm/model_executor/models in your vllm installation directory (in python/conda env).
Copy vllm/serving_chat.py into vllm/entrypoints/openai in your vllm installation directory.
Then add a line in vllm/model_executor/models/__init__.py
```
"ZhinaoForCausalLM": ("zhinao", "ZhinaoForCausalLM"),
```

vLLM Service Start

Start the service:

python -m vllm.entrypoints.openai.api_server \
    --served-model-name 360Zhinao2-7B-Chat-4K \
    --model qihoo360/360Zhinao2-7B-Chat-4K \
    --trust-remote-code \
    --tensor-parallel-size 1 \
    --max-model-len 4096 \
    --host 0.0.0.0 \
    --port 8360

Use curl to request the service:

curl http://localhost:8360/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
    "model": "360Zhinao2-7B-Chat-4K",
    "max_tokens": 200,
    "top_k": -1,
    "top_p": 0.8,
    "temperature": 1.0,
    "presence_penalty": 0.0,
    "frequency_penalty": 0.0,
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "你好"}
    ],
    "stop": [
        "<eod>",
        "<|im_end|>",
        "<|im_start|>"
    ]
}'

Use python to request the service:

from openai import OpenAI
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8360/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

chat_response = client.chat.completions.create(
    model="360Zhinao2-7B-Chat-4K",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "你好"},
    ],
    stop=[
        "<eod>",
        "<|im_end|>",
        "<|im_start|>"
    ],
    presence_penalty=0.0,
    frequency_penalty=0.0
)
print("Chat response:", chat_response)

If you need to enable repetition penalty, we recommend setting presence_penalty and frequency_penalty instead of repetition_penalty.

Model Finetune

Training data

Training Data: data/training_data_sample.json. This example data has 10,000 rows sampled from multiturn_chat_0.8M with converted format.

Data Format:

[
  {
    "id": 1,
    "conversations": [
        {
            "from": "system",
            "value": "You are a helpful assistant."
        },
        {
            "from": "user",
            "value": "您好啊"
        },
        {
            "from": "assistant",
            "value": "你好！我今天能为您做些什么？有什么问题或需要帮助吗? 我在这里为您提供服务。"
        }
    ]
  }
]

Finetuning scripts

set -x

HOSTFILE=hostfile
DS_CONFIG=./finetune/ds_config_zero2.json

# PARAMS
LR=5e-6
EPOCHS=3
MAX_LEN=4096
BATCH_SIZE=4
NUM_NODES=1
NUM_GPUS=8
MASTER_PORT=29500

IS_CONCAT=False # Whether to concatenate to maximum length (MAX_LEN)

DATA_PATH="./data/training_data_sample.json"
MODEL_PATH="qihoo360/360Zhinao2-7B-Base"
OUTPUT_DIR="./outputs/"

deepspeed --hostfile ${HOSTFILE} \
        --master_port ${MASTER_PORT} \
        --num_nodes ${NUM_NODES} \
        --num_gpus ${NUM_GPUS} \
        finetune.py \
        --report_to "tensorboard" \
        --data_path ${DATA_PATH} \
        --model_name_or_path ${MODEL_PATH} \
        --output_dir ${OUTPUT_DIR} \
        --model_max_length ${MAX_LEN} \
        --num_train_epochs ${EPOCHS} \
        --per_device_train_batch_size ${BATCH_SIZE} \
        --gradient_accumulation_steps 1 \
        --save_strategy steps \
        --save_steps 200 \
        --learning_rate ${LR} \
        --lr_scheduler_type cosine \
        --adam_beta1 0.9 \
        --adam_beta2 0.95 \
        --adam_epsilon 1e-8 \
        --max_grad_norm 1.0 \
        --weight_decay 0.1 \
        --warmup_ratio 0.01 \
        --gradient_checkpointing True \
        --bf16 True \
        --tf32 True \
        --deepspeed ${DS_CONFIG} \
        --is_concat ${IS_CONCAT} \
        --logging_steps 1 \
        --log_on_each_node False

bash finetune/ds_finetune.sh

Configuring HOSTFILE switches between single-machine and multi-machine training.
configuring ds_config switches between zero1, zero2 and zero3.
fp16, bf16 could configure mixed precision training. bf16 is recommended to be consistent with the pretrained model.
is_concat configures whether the training data is concatenated or not.

License

The source code of this repository follows the open-source license Apache 2.0.

360Zhinao open-source models support free commercial use. It is not necessary for you to submit a request for commercial usage.

qihoo360
/

360Zhinao2-7B-Chat-32K-Int4

360Zhinao2 (360智脑)

Introduction

News and Updates

Table of contents

Download URL

Model Evaluation

Base Model

Quickstart

Dependency Installation

🤗 Transformers

Demonstration of Base Model Inference

Demonstration of Chat Model Inference

🤖 ModelScope

Demonstration of Base Model Inference

Demonstration of Chat Model Inference

CLI Demo

Web Demo

API Demo

Model Inference

Quantization

Deployment

vLLM Installation

vLLM Service Start

Model Finetune

Training data

Finetuning scripts

License