tainc commited on
Commit
a28abe3
1 Parent(s): 881b143

update links to new naming scheme

Browse files
Files changed (1) hide show
  1. README.md +11 -11
README.md CHANGED
@@ -19,7 +19,7 @@ SEA-LION stands for _Southeast Asian Languages In One Network_.
19
 
20
  ## Model Details
21
  ### Base model
22
- We perform instruction tuning in English and Indonesian on our [pre-trained SEA-LION-7B](https://huggingface.co/aisingapore/sealion7b), a decoder model using the MPT architecture, to create SEA-LION-7B-Instruct.
23
 
24
  ### Benchmark Performance
25
  We evaluated SEA-LION-7B-Instruct on the BHASA benchmark ([arXiv](https://arxiv.org/abs/2309.06085v2) and [GitHub](https://github.com/aisingapore/bhasa)) across a variety of tasks.
@@ -28,15 +28,15 @@ BHASA stands out amongst other evaluations for SEA languages for its holistic ap
28
 
29
  The scores shown in the table below have been adjusted to only consider answers provided in the appropriate language.
30
 
31
- | Model | QA (F1) | Sentiment (F1) | Toxicity (F1) | Eng>Indo (ChrF++) | Indo>Eng (ChrF++) | Summary (ROUGE-L) | NLI (Acc) | Causal (Acc) |
32
  |--------------------------------|---------|----------------|---------------|-------------------|-------------------|-------------------|-----------|--------------|
33
- | SEA-LION-7B-Instruct-Research | 24.86 | 76.13 | 24.45 | 52.50 | 46.82 | 15.44 | 33.20 | 23.80 |
34
- | SEA-LION-7B-Instruct | 68.41 | 91.45 | 17.98 | 57.48 | 58.04 | 17.54 | 53.10 | 60.80 |
35
- | SeaLLM 7B v1 | 30.96 | 56.29 | 22.60 | 62.23 | 41.55 | 14.03 | 26.50 | 56.60 |
36
- | SeaLLM 7B v2 | 44.40 | 80.13 | 55.24 | 64.01 | 63.28 | 17.31 | 43.60 | 82.00 |
37
- | Sailor-7B | 65.43 | 59.48 | 20.48 | 64.27 | 60.68 | 8.69 | 15.10 | 38.40 |
38
- | Llama 2 7B Chat | 11.12 | 52.32 | 0.00 | 44.09 | 57.58 | 9.24 | 0.00 | 0.00 |
39
- | Mistral 7B Instruct v0.1 | 38.85 | 74.38 | 20.83 | 30.60 | 51.43 | 15.63 | 28.60 | 50.80 |
40
  | GPT-4 | 73.60 | 74.14 | 63.96 | 69.38 | 67.53 | 18.71 | 83.20 | 96.00 |
41
 
42
  ### Usage
@@ -46,8 +46,8 @@ SEA-LION can be run using the 🤗 Transformers library
46
 
47
  from transformers import AutoModelForCausalLM, AutoTokenizer
48
 
49
- tokenizer = AutoTokenizer.from_pretrained("aisingapore/sealion7b-instruct", trust_remote_code=True)
50
- model = AutoModelForCausalLM.from_pretrained("aisingapore/sealion7b-instruct", trust_remote_code=True)
51
 
52
  prompt_template = "### USER:\n{human_prompt}\n\n### RESPONSE:\n"
53
  prompt = """Apa sentimen dari kalimat berikut ini?
 
19
 
20
  ## Model Details
21
  ### Base model
22
+ We perform instruction tuning in English and Indonesian on our [pre-trained SEA-LION-7B](https://huggingface.co/aisingapore/sea-lion-7b), a decoder model using the MPT architecture, to create SEA-LION-7B-Instruct.
23
 
24
  ### Benchmark Performance
25
  We evaluated SEA-LION-7B-Instruct on the BHASA benchmark ([arXiv](https://arxiv.org/abs/2309.06085v2) and [GitHub](https://github.com/aisingapore/bhasa)) across a variety of tasks.
 
28
 
29
  The scores shown in the table below have been adjusted to only consider answers provided in the appropriate language.
30
 
31
+ | Model                          | QA (F1) | Sentiment (F1) | Toxicity (F1) | Eng>Indo (ChrF++) | Indo>Eng (ChrF++) | Summary (ROUGE-L) | NLI (Acc) | Causal (Acc) |
32
  |--------------------------------|---------|----------------|---------------|-------------------|-------------------|-------------------|-----------|--------------|
33
+ | SEA-LION-7B-Instruct-Research  | 24.86   | 76.13          | 24.45         | 52.50             | 46.82             | 15.44             | 33.20     | 23.80        |
34
+ | SEA-LION-7B-Instruct           | **68.41**   | **91.45**          | 17.98         | 57.48             | 58.04             | **17.54**             | 53.10     | 60.80        |
35
+ | SeaLLM 7B v1                   | 30.96   | 56.29          | 22.60         | 62.23             | 41.55             | 14.03             | 26.50     | 56.60        |
36
+ | SeaLLM 7B v2                   | 44.40   | 80.13          | **55.24**         | 64.01             | **63.28**             | 17.31             | 43.60     | **82.00**        |
37
+ | Sailor-7B                      | 65.43   | 59.48          | 20.48         | **64.27**             | 60.68             | 8.69              | 15.10     | 38.40        |
38
+ | Llama 2 7B Chat                | 11.12   | 52.32          | 0.00          | 44.09             | 57.58             | 9.24              | 0.00      | 0.00         |
39
+ | Mistral 7B Instruct v0.1       | 38.85   | 74.38          | 20.83         | 30.60             | 51.43             | 15.63             | **53.10**     | 50.80        |
40
  | GPT-4 | 73.60 | 74.14 | 63.96 | 69.38 | 67.53 | 18.71 | 83.20 | 96.00 |
41
 
42
  ### Usage
 
46
 
47
  from transformers import AutoModelForCausalLM, AutoTokenizer
48
 
49
+ tokenizer = AutoTokenizer.from_pretrained("aisingapore/sea-lion-7b-instruct", trust_remote_code=True)
50
+ model = AutoModelForCausalLM.from_pretrained("aisingapore/sea-lion-7b-instruct", trust_remote_code=True)
51
 
52
  prompt_template = "### USER:\n{human_prompt}\n\n### RESPONSE:\n"
53
  prompt = """Apa sentimen dari kalimat berikut ini?