Jian-Gang commited on
Commit
d12aece
1 Parent(s): 45d3b8e

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +7 -1
README.md CHANGED
@@ -27,7 +27,7 @@ SEA-LION stands for <i>Southeast Asian Languages In One Network</i>.
27
  - **Developed by:** Products Pillar, AI Singapore
28
  - **Funded by:** Singapore NRF
29
  - **Model type:** Decoder
30
- - **Languages supported:** English, Chinese, Vietnamese, Indonesian, Thai, Filipino, Tamil, Malay, Khmer, Lao, Burmese
31
  - **License:** [Llama 3.1 Community License](https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/LICENSE)
32
 
33
  ## Model Details
@@ -47,8 +47,14 @@ Note: SEA-HELM is implemented using prompts to elicit answers in a strict format
47
 
48
  The evaluation was done **five-shot** with native prompts on a sample of 100-1000 instances for each dataset.
49
 
 
 
50
  For more details on Llama3.1 8B CPT SEA-LIONv3 base benchmark performance, please refer to the SEA-HELM leaderboard, https://leaderboard.sea-lion.ai/
51
 
 
 
 
 
52
  ## Technical Specifications
53
  ### Infrastructure
54
  Llama3.1 8B CPT SEA-LIONv3 was trained using [MosaicML Composer](https://github.com/mosaicml/composer) on the following hardware:
 
27
  - **Developed by:** Products Pillar, AI Singapore
28
  - **Funded by:** Singapore NRF
29
  - **Model type:** Decoder
30
+ - **Languages supported:** Burmese, Chinese, English, Filipino, Indonesia, Khmer, Lao, Malay, Tamil, Thai, Vietnamese
31
  - **License:** [Llama 3.1 Community License](https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/LICENSE)
32
 
33
  ## Model Details
 
47
 
48
  The evaluation was done **five-shot** with native prompts on a sample of 100-1000 instances for each dataset.
49
 
50
+ Following the implementation of IFEval in OpenLLM leaderboard, we also implement SEA-IFEval to provide a comparison of the ability of the model to follow specific constraints in English and in SEA languages.
51
+
52
  For more details on Llama3.1 8B CPT SEA-LIONv3 base benchmark performance, please refer to the SEA-HELM leaderboard, https://leaderboard.sea-lion.ai/
53
 
54
+ **SEA-IFEval**
55
+
56
+ SEA-IFEval evaluates a model's ability to adhere to constraints provided in the prompt, for example beginning a response with a specific word/phrase or answering with a certain number of sections. Additionally, accuracy is normalised by the proportion of responses in the correct language (if the model performs the task correctly but responds in the wrong language, it is judged to have failed the task).
57
+
58
  ## Technical Specifications
59
  ### Infrastructure
60
  Llama3.1 8B CPT SEA-LIONv3 was trained using [MosaicML Composer](https://github.com/mosaicml/composer) on the following hardware: