sagarshf commited on
Commit
cc2ea75
·
verified ·
1 Parent(s): cf4d943

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +10 -13
README.md CHANGED
@@ -35,23 +35,20 @@ Detailed blog [here](https://smallstep.ai/making-misal).
35
  ## Evaluation :
36
  We did a manual round of evaluations using internet data. This is a fairly small dataset with 100 questions taken from the internet. We understand that a better evaluation method is needed to benchmark our model, this being the first iteration we decided to proceed with manual evaluation. Our main aim was to see if the model understands basic instructions, if so how well is it able to understand it, hence we have limited our evaluation to Reading comprehension, Translation, Sentiment Analysis, Paraphrasing like tasks.
37
 
38
- | | Misal | ChatGPT3.5 | Krutrim | MahaMarathi |
39
- | --------------------- | ----- | ---------- | ------- | ----------- |
40
- | reading comprehension | **88** | 68 | 40 | 0 |
41
- | sentiment analysis | 68 | **76** | 60 | 0 |
42
- | paraphrase | 92 | **100** | 88 | 0 |
43
- | translation | 76 | **96** | 80 | 0 |
44
- | average | 81 | **85** | 67 | 0 |
 
45
 
46
  We have released the evaluation data here:
47
  - [Manual Evaluation Set](https://huggingface.co/datasets/smallstepai/Misal-Evaluation-v0.1)
48
 
49
- ## Summary :
50
-
51
- Our model beats ChatGPT 3.5 at reading comprehension.
52
- While we are not able to beat ChatGPT 3.5 on remaining tasks like sentiment analysis, paraphrasing, translation, our model beats Ola Krutrim at all the tasks except translation.
53
 
54
- ![image/png](https://framerusercontent.com/images/BB9D44882aH8mL5Pf9Ps4lc.jpeg)
55
 
56
 
57
  ## License
@@ -103,6 +100,6 @@ resp = ask_misal(model, tokenizer, instruction=instruction, max_new_tokens=1024)
103
  print(resp)
104
  ```
105
 
106
- ### Team
107
 
108
  Sagar Sarkale, Abhijeet Katte, Prasad Mane, Shravani Chavan
 
35
  ## Evaluation :
36
  We did a manual round of evaluations using internet data. This is a fairly small dataset with 100 questions taken from the internet. We understand that a better evaluation method is needed to benchmark our model, this being the first iteration we decided to proceed with manual evaluation. Our main aim was to see if the model understands basic instructions, if so how well is it able to understand it, hence we have limited our evaluation to Reading comprehension, Translation, Sentiment Analysis, Paraphrasing like tasks.
37
 
38
+ | Model | Reading Comprehension | Sentiment Analysis | Paraphrase | Translation | Average |
39
+ |-------------|-----------------------|--------------------|------------|-------------|---------|
40
+ | Misal-7B | 88 | 68 | 92 | 76 | 81 |
41
+ | Misal-1B | 48 | 68 | 72 | 36 | 56 |
42
+ | ChatGPT3.5 | 68 | 76 | 100 | 96 | 85 |
43
+ | Krutrim | 40 | 60 | 88 | 80 | 67 |
44
+ | MahaMarathi | 0 | 0 | 0 | 0 | 0 |
45
+
46
 
47
  We have released the evaluation data here:
48
  - [Manual Evaluation Set](https://huggingface.co/datasets/smallstepai/Misal-Evaluation-v0.1)
49
 
 
 
 
 
50
 
51
+ ![image/png](https://framerusercontent.com/images/oYRJ925hmTBDjd6RMucvD1qtl7s.jpeg)
52
 
53
 
54
  ## License
 
100
  print(resp)
101
  ```
102
 
103
+ ## Team
104
 
105
  Sagar Sarkale, Abhijeet Katte, Prasad Mane, Shravani Chavan