Commit
•
6a87c45
1
Parent(s):
014fe2e
Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -71,76 +71,80 @@ Hermes 3 is competitive, if not superior, to Llama-3.1 Instruct models at genera
|
|
| 71 |
|
| 72 |
## GPT4All:
|
| 73 |
```
|
| 74 |
-
|
|
| 75 |
-
|
| 76 |
-
|arc_challenge|
|
| 77 |
-
|
|
| 78 |
-
|arc_easy
|
| 79 |
-
|
|
| 80 |
-
|boolq
|
| 81 |
-
|hellaswag
|
| 82 |
-
|
|
| 83 |
-
|openbookqa
|
| 84 |
-
|
|
| 85 |
-
|piqa
|
| 86 |
-
|
|
| 87 |
-
|winogrande
|
| 88 |
```
|
| 89 |
|
| 90 |
-
Average:
|
| 91 |
|
| 92 |
## AGIEval:
|
| 93 |
```
|
| 94 |
-
|
|
| 95 |
-
|
| 96 |
-
|agieval_aqua_rat
|
| 97 |
-
|
|
| 98 |
-
|agieval_logiqa_en
|
| 99 |
-
|
|
| 100 |
-
|agieval_lsat_ar
|
| 101 |
-
|
|
| 102 |
-
|agieval_lsat_lr
|
| 103 |
-
|
|
| 104 |
-
|agieval_lsat_rc
|
| 105 |
-
|
|
| 106 |
-
|agieval_sat_en
|
| 107 |
-
|
|
| 108 |
-
|agieval_sat_en_without_passage|
|
| 109 |
-
|
|
| 110 |
-
|agieval_sat_math
|
| 111 |
-
|
|
| 112 |
```
|
| 113 |
|
| 114 |
-
Average:
|
| 115 |
|
| 116 |
## BigBench:
|
| 117 |
|
| 118 |
```
|
| 119 |
-
|
| 120 |
-
|
| 121 |
-
|
| 122 |
-
|
|
| 123 |
-
|
|
| 124 |
-
|
|
| 125 |
-
|
|
| 126 |
-
|
|
| 127 |
-
|
|
| 128 |
-
|
|
| 129 |
-
|
|
| 130 |
-
|
|
| 131 |
-
|
|
| 132 |
-
|
|
| 133 |
-
|
|
| 134 |
-
|
|
| 135 |
-
|
|
| 136 |
-
|
|
| 137 |
-
|
|
| 138 |
-
|
|
| 139 |
-
|
|
| 140 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 141 |
```
|
| 142 |
|
| 143 |
-
Average:
|
| 144 |
|
| 145 |
|
| 146 |
# Prompt Format
|
|
@@ -183,7 +187,7 @@ To utilize the prompt format without a system prompt, simply leave the line out.
|
|
| 183 |
|
| 184 |
## Prompt Format for Function Calling
|
| 185 |
|
| 186 |
-
# Note:
|
| 187 |
|
| 188 |
Our model was trained on specific system prompts and structures for Function Calling.
|
| 189 |
|
|
@@ -212,7 +216,7 @@ The model will then generate a tool call, which your inference code must parse,
|
|
| 212 |
|
| 213 |
Once you parse the tool call, call the api and get the returned values for the call, and pass it back in as a new role, `tool` like so:
|
| 214 |
```
|
| 215 |
-
<|im_start|>
|
| 216 |
<tool_response>
|
| 217 |
{"name": "get_stock_fundamentals", "content": {'symbol': 'TSLA', 'company_name': 'Tesla, Inc.', 'sector': 'Consumer Cyclical', 'industry': 'Auto Manufacturers', 'market_cap': 611384164352, 'pe_ratio': 49.604652, 'pb_ratio': 9.762013, 'dividend_yield': None, 'eps': 4.3, 'beta': 2.427, '52_week_high': 299.29, '52_week_low': 152.37}}
|
| 218 |
</tool_response>
|
|
@@ -318,6 +322,3 @@ GGUF Quants: https://huggingface.co/NousResearch/Hermes-3-Llama-3.2-3B-GGUF
|
|
| 318 |
url={https://arxiv.org/abs/2408.11857},
|
| 319 |
}
|
| 320 |
```
|
| 321 |
-
|
| 322 |
-
|
| 323 |
-
|
|
|
|
| 71 |
|
| 72 |
## GPT4All:
|
| 73 |
```
|
| 74 |
+
| Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr|
|
| 75 |
+
|-------------|------:|------|-----:|--------|---|-----:|---|-----:|
|
| 76 |
+
|arc_challenge| 1|none | 0|acc |↑ |0.4411|± |0.0145|
|
| 77 |
+
| | |none | 0|acc_norm|↑ |0.4377|± |0.0145|
|
| 78 |
+
|arc_easy | 1|none | 0|acc |↑ |0.7399|± |0.0090|
|
| 79 |
+
| | |none | 0|acc_norm|↑ |0.6566|± |0.0097|
|
| 80 |
+
|boolq | 2|none | 0|acc |↑ |0.8327|± |0.0065|
|
| 81 |
+
|hellaswag | 1|none | 0|acc |↑ |0.5453|± |0.0050|
|
| 82 |
+
| | |none | 0|acc_norm|↑ |0.7047|± |0.0046|
|
| 83 |
+
|openbookqa | 1|none | 0|acc |↑ |0.3480|± |0.0213|
|
| 84 |
+
| | |none | 0|acc_norm|↑ |0.4280|± |0.0221|
|
| 85 |
+
|piqa | 1|none | 0|acc |↑ |0.7639|± |0.0099|
|
| 86 |
+
| | |none | 0|acc_norm|↑ |0.7584|± |0.0100|
|
| 87 |
+
|winogrande | 1|none | 0|acc |↑ |0.6590|± |0.0133|
|
| 88 |
```
|
| 89 |
|
| 90 |
+
Average: 64.00
|
| 91 |
|
| 92 |
## AGIEval:
|
| 93 |
```
|
| 94 |
+
| Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr|
|
| 95 |
+
|------------------------------|------:|------|-----:|--------|---|-----:|---|-----:|
|
| 96 |
+
|agieval_aqua_rat | 1|none | 0|acc |↑ |0.2283|± |0.0264|
|
| 97 |
+
| | |none | 0|acc_norm|↑ |0.2441|± |0.0270|
|
| 98 |
+
|agieval_logiqa_en | 1|none | 0|acc |↑ |0.3057|± |0.0181|
|
| 99 |
+
| | |none | 0|acc_norm|↑ |0.3272|± |0.0184|
|
| 100 |
+
|agieval_lsat_ar | 1|none | 0|acc |↑ |0.2304|± |0.0278|
|
| 101 |
+
| | |none | 0|acc_norm|↑ |0.1957|± |0.0262|
|
| 102 |
+
|agieval_lsat_lr | 1|none | 0|acc |↑ |0.3784|± |0.0215|
|
| 103 |
+
| | |none | 0|acc_norm|↑ |0.3588|± |0.0213|
|
| 104 |
+
|agieval_lsat_rc | 1|none | 0|acc |↑ |0.4610|± |0.0304|
|
| 105 |
+
| | |none | 0|acc_norm|↑ |0.4275|± |0.0302|
|
| 106 |
+
|agieval_sat_en | 1|none | 0|acc |↑ |0.6019|± |0.0342|
|
| 107 |
+
| | |none | 0|acc_norm|↑ |0.5340|± |0.0348|
|
| 108 |
+
|agieval_sat_en_without_passage| 1|none | 0|acc |↑ |0.3981|± |0.0342|
|
| 109 |
+
| | |none | 0|acc_norm|↑ |0.3981|± |0.0342|
|
| 110 |
+
|agieval_sat_math | 1|none | 0|acc |↑ |0.2500|± |0.0293|
|
| 111 |
+
| | |none | 0|acc_norm|↑ |0.2636|± |0.0298|
|
| 112 |
```
|
| 113 |
|
| 114 |
+
Average: 34.36
|
| 115 |
|
| 116 |
## BigBench:
|
| 117 |
|
| 118 |
```
|
| 119 |
+
| Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr|
|
| 120 |
+
|-------------------------------------------------------|------:|------|-----:|--------|---|-----:|---|-----:|
|
| 121 |
+
|leaderboard_bbh_boolean_expressions | 1|none | 3|acc_norm|↑ |0.7560|± |0.0272|
|
| 122 |
+
|leaderboard_bbh_causal_judgement | 1|none | 3|acc_norm|↑ |0.6043|± |0.0359|
|
| 123 |
+
|leaderboard_bbh_date_understanding | 1|none | 3|acc_norm|↑ |0.3280|± |0.0298|
|
| 124 |
+
|leaderboard_bbh_disambiguation_qa | 1|none | 3|acc_norm|↑ |0.5880|± |0.0312|
|
| 125 |
+
|leaderboard_bbh_formal_fallacies | 1|none | 3|acc_norm|↑ |0.5280|± |0.0316|
|
| 126 |
+
|leaderboard_bbh_geometric_shapes | 1|none | 3|acc_norm|↑ |0.3560|± |0.0303|
|
| 127 |
+
|leaderboard_bbh_hyperbaton | 1|none | 3|acc_norm|↑ |0.6280|± |0.0306|
|
| 128 |
+
|leaderboard_bbh_logical_deduction_five_objects | 1|none | 3|acc_norm|↑ |0.3400|± |0.0300|
|
| 129 |
+
|leaderboard_bbh_logical_deduction_seven_objects | 1|none | 3|acc_norm|↑ |0.2880|± |0.0287|
|
| 130 |
+
|leaderboard_bbh_logical_deduction_three_objects | 1|none | 3|acc_norm|↑ |0.4160|± |0.0312|
|
| 131 |
+
|leaderboard_bbh_movie_recommendation | 1|none | 3|acc_norm|↑ |0.6760|± |0.0297|
|
| 132 |
+
|leaderboard_bbh_navigate | 1|none | 3|acc_norm|↑ |0.5800|± |0.0313|
|
| 133 |
+
|leaderboard_bbh_object_counting | 1|none | 3|acc_norm|↑ |0.3640|± |0.0305|
|
| 134 |
+
|leaderboard_bbh_penguins_in_a_table | 1|none | 3|acc_norm|↑ |0.3836|± |0.0404|
|
| 135 |
+
|leaderboard_bbh_reasoning_about_colored_objects | 1|none | 3|acc_norm|↑ |0.3560|± |0.0303|
|
| 136 |
+
|leaderboard_bbh_ruin_names | 1|none | 3|acc_norm|↑ |0.4160|± |0.0312|
|
| 137 |
+
|leaderboard_bbh_salient_translation_error_detection | 1|none | 3|acc_norm|↑ |0.3080|± |0.0293|
|
| 138 |
+
|leaderboard_bbh_snarks | 1|none | 3|acc_norm|↑ |0.5618|± |0.0373|
|
| 139 |
+
|leaderboard_bbh_sports_understanding | 1|none | 3|acc_norm|↑ |0.6600|± |0.0300|
|
| 140 |
+
|leaderboard_bbh_temporal_sequences | 1|none | 3|acc_norm|↑ |0.2320|± |0.0268|
|
| 141 |
+
|leaderboard_bbh_tracking_shuffled_objects_five_objects | 1|none | 3|acc_norm|↑ |0.1640|± |0.0235|
|
| 142 |
+
|leaderboard_bbh_tracking_shuffled_objects_seven_objects| 1|none | 3|acc_norm|↑ |0.1480|± |0.0225|
|
| 143 |
+
|leaderboard_bbh_tracking_shuffled_objects_three_objects| 1|none | 3|acc_norm|↑ |0.3120|± |0.0294|
|
| 144 |
+
|leaderboard_bbh_web_of_lies | 1|none | 3|acc_norm|↑ |0.5080|± |0.0317|
|
| 145 |
```
|
| 146 |
|
| 147 |
+
Average: 43.76
|
| 148 |
|
| 149 |
|
| 150 |
# Prompt Format
|
|
|
|
| 187 |
|
| 188 |
## Prompt Format for Function Calling
|
| 189 |
|
| 190 |
+
# Note: A previous version used USER as both the user prompt and the tool response role, but this has now been fixed. Please use USER for the user prompt role and TOOL for the tool response role.
|
| 191 |
|
| 192 |
Our model was trained on specific system prompts and structures for Function Calling.
|
| 193 |
|
|
|
|
| 216 |
|
| 217 |
Once you parse the tool call, call the api and get the returned values for the call, and pass it back in as a new role, `tool` like so:
|
| 218 |
```
|
| 219 |
+
<|im_start|>tool
|
| 220 |
<tool_response>
|
| 221 |
{"name": "get_stock_fundamentals", "content": {'symbol': 'TSLA', 'company_name': 'Tesla, Inc.', 'sector': 'Consumer Cyclical', 'industry': 'Auto Manufacturers', 'market_cap': 611384164352, 'pe_ratio': 49.604652, 'pb_ratio': 9.762013, 'dividend_yield': None, 'eps': 4.3, 'beta': 2.427, '52_week_high': 299.29, '52_week_low': 152.37}}
|
| 222 |
</tool_response>
|
|
|
|
| 322 |
url={https://arxiv.org/abs/2408.11857},
|
| 323 |
}
|
| 324 |
```
|
|
|
|
|
|
|
|
|