treehugg3 commited on
Commit
8507cc0
·
verified ·
1 Parent(s): 0428d78

Update with imatrix training dataset info

Browse files
Files changed (1) hide show
  1. README.md +12 -0
README.md CHANGED
@@ -149,6 +149,18 @@ is unfortunately very frequent).
149
  * RAM: A lot of RAM is required to compute imatrix files. Example: 512 GB is just enough to compute 405B imatrix quants in Q8.
150
  * GPU: At least 8 GB of memory.
151
 
 
 
 
 
 
 
 
 
 
 
 
 
152
  ### Extra tips
153
 
154
  * Computing 405B imatrix quants in Q8 does not seem to have any noticeable quality impact compared to BF16, so to save on hardware
 
149
  * RAM: A lot of RAM is required to compute imatrix files. Example: 512 GB is just enough to compute 405B imatrix quants in Q8.
150
  * GPU: At least 8 GB of memory.
151
 
152
+ ### Dataset
153
+
154
+ * You want to create a dataset that is around double the size of bartowski1182's imatrix dataset. Quality is far more important
155
+ than size. If you don't mind long training times, you can make it massive, but if you go beyond 1 MB there will
156
+ probably be diminishing returns.
157
+ * Your imatrix dataset should contain the typical output the model would generate when used for the workload you plan on using
158
+ the model for. If you plan on using the model as a programming assistant, your imatrix dataset should contain the typical code
159
+ you would ask it to write. The same applies for language. Our dataset is mostly English. If one would use our imatrix models in
160
+ a different language they will likely perform worse than static quants as only a very small portion of our imatrix training data
161
+ is multilingual. We only have the resources to generate single generic imatrix quants so our imatrix dataset must contain examples
162
+ of every common use-case of an LLM.
163
+
164
  ### Extra tips
165
 
166
  * Computing 405B imatrix quants in Q8 does not seem to have any noticeable quality impact compared to BF16, so to save on hardware