File size: 535 Bytes
6c859be |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
---
license: apache-2.0
datasets:
- athirdpath/Merge_Glue
---
### TeeZee/NEBULA-XB-v1.0_SFT_2_epoch ###
Experiment, can DUS be taken one or more steps further?
### Technical notes:
- pretrained model NEBULA-XB-v1.0 finetuned on 30k entries from Merge_Glue dataset
- 18 layers removed from both models of finetuned GALAXY-XB-v03
- model has 108 layers (((48-12)*2)-18)*2 = 108
- second step in scaling DUS procedure
### To evaluate
- model performance after merge, should be a little lover that GALAXY finetuned on 50k of slimorca |