|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- athirdpath/Merge_Glue |
|
--- |
|
|
|
### TeeZee/NEBULA-XB-v1.0_SFT_2_epoch ### |
|
|
|
Experiment, can DUS be taken one or more steps further? |
|
|
|
|
|
### Technical notes: |
|
- pretrained model NEBULA-XB-v1.0 finetuned on 30k entries from Merge_Glue dataset |
|
- 18 layers removed from both models of finetuned GALAXY-XB-v03 |
|
- model has 108 layers (((48-12)*2)-18)*2 = 108 |
|
- second step in scaling DUS procedure |
|
|
|
|
|
### To evaluate |
|
- model performance after merge, should be a little lover that GALAXY finetuned on 50k of slimorca |
