|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- augmxnt/ultra-orca-boros-en-ja-v1 |
|
language: |
|
- ja |
|
- en |
|
tags: |
|
- jamba |
|
- axolotl |
|
--- |
|
Over the weekend after a failed initial run I got excited by Pete's success [Jamba Tuning](https://huggingface.co/lightblue/Jamba-v0.1-chat-multilingual) and decided to throw a little compute on a similar-sized dataset (the main [shisa-v1 bilingual tuning set](https://huggingface.co/datasets/augmxnt/ultra-orca-boros-en-ja-v1)). |
|
|
|
Like my [initial runs](https://api.wandb.ai/links/augmxnt/h4mc4dd5), [training graphs look fine](https://wandb.ai/augmxnt/shisa-v2/runs/o830e1kw), but the results were less than spectacular. |
|
|
|
Here are the JA MT-Bench evals for the 2416 checkpoint (eval/loss plateau) and the 4228 (3 epoch) tune: |
|
|
|
``` |
|
shisa-jamba-v1-checkpoint-2416 2.491525 |
|
shisa-jamba-v1-checkpoint-4228 2.508475 |
|
``` |
|
|
|
You can view the answers in the repo (lots of repetitions and nonsense) and compare to [proper JA MT-Bench scores](https://github.com/AUGMXNT/shisa/wiki/Evals-:-JA-MT%E2%80%90Bench) from my testing. |
|
|
|
While an "unsuccessful" experiment, it was still worth the practice, although I got a little excited and should have gone w/ my more typical lighter testing obviously. |
|
|
|
This kicks off official `shisa-v2` base model evaluation. I was a bit hesitant about throwing this model out there (since it's useless as an artifact), but since I've actually made the in-process code available while working on it, I'll share this as well just in case (and to do this writeup). |
|
|
|
Here is the current full code/steps for Axolotl training and eval (modified llm-judge inferencing code): |
|
- https://github.com/shisa-ai/shisa-v2/tree/main/_base-evals/jamba/axolotl |
|
- https://github.com/shisa-ai/shisa-v2/tree/main/_base-evals/jamba/eval |
|
|
|
Thanks to Pete for the useful [initial report](https://wandb.ai/peterd/axolotl/reports/Jamba-chat-finetuning-Airoboros-and-ShareGPT---Vmlldzo3MzUwNTc1?accessToken=syabbmhblmwslnizuwws62tig3f0op75zh2jm2owbranaziq7thbgizl78nnowgc) and the [axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) team for their [fast integration of Jamba](https://github.com/OpenAccess-AI-Collective/axolotl/issues/1449) (way better than my [raw tune code](https://github.com/shisa-ai/shisa-v2/tree/main/_base-evals/jamba)). |
