Update README.md

README.md

@@ -65,16 +65,15 @@ This repo contains the following configurations under `./models/`:
     + Addendum: this replaced the `ar+nar-llama-8` as the defacto model (taking its name), so the above does apply.
 
 * `config.llama[layerskip].yaml` / `ar+nar-layerskip-llama-8`: The above, but with very brief training for LayerSkip:
-    + Trained on a small English subset of Emilia and a small private corpus, and Japanese+French+German from Emilia.
+    + Post-trained on a small English subset of Emilia and a small private corpus, and Japanese+French+German from Emilia.
     + Using shuffled batches (where each batch has the same durations) and a modified `rvq_levels_p` to help the NAR.
     + This model received LayerSkip-aware training, with layer dropout and early-exit loss to help try and bolster the model and enable self-speculation sampling.
     + I *need* to do heavy evaluation against the base model to ensure output quality does not drop before considering replacing the base model with this.
-      + It currently does not seem to perform better even without early-exit...
     + Goal is to utilize self-speculation sampling to enable speedups when possible.
-      + Current implementation will early-exit if the entropy/varentropy of the logits are low enough.
-      + There doesn't seem to be any significant speedups...
-    + Training is a pain, as float16 + AMP will fry the model fast, and training bfloat16 (with/without AMP) seems to harm the model overall.
-      + I'd like to think more time training will help, but it doesn't seem to be worth it for a marginal speedup.
+      + Current implementation will early-exit if the entropy/varentropy of the logits are low enough
+    + Training is a pain.
+      + LayerSkip-aware training does *not* like to train under ROCm.
+      + Training under float16+AMP with loss scaling will fry the model with a large enough de facto batch size (>512 samples/update step) and/or too low of a loss scale (<=8K).
 
 Some additional configurations have been explored with, but experiments have not been fruitful:
 * Exotic wrappers like `BitNet` seemed to yield little gains in inferencing, somehow. The memory savings is pretty much unneccessary as the models are already manageable at ~200M parameters.