crumb commited on
Commit
f1d32d9
1 Parent(s): cb1d051

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +2 -0
README.md CHANGED
@@ -18,6 +18,8 @@ A demo that runs in free Google Colab can be run here: https://bit.ly/3K1P4PQ ju
18
 
19
  The [EleutherAI/gpt-j-6B](https://hf.co/EleutherAI/gpt-j-6B) model finetuned on the [Alpaca](https://huggingface.co/datasets/tatsu-lab/alpaca) instruction dataset with [low rank adaptation](https://arxiv.org/abs/2106.09685). This is not a model from Eleuther but a personal project.
20
 
 
 
21
  ## Use:
22
 
23
  ```python
 
18
 
19
  The [EleutherAI/gpt-j-6B](https://hf.co/EleutherAI/gpt-j-6B) model finetuned on the [Alpaca](https://huggingface.co/datasets/tatsu-lab/alpaca) instruction dataset with [low rank adaptation](https://arxiv.org/abs/2106.09685). This is not a model from Eleuther but a personal project.
20
 
21
+ Don't knock LoRA, all it is is finetuning how the internal representations should change (simplified, the residual of the weights) instead of finetuning just the internal representations! All the previous weights are in tact meaning LoRA tuning makes the model less likely to forget what it was trained on, and also less likely to push the model into mode collapse. Check table 2 of the LoRA paper and you can see that LoRA sometimes outperforms traditional finetuning as well.
22
+
23
  ## Use:
24
 
25
  ```python