cekal

/

mpt-7b-peft-compatible

like 12

Thanks for your shared colab!
https://colab.research.google.com/drive/1iBeY5UTLHE3aL6yNLiCIJHOBDqWBYbi5?usp=sharing#scrollTo=VXNEwdDL83NK

I managed to get the trainable params as follows:
trainable params: 4194304 || all params: 6653480960 || trainable%: 0.0630392425441013

Two question need your guidance:
a. Is that possible to runs it with 'PromptTuningConfig' instead of 'LoraConfig' ?
It need me to input the parameters of num_attention_heads and num_layers, but I don't know where to find it.

b. for the stated trainable params, how do we know when the training data is larger the the 'trainable params' ?

Appreciate for your work, MosaicML seems to more focus on the the whole finetuning approach instead of PEFT.