Control-Vectors with tabbyAPI and Exui (using ExllamaV2, via forked repos)
#13
by
gghfez
- opened
TabbyAPI - https://github.com/gapeleon/tabbyAPI-control-vectors
Exui - https://github.com/gapeleon/exui-control-vectors
They don't work with tensor parallel. Still happy to be able to use exl2 and clear out a couple of terabytes worth of .gguf files :)
If you already use exui and try this fork, I suggest backing up ~/exui first as this fork uses the same config directory and will write the control-vector config into models.json (which might break compatibility if you go back to the original exui?)
Also, I ran them in a separate venv since they required additional dependencies (gguf library and downgrading numpy to <= 2.0 )