chandar-lab

/

NeoBERT

like 92

Follow

Chandar Research Lab 28

This adapts the LlamaMLP from the llama modeling code in transformers to handle splitting the w12 weight during the forward pass, and uses it in case xformers is not available on the system.

This enables the model to be used on MacOS for example.