mustafaaljadery

/

gemma-2B-10M

like 225

This is from the original Gemma Config *kwargs, the segment_size is what scales attention. It splits the attention into recurrent segment_size chunks per https://arxiv.org/abs/2404.07143 🙂