Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
wassemgtk 
posted an update 7 days ago
Post
2687
I’ve been diving into the iRoPE architecture from Llama 4—a game-changer for long-context models! It interleaves local attention (with RoPE) for short contexts and global attention (with inference-time temp scaling) for long-range reasoning, aiming for infinite context. I’m going to try writing iRoPE—who wants to help?

Code: https://github.com/wassemgtk/iRoPE-try/blob/main/iRoPE.ipynb
wassemgtk

Applied iRoPE to any model like LLaMA 3.2-3B VERY possible! Interleaved local (RoPE) & global (temp-scaled) attention boosts long-context (10M tokens) handling. With chunking & weight transfer, it’s adaptable to "any" transformer model.
Infinite context feels closer 🤯

https://github.com/wassemgtk/iRoPE-try

In this post