Mariusz Kurman's picture

Mariusz Kurman PRO

mkurman

·

AI & ML interests

AI Tech Lead | MD

Recent Activity

liked a dataset 18 days ago

MaziyarPanahi/Llama-Nemotron-Post-Training-Dataset-v1-ShareGPT

liked a model 19 days ago

MaziyarPanahi/DeepSeek-V3-0324-GGUF

reacted to Kseniase's post with 👀 21 days ago

8 types of RoPE As we always use Transformers, it's helpful to understand RoPE—Rotary Position Embedding. Since token order matters, RoPE encodes it by rotating token embeddings based on their position, so the model knows how to interpret which token comes first, second, and so on. Here are 8 types of RoPE that can be implemented in different cases: 1. Original RoPE -> https://huggingface.co/papers/2104.09864 Encodes token positions by rotating token embeddings in the complex plane via a position-based rotation matrix, thereby providing the self-attention mechanism with relative positional info. 2. LongRoPE -> https://huggingface.co/papers/2402.13753 Extends the context window of pre-trained LLMs to 2048k tokens, leveraging non-uniformities in positional interpolation with an efficient search. 3. LongRoPE2 -> https://huggingface.co/papers/2502.20082 Extends the effective context window of pre-trained LLMs to the target! length, rescaling RoPE guided by “needle-driven” perplexity. 4. Multimodal RoPE (MRoPE) -> https://huggingface.co/papers/2502.13923 Decomposes positional embedding into 3 components: temporal, height and width, so that positional features are aligned across modalities: text, images and videos. 5. Directional RoPE (DRoPE) -> https://huggingface.co/papers/2503.15029 Adds an identity scalar, improving how angles are handled without extra complexity. It helps balance accuracy, speed, and memory usage. 6. VideoRoPE -> https://huggingface.co/papers/2502.05173 Adapts RoPE for video, featuring 3D structure, low-frequency temporal allocation, diagonal layout, and adjustable spacing. 7. VRoPE -> https://huggingface.co/papers/2502.11664 An another RoPE for video, which restructures positional indices and balances encoding for uniform spatial focus. 8. XPos (Extrapolatable Position Embedding) -> https://huggingface.co/papers/2212.10 Introduces an exponential decay factor into the rotation matrix, improving stability on long sequences.

View all activity

Organizations

mkurman's activity

New activity in meditsolutions/medit-one-140M-9B-tokens-checkpoint about 1 month ago

Question on meaning of parameter of this model

#2 opened about 1 month ago by

Can't install

#1 opened about 1 month ago by

New activity in mkurman/Llama-3.2-MedIT-SUN-2.5B-BT-GRPO 2 months ago

Issue with Padding

#1 opened 2 months ago by

New activity in mkurman/Qwen2.5-14B-DeepSeek-R1-1M 3 months ago

Merge strategy

#1 opened 3 months ago by

Mergekit config

#2 opened 3 months ago by

New activity in mkurman/llama-3.2-MEDIT-3B-o1 3 months ago

space

#1 opened 3 months ago by

New activity in UCSC-VLAA/o1_medical 4 months ago

a runnable script to download all the dataset to colab or kaggle notebook

#1 opened 7 months ago by

New activity in Datou1111/shou_xin 4 months ago

Add generated example

#3 opened 4 months ago by

New activity in meditsolutions/SmolLM2-MedIT-Upscale-2B 4 months ago

Adding Evaluation Results

#1 opened 4 months ago by

leaderboard-pr-bot

New activity in meditsolutions/Llama-3.2-SUN-1B-Instruct 4 months ago

Adding Evaluation Results

#1 opened 4 months ago by

New activity in meditsolutions/Llama-3.1-MedIT-SUN-8B 5 months ago

Adding Evaluation Results

#1 opened 5 months ago by

leaderboard-pr-bot

New activity in Weyaxi/leaderboard-results-to-modelcard 5 months ago

evaluation

#18 opened 5 months ago by

New activity in mkurman/Llama-3.2-MEDIT-3B-o1 5 months ago

Good one, But 😶‍🌫️

#1 opened 5 months ago by

New activity in meditsolutions/Llama-3.2-SUN-2.5B-chat 6 months ago

Adding Evaluation Results

#5 opened 6 months ago by

Adding Evaluation Results

#4 opened 6 months ago by

Adding Evaluation Results

#3 opened 6 months ago by

leaderboard-pr-bot

Adding Evaluation Results

#2 opened 6 months ago by

Adding Evaluation Results

#1 opened 6 months ago by

leaderboard-pr-bot

New activity in open-llm-leaderboard/open_llm_leaderboard 6 months ago

Sign in button not working - cannot submit model

#963 opened 6 months ago by

Sign in button not working - cannot submit model

#963 opened 6 months ago by