README.md

metadata

library_name: transformers
license: apache-2.0
language:
  - it
tags:
  - fill-mask
  - masked-lm
  - long-context
  - modernbert
  - italian
pipeline_tag: fill-mask

Italian ModernBERT

Model Description

Italian ModernBERT (DeepMount00/Italian-ModernBERT-base) is a specialized Italian language version of ModernBERT, specifically pre-trained on Italian text corpora. This model is designed exclusively for Italian language tasks.

Key Features

• Architecture: Based on ModernBERT-base (22 layers, 149M parameters)
• Context Length: 8,192 tokens
• Language: Italian-only
• Tokenizer: Custom tokenizer optimized for Italian language
• Training: Pre-trained on Italian text corpus

Technical Details

• Uses Rotary Positional Embeddings (RoPE)
• Implements Local-Global Alternating Attention
• Supports Flash Attention 2 for optimal performance
• No token type IDs required

Limitations

• Optimized only for Italian language processing
• Not suitable for other languages
• May reflect biases present in training data