Crispin Almodovar

calmodovar

AI & ML interests

NLP, log anomaly detection, cyber intelligence

Recent Activity

reacted to merve's post with 🔥 12 days ago

supercharge your LLM apps with smolagents 🔥 however cool your LLM is, without being agentic it can only go so far enter smolagents: a new agent library by Hugging Face to make the LLM write code, do analysis and automate boring stuff! Here's our blog for you to get started https://huggingface.co/blog/smolagents

reacted to MoritzLaurer's post with 👍 23 days ago

Quite excited by the ModernBERT release! 0.15/0.4B small, 2T modern pre-training data and tokenizer with code, 8k context window, great efficient model for embeddings & classification! This will probably be the basis for many future SOTA encoders! And I can finally stop using DeBERTav3 from 2021 :D Congrats @answerdotai, @LightOnIO and collaborators like @tomaarsen ! Paper and models here 👇https://huggingface.co/collections/answerdotai/modernbert-67627ad707a4acbf33c41deb

upvoted an article 2 months ago

Visually Multilingual: Introducing mcdse-2b

View all activity

Organizations

calmodovar's activity

reacted to merve's post with 🔥 12 days ago

Post

4731

supercharge your LLM apps with smolagents 🔥

however cool your LLM is, without being agentic it can only go so far

enter smolagents: a new agent library by Hugging Face to make the LLM write code, do analysis and automate boring stuff!

Here's our blog for you to get started https://huggingface.co/blog/smolagents

reacted to MoritzLaurer's post with 👍 23 days ago

Post

2597

Quite excited by the ModernBERT release! 0.15/0.4B small, 2T modern pre-training data and tokenizer with code, 8k context window, great efficient model for embeddings & classification!

This will probably be the basis for many future SOTA encoders! And I can finally stop using DeBERTav3 from 2021 :D

Congrats @answerdotai , @LightOnIO and collaborators like @tomaarsen !

Paper and models here 👇https://huggingface.co/collections/answerdotai/modernbert-67627ad707a4acbf33c41deb

3 replies

upvoted an article 2 months ago

Article

Visually Multilingual: Introducing mcdse-2b

•

Oct 27, 2024

• 37

reacted to singhsidhukuldeep's post with 👀 3 months ago

Post

2163

While Google's Transformer might have introduced "Attention is all you need," Microsoft and Tsinghua University are here with the DIFF Transformer, stating, "Sparse-Attention is all you need."

The DIFF Transformer outperforms traditional Transformers in scaling properties, requiring only about 65% of the model size or training tokens to achieve comparable performance.

The secret sauce? A differential attention mechanism that amplifies focus on relevant context while canceling out noise, leading to sparser and more effective attention patterns.

How?
- It uses two separate softmax attention maps and subtracts them.
- It employs a learnable scalar λ for balancing the attention maps.
- It implements GroupNorm for each attention head independently.
- It is compatible with FlashAttention for efficient computation.

What do you get?
- Superior long-context modeling (up to 64K tokens).
- Enhanced key information retrieval.
- Reduced hallucination in question-answering and summarization tasks.
- More robust in-context learning, less affected by prompt order.
- Mitigation of activation outliers, opening doors for efficient quantization.

Extensive experiments show DIFF Transformer's advantages across various tasks and model sizes, from 830M to 13.1B parameters.

This innovative architecture could be a game-changer for the next generation of LLMs. What are your thoughts on DIFF Transformer's potential impact?