SmolVLM: Redefining small and efficient multimodal models
Paper
ā¢
2504.05299
ā¢
Published
ā¢
164
Mohammed Hamdy
mmhamdy
AI & ML interests
TechBio | AI4Sci | NLP | Reinforcement Learning
Recent Activity
upvoted
a
paper
13 days ago
SmolVLM: Redefining small and efficient multimodal models
posted
an
update
22 days ago
What inspired the Transformer architecture in the "Attention Is All You Need" paper? And how were various ideas combined to create this groundbreaking model?
In this lengthy article, I explore the story and the origins of some of the ideas introduced in the paper. We'll explore everything from the fundamental attention mechanism that lies at its heart to the surprisingly simple explanation for its name, Transformer.
š” Examples of ideas explored in the article:
ā
What was the inspiration for the attention mechanism?
ā
How did we go from attention to self-attention?
ā
Did the team have any other names in mind for the model?
and more...
I aim to tell the story of Transformers as I would have wanted to read it, and hopefully, one that appeals to others interested in the details of this fascinating idea. This narrative draws from video interviews, lectures, articles, tweets/Xs, and some digging into the literature. I have done my best to be accurate, but errors are possible. If you find inaccuracies or have any additions, please do reach out, and I will gladly make the necessary updates.
Read the article: https://huggingface.co/blog/mmhamdy/pandemonium-the-transformers-story
published
an
article
22 days ago
Pandemonium: The Transformers Story
Organizations
mmhamdy's activity
upvoted
a
paper
13 days ago
posted
an
update
22 days ago
Post
1575
What inspired the Transformer architecture in the "Attention Is All You Need" paper? And how were various ideas combined to create this groundbreaking model?
In this lengthy article, I explore the story and the origins of some of the ideas introduced in the paper. We'll explore everything from the fundamental attention mechanism that lies at its heart to the surprisingly simple explanation for its name, Transformer.
š” Examples of ideas explored in the article:
ā What was the inspiration for the attention mechanism?
ā How did we go from attention to self-attention?
ā Did the team have any other names in mind for the model?
and more...
I aim to tell the story of Transformers as I would have wanted to read it, and hopefully, one that appeals to others interested in the details of this fascinating idea. This narrative draws from video interviews, lectures, articles, tweets/Xs, and some digging into the literature. I have done my best to be accurate, but errors are possible. If you find inaccuracies or have any additions, please do reach out, and I will gladly make the necessary updates.
Read the article: https://huggingface.co/blog/mmhamdy/pandemonium-the-transformers-story
In this lengthy article, I explore the story and the origins of some of the ideas introduced in the paper. We'll explore everything from the fundamental attention mechanism that lies at its heart to the surprisingly simple explanation for its name, Transformer.
š” Examples of ideas explored in the article:
ā What was the inspiration for the attention mechanism?
ā How did we go from attention to self-attention?
ā Did the team have any other names in mind for the model?
and more...
I aim to tell the story of Transformers as I would have wanted to read it, and hopefully, one that appeals to others interested in the details of this fascinating idea. This narrative draws from video interviews, lectures, articles, tweets/Xs, and some digging into the literature. I have done my best to be accurate, but errors are possible. If you find inaccuracies or have any additions, please do reach out, and I will gladly make the necessary updates.
Read the article: https://huggingface.co/blog/mmhamdy/pandemonium-the-transformers-story
published
an
article
22 days ago
Article
Pandemonium: The Transformers Story
By
ā¢
ā¢
6
published
an
article
26 days ago
Article
Osirian AI: A Call For The Resurrection And Reuse Of Deep Learning Models.
By
ā¢
liked
a
model
about 1 month ago
liked
a
Space
about 1 month ago
upvoted
a
paper
about 1 month ago
commented
a
paper
about 2 months ago
liked
a
model
about 2 months ago
upvoted
an
article
about 2 months ago
Article
A Deepdive into Aya Vision: Advancing the Frontier of Multilingual Multimodality
ā¢
73
upvoted
2
collections
about 2 months ago
upvoted
a
paper
about 2 months ago
liked
a
model
about 2 months ago
upvoted
a
collection
about 2 months ago
posted
an
update
about 2 months ago
Post
2754
š We're excited to introduce MemoryCode, a novel synthetic dataset designed to rigorously evaluate LLMs' ability to track and execute coding instructions across multiple sessions. MemoryCode simulates realistic workplace scenarios where a mentee (the LLM) receives coding instructions from a mentor amidst a stream of both relevant and irrelevant information.
š” But what makes MemoryCode unique?! The combination of the following:
ā Multi-Session Dialogue Histories: MemoryCode consists of chronological sequences of dialogues between a mentor and a mentee, mirroring real-world interactions between coworkers.
ā Interspersed Irrelevant Information: Critical instructions are deliberately interspersed with unrelated content, replicating the information overload common in office environments.
ā Instruction Updates: Coding rules and conventions can be updated multiple times throughout the dialogue history, requiring LLMs to track and apply the most recent information.
ā Prospective Memory: Unlike previous datasets that cue information retrieval, MemoryCode requires LLMs to spontaneously recall and apply relevant instructions without explicit prompts.
ā Practical Task Execution: LLMs are evaluated on their ability to use the retrieved information to perform practical coding tasks, bridging the gap between information recall and real-world application.
š Our Findings
1ļøā£ While even small models can handle isolated coding instructions, the performance of top-tier models like GPT-4o dramatically deteriorates when instructions are spread across multiple sessions.
2ļøā£ This performance drop isn't simply due to the length of the context. Our analysis indicates that LLMs struggle to reason compositionally over sequences of instructions and updates. They have difficulty keeping track of which instructions are current and how to apply them.
š Paper: From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions (2502.13791)
š¦ Code: https://github.com/for-ai/MemoryCode
š” But what makes MemoryCode unique?! The combination of the following:
ā Multi-Session Dialogue Histories: MemoryCode consists of chronological sequences of dialogues between a mentor and a mentee, mirroring real-world interactions between coworkers.
ā Interspersed Irrelevant Information: Critical instructions are deliberately interspersed with unrelated content, replicating the information overload common in office environments.
ā Instruction Updates: Coding rules and conventions can be updated multiple times throughout the dialogue history, requiring LLMs to track and apply the most recent information.
ā Prospective Memory: Unlike previous datasets that cue information retrieval, MemoryCode requires LLMs to spontaneously recall and apply relevant instructions without explicit prompts.
ā Practical Task Execution: LLMs are evaluated on their ability to use the retrieved information to perform practical coding tasks, bridging the gap between information recall and real-world application.
š Our Findings
1ļøā£ While even small models can handle isolated coding instructions, the performance of top-tier models like GPT-4o dramatically deteriorates when instructions are spread across multiple sessions.
2ļøā£ This performance drop isn't simply due to the length of the context. Our analysis indicates that LLMs struggle to reason compositionally over sequences of instructions and updates. They have difficulty keeping track of which instructions are current and how to apply them.
š Paper: From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions (2502.13791)
š¦ Code: https://github.com/for-ai/MemoryCode
authored
2
papers
about 2 months ago
liked
a
Space
about 2 months ago