Article
From DeepSpeed to FSDP and Back Again with Hugging Face Accelerate
•
45
Stas Bekman
stas
AI & ML interests
Toolmaker. Software creator, optimizer and harmonizer.
Makes things work and fly at Contextual.AI
Training LLM/RAG/Generative AI/Machine Learning/Scalability
Recent Activity
updated
a model
23 days ago
stas/ml-engineering-book
updated
a model
26 days ago
stas/ml-engineering-book
posted
an
update
2 months ago
If you remember my work on MAMF - to find the realistic TFLOPS achievable ceiling - the Intel AI team has shared their measurements and they scored ...
an incredible 99.4% TFLOPS efficiency for Gaudi 2!
That's quite amazing! Your ROI on these accelerators will be very high.
The full table is here: https://github.com/stas00/ml-engineering/tree/master/compute/accelerator#maximum-achievable-matmul-flops-comparison-table
As we have seen the competitors get their achievable efficiency worse with each new generation, I'm looking forward to see if Gaudi 3 will keep the high bar!
Thanks to Avi Rubin, Lakshman Chari, Imtiaz Sajwani, Ramy J and Zhiqi Tao for helping to get these numbers to the community.
Organizations
stas's activity
published
an
article
8 months ago
published
an
article
over 1 year ago
Article
Introducing IDEFICS: An Open Reproduction of State-of-the-art Visual Language Model
•
29
published
an
article
over 2 years ago
Article
Incredibly Fast BLOOM Inference with DeepSpeed and Accelerate
•
1
published
an
article
over 2 years ago
Article
The Technology Behind BLOOM Training
•
26
published
an
article
about 4 years ago
Article
Fit More and Train Faster With ZeRO via DeepSpeed and FairScale
•
4
published
an
article
over 4 years ago
Article