Wan Xinyi
commited on
Commit
•
f8e95f6
1
Parent(s):
377fb85
Add links to README
Browse files
README.md
CHANGED
@@ -10,4 +10,15 @@ pinned: false
|
|
10 |
license: apache-2.0
|
11 |
---
|
12 |
|
13 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
10 |
license: apache-2.0
|
11 |
---
|
12 |
|
13 |
+
|
14 |
+
# Zero Bubble Pipeline Parallelism
|
15 |
+
|
16 |
+
Zero Bubble Pipeline Parallelism is a novel pipeline parallelism algorithm able to reduce the bubble of pipeline parallelism to almost zero while preserving synchronous semantics.
|
17 |
+
|
18 |
+
Check out our paper at:
|
19 |
+
* [Arxiv Version with ZBV](https://arxiv.org/abs/2401.10241)
|
20 |
+
* [ICLR Accepted version with ZB1P and ZB2P](https://openreview.net/pdf?id=tuzTN0eIO5)
|
21 |
+
|
22 |
+
Try out our implementation based on Megatron on [https://github.com/sail-sg/zero-bubble-pipeline-parallelism](https://github.com/sail-sg/zero-bubble-pipeline-parallelism)
|
23 |
+
|
24 |
+
Experiments shows zero bubble pipeline parallelism can accelerate training up to 30% with a similar memory comsumption. A detailed table of experiments is coming soon.
|