Wan Xinyi commited on
Commit
f8e95f6
1 Parent(s): 377fb85

Add links to README

Browse files
Files changed (1) hide show
  1. README.md +12 -1
README.md CHANGED
@@ -10,4 +10,15 @@ pinned: false
10
  license: apache-2.0
11
  ---
12
 
13
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
 
 
 
 
 
 
 
 
 
 
 
10
  license: apache-2.0
11
  ---
12
 
13
+
14
+ # Zero Bubble Pipeline Parallelism
15
+
16
+ Zero Bubble Pipeline Parallelism is a novel pipeline parallelism algorithm able to reduce the bubble of pipeline parallelism to almost zero while preserving synchronous semantics.
17
+
18
+ Check out our paper at:
19
+ * [Arxiv Version with ZBV](https://arxiv.org/abs/2401.10241)
20
+ * [ICLR Accepted version with ZB1P and ZB2P](https://openreview.net/pdf?id=tuzTN0eIO5)
21
+
22
+ Try out our implementation based on Megatron on [https://github.com/sail-sg/zero-bubble-pipeline-parallelism](https://github.com/sail-sg/zero-bubble-pipeline-parallelism)
23
+
24
+ Experiments shows zero bubble pipeline parallelism can accelerate training up to 30% with a similar memory comsumption. A detailed table of experiments is coming soon.