| # Zero Bubble Pipeline Parallelism | |
| Zero Bubble Pipeline Parallelism is a novel pipeline parallelism algorithm able to reduce the bubble of pipeline parallelism to almost zero while preserving synchronous semantics. | |
| Our paper is coming soon. | |
| Try out our implementation based on Megatron on [https://github.com/sail-sg/zero-bubble-pipeline-parallelism](https://github.com/sail-sg/zero-bubble-pipeline-parallelism) | |
| Experiments shows zero bubble pipeline parallelism can accelerate training up to 30% with a similar memory comsumption. A detailed table of experiments is coming soon. | |
| ## Zero Bubble Schedules | |
| The key of achieving zero bubble is to breaking a backward pass into a B pass and W pass. B on one stage will only depend on the B on its next stage, compared to depending on both B and W of in 1F1B. | |
|  | |
| ### Comparision of Schedules | |
| * 1F1B | |
|  | |
| * ZB1P | |
|  | |
| * ZB2P | |
|  | |
| * ZBV - Each device is assigned to exactly 2 chunks (virtual stages), where white text colors represent the first chunk and black text colors represent the second chunk. The sequence of dependencies among model chunks follows a ”V” shape pattern for both the forward and backward passes. | |
|  | |
| | Comparison assuming T_F=T_B=T_W | 1F1B | ZB1P | ZB2P | ZBV (Recommended) | | |
| | ----------------------------------------------------- | ------- | -------- | ---- | --- | | |
| | Bubble Rate | (p-1)/m | (p-1)/3m | 0 | 0 | | |
| | Activation Memory <br> (Compared to 1F1B) | 1x | 1x | 2x | 1x | | |
| | Pipeline Communication Volume <br> (Compared to 1F1B) | 1x | 1x | 1x | 2x | | |