pipeline-parallelism-with-controllable-memory

Running

nvmdava commited on Jun 19

Commit

964a6f1

•

1 Parent(s): 7a9d367

Update description1.md

Files changed (1) hide show

description1.md CHANGED Viewed

@@ -6,9 +6,11 @@ From our findings, we need approximately 1/3 memory under ideal conditions (F, B
 Check out our paper at [Arxiv](https://arxiv.org/abs/2405.15362).
 | Comparison assuming T_F=T_B=T_W                       | 1F1B    | V-Min  | V-Half     | V-ZB |
 | ----------------------------------------------------- | ------- |------- | ---------- | ---- |
-| Bubble Rate                                           | (p-1)/m | ????   | ???        |   0  |
-| Activation Memory <br> (Compared to 1F1B)             |    p   |  (p+4)//3 | (p+2)//2 |   p  |
 Bubble Rate here is calculated as (1 - longest stage time/(F+B+W)/m).

 Check out our paper at [Arxiv](https://arxiv.org/abs/2405.15362).
 | Comparison assuming T_F=T_B=T_W                       | 1F1B    | V-Min  | V-Half     | V-ZB |
 | ----------------------------------------------------- | ------- |------- | ---------- | ---- |
+| Bubble Rate                                           | (p-1)/m |  ~ 2p/3m     |  ~ p/ 2m    |   0  |
+| Activation Memory <br> (Compared to 1F1B)             |    p   |  (p+4)/3 | (p+2)/2 |   p  |
 Bubble Rate here is calculated as (1 - longest stage time/(F+B+W)/m).