Spaces:

---

nanotron

/

ultrascale-playbook

like 2.07k

Running

In the Activation Memory section, the calculations for $$m_act$$ is the following

$$m_{act}=L\cdot seq\cdot bs\cdot h\cdot (34+\frac{5\cdot n_{heads}\cdot seq}{h})m\_\{act\}\; =\; L\; \backslash cdot\; seq\; \backslash cdot\; bs\; \backslash cdot\; h\; \backslash cdot\; \backslash left(34\; +\; \backslash frac\{5\; \backslash cdot\; n\_\{heads\}\; \backslash cdot\; seq\}\{h\}\backslash right)$$

And you guys state that this scales linearly with $$seq_len$$ and $$bs$$. Though, this actually scales quadratically with $$seq_len$$:

$$m_{act}=L\cdot seq\cdot bs\cdot h\cdot 34+L\cdot seq\cdot bs\cdot h\cdot \frac{5\cdot n_{heads}\cdot seq}{h}{m}_{act}=34\cdot L\cdot seq\cdot bs\cdot h+5\cdot L\cdot seq\cdot bs\cdot n_{heads}\cdot seq{m}_{act}=34\cdot L\cdot seq\cdot bs\cdot h+5\cdot L\cdot bs\cdot n_{heads}\cdot se{q}^{2}m\_\{act\}\; =\; L\; \backslash cdot\; seq\; \backslash cdot\; bs\; \backslash cdot\; h\; \backslash cdot\; 34\; +\; L\; \backslash cdot\; seq\; \backslash cdot\; bs\; \backslash cdot\; h\; \backslash cdot\; \backslash frac\{5\; \backslash cdot\; n\_\{heads\}\; \backslash cdot\; seq\}\{h\}\; \backslash \backslash \; m\_\{act\}\; =\; 34\; \backslash cdot\; L\; \backslash cdot\; seq\; \backslash cdot\; bs\; \backslash cdot\; h\; +\; 5\; \backslash cdot\; L\; \backslash cdot\; seq\; \backslash cdot\; bs\; \backslash cdot\; n\_\{heads\}\; \backslash cdot\; seq\; \backslash \backslash \; m\_\{act\}\; =\; 34\; \backslash cdot\; L\; \backslash cdot\; seq\; \backslash cdot\; bs\; \backslash cdot\; h\; +\; 5\; \backslash cdot\; L\; \backslash cdot\; bs\; \backslash cdot\; n\_\{heads\}\; \backslash cdot\; seq^2$$

• The first term, $$34 \cdot L \cdot seq \cdot bs \cdot h$$, scales linearly with $$seq$$.
• The second term, $$5 \cdot L \cdot bs \cdot n_{heads} \cdot seq^2$$, scales quadratically with $$seq$$