diff --git "a/logs/main_log.txt" "b/logs/main_log.txt" --- "a/logs/main_log.txt" +++ "b/logs/main_log.txt" @@ -43005,3 +43005,5828 @@ saving checkpoint at iteration 34 to /gpfsscratch/rech/six/commun/checkpoin successfully saved checkpoint at iteration 34 to /gpfsscratch/rech/six/commun/checkpoints/tr8b-104B/checkpoints time (ms) | save-checkpoint: 30665.91 [exiting program after 55.0033370534579 minutes] datetime: 2021-10-21 22:22:26 +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +ninja .................. [OKAY] +-------------------------------------------------- +op name ................ installed .. compatible +-------------------------------------------------- +cpu_adam ............... [NO] ....... [OKAY] +fused_adam ............. [NO] ....... [OKAY] +fused_lamb ............. [NO] ....... [OKAY] +sparse_attn ............ [NO] ....... [OKAY] +transformer ............ [NO] ....... [OKAY] +stochastic_transformer . [NO] ....... [OKAY] +ninja .................. [OKAY] +-------------------------------------------------- +op name ................ installed .. compatible +-------------------------------------------------- +cpu_adam ............... [NO] ....... [OKAY] +fused_adam ............. [NO] ....... [OKAY] +fused_lamb ............. [NO] ....... [OKAY] +sparse_attn ............ [NO] ....... [OKAY] +transformer ............ [NO] ....... [OKAY] +stochastic_transformer . [NO] ....... [OKAY] +ninja .................. [OKAY] +-------------------------------------------------- +op name ................ installed .. compatible +-------------------------------------------------- +cpu_adam ............... [NO] ....... [OKAY] +fused_adam ............. [NO] ....... [OKAY] +fused_lamb ............. [NO] ....... [OKAY] +sparse_attn ............ [NO] ....... [OKAY] +transformer ............ [NO] ....... [OKAY] +stochastic_transformer . [NO] ....... [OKAY] +ninja .................. [OKAY] +-------------------------------------------------- +op name ................ installed .. compatible +-------------------------------------------------- +cpu_adam ............... [NO] ....... [OKAY] +fused_adam ............. [NO] ....... [OKAY] +fused_lamb ............. [NO] ....... [OKAY] +sparse_attn ............ [NO] ....... [OKAY] +transformer ............ [NO] ....... [OKAY] +stochastic_transformer . [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +--------------------------------------------------JIT compiled ops requires ninja + +--------------------------------------------------DeepSpeed C++/CUDA extension op report + +--------------------------------------------------DeepSpeed C++/CUDA extension op report +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +---------------------------------------------------------------------------------------------------- + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.JIT compiled ops requires ninja + +-------------------------------------------------- +JIT compiled ops requires ninja +------------------------------------------------------------------------------------------------------------------------------------------------------ + + +DeepSpeed C++/CUDA extension op report--------------------------------------------------DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op report + + + +----------------------------------------------------------------------------------------------------DeepSpeed C++/CUDA extension op report-------------------------------------------------- + + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- + +---------------------------------------------------------------------------------------------------- + +-------------------------------------------------- +JIT compiled ops requires ninja +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.JIT compiled ops requires ninjaJIT compiled ops requires ninja + + + +-------------------------------------------------- +JIT compiled ops requires ninja +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja-------------------------------------------------- + +DeepSpeed C++/CUDA extension op report +------------------------------------------------------------------------------------------------------------------------------------------------------ +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + + +DeepSpeed C++/CUDA extension op report--------------------------------------------------DeepSpeed C++/CUDA extension op report + + +JIT compiled ops requires ninja-------------------------------------------------- +-------------------------------------------------- + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +---------------------------------------------------------------------------------------------------- + +JIT compiled ops requires ninjaJIT compiled ops requires ninja + +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +ninja .................. [OKAY] +-------------------------------------------------- +ninja .................. [OKAY] +-------------------------------------------------- +op name ................ installed .. compatible +-------------------------------------------------- +cpu_adam ............... [NO] ....... [OKAY]ninja +ninja .................. [OKAY] +-------------------------------------------------- +op name ................ installed .. compatible +-------------------------------------------------- +cpu_adam ............... [NO] ....... [OKAY] + .................. [OKAY] +-------------------------------------------------- +fused_adam ............. [NO] ....... [OKAY] +op name ................fused_adam installed ............... compatible[NO] +op name ................ installed ..ninja ninja compatible +fused_lamb ............. ninja[NO]ninja .......ninja .................. .................. [OKAY] + .......-------------------------------------------------- +[OKAY]ninja +.................. ninja.................. [OKAY] +.................. [OKAY][OKAY][OKAY] + + +------------------------------------------------------------------------------------------------------------------------------------------------------ + + + .................. ninja[OKAY]fused_lambcpu_adam +............. ..................--------------------------------------------------............... +-------------------------------------------------- -------------------------------------------------- +[OKAY].................. + + [OKAY]op name-------------------------------------------------- + +................-------------------------------------------------- cpu_adamop name installed +op nameop nameop name ................................................ installedinstalledinstalledsparse_attn .... .............. compatible compatible +[NO] + [NO][NO][OKAY]op name + .............................. --------------------------------------------------[OKAY]installed + + [OKAY]..op name + ............... op name................ .. ................ compatible[NO]installed +compatible ---------------------------------------------------------------------------------------------------- +....... + + --------------------------------------------------[OKAY] + + compatible................ + --------------------------------------------------installed +installed -------------------------------------------------- .. + fused_adam.. .............compatible +sparse_attn[NO]--------------------------------------------------cpu_adam ............ +.. compatible compatible +....... +transformer ............cpu_adam cpu_adam [NO]cpu_adam............... ......................[NO] ...............[NO] [OKAY].......[NO]....... + [OKAY][OKAY]....... + +...................... [OKAY][NO] +----------------------------------------------------------------------------------------------------cpu_adam + + [OKAY]............... + stochastic_transformer[OKAY] +[NO] ..............cpu_adamfused_lamb ............... [OKAY][OKAY].............[NO] + + [NO]cpu_adam cpu_adam...................... ...............[OKAY][NO] +. [NO] .......fused_adam [OKAY]fused_adam............. + fused_adam.............[NO] .............[NO]....... [NO].......[OKAY] + [NO]....... transformer....... [OKAY]............[OKAY] + +[NO]....... ....... [OKAY][OKAY] + +[OKAY]....... + [OKAY] +[NO] .......fused_adam .............[OKAY] +[NO] fused_adam....... stochastic_transformer[OKAY]............. +fused_adamfused_adam ............. .............[NO] fused_adam....... [OKAY][NO]fused_adam +fused_lamb fused_lamb............. fused_lamb ............. [NO]............. [NO].......[NO] .......[OKAY]....... + [OKAY][OKAY] + +sparse_attn . [NO] ............ [NO] fused_lamb.......[NO] ....................[OKAY]....... + ............. .............fused_lamb[NO] .......[NO].................... [OKAY][NO]....... +sparse_attn ............sparse_attn [NO]............sparse_attn ...................[NO] [OKAY][NO]....... + [OKAY][NO][OKAY] + +fused_lamb....... transformer[OKAY] + [OKAY] +.......[OKAY] +fused_lamb[OKAY] + .......[OKAY]transformer + [OKAY]............ +......................... [NO][NO] .............. [OKAY][OKAY] + +............. fused_lambfused_lamb[NO] ................................. [OKAY][NO]sparse_attn .......[NO] + transformer[NO] transformer ............ ....... ............ [NO] [OKAY] [NO] +stochastic_transformersparse_attn ............. [NO][NO] .............. [OKAY][OKAY]sparse_attn + + ............[OKAY] +[NO]....... .......[OKAY] +....... .......[OKAY] +stochastic_transformer[OKAY] + ............transformer [NO]............ [NO]....... .......[OKAY] [OKAY] + +[OKAY]sparse_attn +.stochastic_transformer [NO]stochastic_transformer ........ [NO].[OKAY] + .......[NO] [OKAY]....... + [OKAY] +transformer stochastic_transformer............ [NO]. .......[NO] .......[OKAY] +[OKAY] + ............ sparse_attn[NO]transformer ............................... [OKAY][NO][NO] +sparse_attn .............. [OKAY]transformer[OKAY] +stochastic_transformer . [NO] ....... [OKAY] + +........................transformer [NO] stochastic_transformer ............ .......[NO] [NO]........[OKAY] +[OKAY] ....... +[NO] [OKAY]transformer.......stochastic_transformer + ............stochastic_transformer[OKAY]. +[NO] .[NO]....... [NO].......[OKAY] +....... [OKAY][OKAY] + +stochastic_transformer . [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- +-------------------------------------------------- + +JIT compiled ops requires ninjaDeepSpeed C++/CUDA extension op report + +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +ninja .................. [OKAY] +-------------------------------------------------- +op name ................ installed .. compatible +-------------------------------------------------- +cpu_adam ............... [NO] ....... [OKAY] +fused_adam ............. [NO] ....... [OKAY] +fused_lamb ............. [NO] ....... [OKAY] +sparse_attn ............ [NO] ....... [OKAY] +transformer ............ [NO] ....... [OKAY] +stochastic_transformer . [NO] ....... [OKAY] +------------------------------------------------------------------------------------------------------------------------------------------------------ + +DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op report + + +DeepSpeed C++/CUDA extension op report---------------------------------------------------------------------------------------------------- +-------------------------------------------------- + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + + + +DeepSpeed C++/CUDA extension op report--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- + + + +--------------------------------------------------JIT compiled ops requires ninjaJIT compiled ops requires ninja-------------------------------------------------- + + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.JIT compiled ops requires ninja + +-------------------------------------------------- +JIT compiled ops requires ninja +ninjaninja .................................... [OKAY][OKAY] + +---------------------------------------------------------------------------------------------------- + +op nameop name ................ ................installed ..installed compatible.. + --------------------------------------------------compatible + +-------------------------------------------------- +cpu_adam ............... [NO]cpu_adam ....... ...............[OKAY] +[NO] ....... [OKAY] +fused_adamfused_adam .......................... [NO] [NO]....... .......[OKAY] +[OKAY] +fused_lambfused_lamb .......................... [NO] [NO]....... .......[OKAY] + [OKAY] +sparse_attn sparse_attn............ [NO]............ .......[NO] [OKAY]....... + [OKAY]transformer + ............ transformer[NO] ................... [OKAY][NO] + ....... [OKAY] +stochastic_transformer . stochastic_transformer[NO] ....... .[OKAY] + [NO] ....... [OKAY] +ninja .................. [OKAY] +-------------------------------------------------- +op name ................ installed .. compatible +-------------------------------------------------- +ninja .................. [OKAY] +cpu_adam ...............-------------------------------------------------- +[NO] op name.......ninja ................[OKAY] +..................installed [OKAY].. + compatible +---------------------------------------------------------------------------------------------------- + +op name ................ninja fused_adaminstalled .................................cpu_adam ...............[OKAY]compatible [NO] + + [NO].......-------------------------------------------------- -------------------------------------------------- ....... + +[OKAY] +op name[OKAY] +................ installedfused_lambcpu_adam .............................. compatible[NO] +[NO] --------------------------------------------------fused_adam....... ....... + ............. [OKAY] [OKAY] +[NO] + ....... [OKAY] +cpu_adam fused_lamb............... [NO]fused_adam............. .............sparse_attn[NO]....... ............[NO] .......[NO][OKAY] +.......[OKAY] ....... +[OKAY] +[OKAY] +fused_lambtransformer ......................... [NO]fused_adam[NO] ...........................sparse_attn [OKAY] [OKAY][NO]............ + + .......[NO] stochastic_transformer [OKAY] ....... + .[OKAY] +[NO]fused_lamb sparse_attntransformer ....... ......................... ............ [OKAY] [NO][NO] +[NO] ..................... [OKAY] +[OKAY][OKAY] + +stochastic_transformertransformer ............ .[NO] .......[NO] [OKAY]....... + [OKAY] +sparse_attn stochastic_transformer............ [NO]. ....... [NO][OKAY] +....... transformer[OKAY] +............ [NO] ....... [OKAY] +stochastic_transformer . [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +ninja .................. [OKAY] +-------------------------------------------------- +op name ................ installed .. compatible +-------------------------------------------------- +cpu_adam ............... [NO] ....... [OKAY] +fused_adam ............. [NO] ....... [OKAY] +fused_lamb ............. [NO] ....... [OKAY] +sparse_attn ............ [NO] ....... [OKAY] +transformer ............ [NO] ....... [OKAY] +stochastic_transformer . [NO] ....... [OKAY] +ninja .................. [OKAY] +-------------------------------------------------- +op name ................ installed .. compatible +-------------------------------------------------- +cpu_adam ............... [NO] ....... [OKAY] +fused_adam ............. [NO] ....... [OKAY] +fused_lamb ............. [NO] ....... [OKAY] +sparse_attn ............ [NO] ....... [OKAY] +transformer ............ [NO] ....... [OKAY] +stochastic_transformer . [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +ninja .................. [OKAY] +-------------------------------------------------- +op name ................ installed .. compatible +-------------------------------------------------- +cpu_adam ............... [NO] ....... [OKAY] +fused_adam ............. [NO] ....... [OKAY] +fused_lamb ............. [NO] ....... [OKAY] +sparse_attn ............ [NO] ....... [OKAY] +transformer ............ [NO] ....... [OKAY] +stochastic_transformer . [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +ninja .................. [OKAY] +-------------------------------------------------- +op name ................ installed .. compatible +-------------------------------------------------- +cpu_adam ............... [NO] ....... [OKAY] +fused_adam ............. [NO] ....... [OKAY] +fused_lamb ............. [NO] ....... [OKAY] +sparse_attn ............ [NO] ....... [OKAY] +transformer ............ [NO] ....... [OKAY] +stochastic_transformer . [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +ninja .................. [OKAY] +-------------------------------------------------- +op name ................ installed .. compatible +-------------------------------------------------- +cpu_adam ............... [NO] ....... [OKAY] +fused_adam ............. [NO] ....... [OKAY] +fused_lamb ............. [NO] ....... [OKAY] +sparse_attn ............ [NO] ....... [OKAY] +transformer ............ [NO] ....... [OKAY] +stochastic_transformer . [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +ninja .................. [OKAY] +-------------------------------------------------- +op name ................ installed .. compatible +-------------------------------------------------- +cpu_adam ............... [NO] ....... [OKAY] +fused_adam ............. [NO] ....... [OKAY] +fused_lamb ............. [NO] ....... [OKAY] +sparse_attn ............ [NO] ....... [OKAY] +transformer ............ [NO] ....... [OKAY] +stochastic_transformer . [NO] ....... [OKAY] +ninja .................. [OKAY] +-------------------------------------------------- +op name ................ installed .. compatible +-------------------------------------------------- +cpu_adam ............... [NO] ....... [OKAY] +fused_adam ............. [NO] ....... [OKAY] +fused_lamb ............. [NO] ....... [OKAY] +sparse_attn ............ [NO] ....... [OKAY] +transformer ............ [NO] ....... [OKAY] +stochastic_transformer . [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +ninja .................. [OKAY] +-------------------------------------------------- +op name ................ installed .. compatible +-------------------------------------------------- +cpu_adam ............... [NO] ....... [OKAY] +fused_adam ............. [NO] ....... [OKAY] +fused_lamb ............. [NO] ....... [OKAY] +sparse_attn ............ [NO] ....... [OKAY] +transformer ............ [NO] ....... [OKAY] +stochastic_transformer . [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +ninja .................. [OKAY] +-------------------------------------------------- +op name ................ installed .. compatible +-------------------------------------------------- +cpu_adam ............... [NO] ....... [OKAY] +fused_adam ............. [NO] ....... [OKAY] +fused_lamb ............. [NO] ....... [OKAY] +sparse_attn ............ [NO] ....... [OKAY] +transformer ............ [NO] ....... [OKAY] +stochastic_transformer . [NO] ....... [OKAY] + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +............... [NO] ....... [NO] +async_io ............... transformer_inference[NO] ......... [NO][NO] +....... [OKAY] +utils .................. [NO] ....... transformer_inference[OKAY] +.. [NO] quantizer....... ..............[OKAY] +[NO] ....... utils[OKAY] + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + +.................. [NO] .......-------------------------------------------------- +[OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + [WARNING]  async_io: please install the libaio-devel package with yum +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + ............... [NO] ....... [NO] +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utilstransformer_inference .................... [NO][NO] ....... .......[OKAY] +[OKAY] +quantizer .............. [NO]utils ......................... [OKAY][NO] + ....... [OKAY] +-------------------------------------------------- +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] [WARNING]  async_io: please install the libaio-devel package with yum + +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ...............DeepSpeed general environment info: [NO] ....... + [NO] +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch']transformer_inference + .. [NO]torch version .................... 1.8.1....... + [OKAY]torch cuda version + ............... 11.1 +utils nvcc version.................. .....................[NO] .......11.2 +[OKAY] +deepspeed install path ...........quantizer .............. [NO]['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +....... deepspeed info[OKAY] + ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w.-------------------------------------------------- +...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info:DeepSpeed general environment info: + +torch install pathtorch install path .............................. ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch']['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] + +torch versiontorch version ........................................ 1.8.11.8.1 + +torch cuda versiontorch cuda version .............................. 11.111.1 + +nvcc versionnvcc version .......................................... 11.211.2 + +deepspeed install pathdeepspeed install path ...................... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed']['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] + +deepspeed infodeepspeed info ...................................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix0.5.5+57dee5a, 57dee5a, pp_deadlock_fix + +deepspeed wheel compiled w.deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info:DeepSpeed general environment info: + +torch install path torch install path............... ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version ....................torch version 1.8.1 +.................... 1.8.1torch cuda version + ...............torch cuda version 11.1............... + nvcc version11.1 +..................... nvcc version11.2 +..................... deepspeed install path11.2 +........... deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +................... deepspeed info0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +................... deepspeed wheel compiled w.0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +...... deepspeed wheel compiled w.torch 1.8, cuda 11.1 +...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path DeepSpeed general environment info:........... +['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info torch install path................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +...............deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. [NO] + +transformer_inferenceasync_io ................. [NO][NO] .............. [OKAY][NO] + +utils .................. [NO] ....... [OKAY] +transformer_inferencequantizer ................ [NO][NO] .............. [OKAY][OKAY] + +-------------------------------------------------- +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info:DeepSpeed general environment info: + +torch install path ...............torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version ....................torch version 1.8.1.................... + 1.8.1torch cuda version + ............... torch cuda version11.1 +...............nvcc version 11.1..................... + nvcc version11.2 +.....................deepspeed install path 11.2........... +deepspeed install path ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed']........... +deepspeed info ...................['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed info deepspeed wheel compiled w.................... ......0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +torch 1.8, cuda 11.1 +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... DeepSpeed general environment info: +['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch']torch install path + ............... torch version .................... 1.8.1 +['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch']torch cuda version + ............... 11.1torch version + nvcc version.................... .....................1.8.1 +11.2 +torch cuda versiondeepspeed install path .......................... 11.1 +nvcc version['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +.....................deepspeed info 11.2................... + deepspeed install path0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +...........deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] + +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main ******** Git info for Megatron: git_hash=829cefd git_branch=main **** + +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main ******** Git info for Megatron: git_hash=829cefd git_branch=main **** + +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main ******** Git info for Megatron: git_hash=829cefd git_branch=main **** + +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +------------------------------------------------------------------------------------------------------------------------------------------------------ + + +DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op report + + +------------------------------------------------------------------------------------------------------------------------------------------------------ + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + + +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + + +JIT compiled ops requires ninjaJIT compiled ops requires ninjaJIT compiled ops requires ninja + + + +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +ninjaninjaninjaninja ...................................................... .................. [OKAY] [OKAY][OKAY] +[OKAY] + + +-------------------------------------------------- +------------------------------------------------------------------------------------------------------------------------------------------------------ + + +op nameop name op nameop name ................ ................ ................................installed installedinstalledinstalled.. .. ..compatible .. +compatible compatible +compatible-------------------------------------------------- + +-------------------------------------------------- +-------------------------------------------------- +-------------------------------------------------- + +cpu_adam cpu_adamcpu_adam............... ...............cpu_adam...............[NO] [NO]...............[NO]....... .......[NO].......[OKAY] + .......[OKAY][OKAY] + +[OKAY] +fused_adam .............fused_adamfused_adam [NO]fused_adam ............. .................... ............. [NO] [NO][OKAY] [NO] +....... ....... ....... [OKAY] fused_lamb[OKAY][OKAY] + + +............. [NO] fused_lamb.......fused_lamb fused_lamb ............. [OKAY]............. ............. + [NO][NO][NO] ..................... [OKAY][OKAY][OKAY] + + +sparse_attn ............ [NO] .......sparse_attn [OKAY]sparse_attnsparse_attn............ + ........................[NO] transformer[NO] [NO] ................... ....... .......[NO] [OKAY] [OKAY] [OKAY] + +....... + [OKAY]transformer + transformer............transformer stochastic_transformer........................[NO] [NO][NO] ........ ....... .......[OKAY][NO] +[OKAY] [OKAY] +....... + [OKAY] +stochastic_transformerstochastic_transformerstochastic_transformer ... [NO][NO][NO] ..................... [OKAY][OKAY][OKAY] + + +------------------------------------------------------------------------------------------------------------------------------------------------------ + + +DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op report +-------------------------------------------------- + +-------------------------------------------------- +-------------------------------------------------- +--------------------------------------------------DeepSpeed C++/CUDA extension op report + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.----------------------------------------------------------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + + + +--------------------------------------------------JIT compiled ops requires ninja--------------------------------------------------JIT compiled ops requires ninja + + + +JIT compiled ops requires ninjaJIT compiled ops requires ninja + +---------------------------------------------------------------------------------------------------- + +DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op report + +---------------------------------------------------------------------------------------------------- + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +---------------------------------------------------------------------------------------------------- + +JIT compiled ops requires ninja +JIT compiled ops requires ninja +DeepSpeed C++/CUDA extension op report + +---------------------------------------------------------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +DeepSpeed C++/CUDA extension op report-------------------------------------------------- + +--------------------------------------------------JIT compiled ops requires ninja + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +---------------------------------------------------------------------------------------------------- + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.DeepSpeed C++/CUDA extension op report + +-------------------------------------------------- +----------------------------------------------------------------------------------------------------JIT compiled ops requires ninja + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.--------------------------------------------------DeepSpeed C++/CUDA extension op report + + +---------------------------------------------------------------------------------------------------- +DeepSpeed C++/CUDA extension op report + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.JIT compiled ops requires ninja +-------------------------------------------------- + +--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +JIT compiled ops requires ninja-------------------------------------------------- + +JIT compiled ops requires ninja +---------------------------------------------------------------------------------------------------- + +DeepSpeed C++/CUDA extension op report--------------------------------------------------DeepSpeed C++/CUDA extension op report + + +------------------------------------------------------------------------------------------------------------------------------------------------------DeepSpeed C++/CUDA extension op report + + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.DeepSpeed C++/CUDA extension op report-------------------------------------------------- + + + +---------------------------------------------------------------------------------------------------- +--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +JIT compiled ops requires ninja + +JIT compiled ops requires ninja +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- + + +JIT compiled ops requires ninja-------------------------------------------------- + +JIT compiled ops requires ninja +---------------------------------------------------------------------------------------------------- + +DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op report + +---------------------------------------------------------------------------------------------------- + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +-------------------------------------------------- +-------------------------------------------------- +-------------------------------------------------- +JIT compiled ops requires ninjaDeepSpeed C++/CUDA extension op report +JIT compiled ops requires ninja + + +--------------------------------------------------DeepSpeed C++/CUDA extension op report + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- + +ninjaninjaninjaninja ........................................................................ [OKAY][OKAY][OKAY][OKAY] + + + +--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +JIT compiled ops requires ninja-------------------------------------------------- + +JIT compiled ops requires ninja +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + + + +op name op nameop nameop name................ ................................installed................ installed installedinstalled .... ..compatiblecompatible.. + +--------------------------------------------------compatiblecompatible-------------------------------------------------- + + + +---------------------------------------------------------------------------------------------------- + +cpu_adam cpu_adamcpu_adam...............cpu_adam ............... ...............[NO][NO]............... [NO].......[NO] ....... ....... [OKAY]....... [OKAY] [OKAY] + +[OKAY] + +fused_adamfused_adamfused_adam fused_adam .......................... ............. ............. [NO][NO] [NO] [NO] ....... .............. ....... [OKAY][OKAY] [OKAY] + + +[OKAY] +fused_lambfused_lamb fused_lamb .............fused_lamb ............. ............. [NO] [NO]............. [NO] ....... ....... [NO] [OKAY]....... + [OKAY] ....... +[OKAY] +[OKAY] +sparse_attn ............ [NO] sparse_attnsparse_attn.......sparse_attn ............ ........................ [OKAY] [NO] [NO] +[NO] .......transformer....... ....... ............[OKAY][OKAY][OKAY] + +[NO] + .......transformer transformertransformer[OKAY]............ +[NO]........................ .......[NO][NO]stochastic_transformer [OKAY]....... +........ [OKAY][NO][OKAY]stochastic_transformer + +....... stochastic_transformer [OKAY]stochastic_transformer. + . [NO] [NO]........ .......[OKAY][NO] + [OKAY]....... + [OKAY] +---------------------------------------------------------------------------------------------------- + +DeepSpeed C++/CUDA extension op report +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.---------------------------------------------------------------------------------------------------- + + +--------------------------------------------------JIT compiled ops requires ninja +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +JIT compiled ops requires ninja +-------------------------------------------------- + +DeepSpeed C++/CUDA extension op reportNOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +---------------------------------------------------------------------------------------------------- + +JIT compiled ops requires ninja +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +ninjaninjaninjaninja ........................................................................ [OKAY][OKAY][OKAY][OKAY] + + + +---------------------------------------------------------------------------------------------------- +---------------------------------------------------------------------------------------------------- + + +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +op nameop nameop name ................op name................................ installed................installedinstalled .. installed .... compatible ..compatible +compatiblecompatible + +-------------------------------------------------- +-------------------------------------------------- +---------------------------------------------------------------------------------------------------- + + +JIT compiled ops requires ninja +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +------------------------------------------------------------------------------------------------------------------------------------------------------ + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +DeepSpeed C++/CUDA extension op report + +DeepSpeed C++/CUDA extension op report-------------------------------------------------- +-------------------------------------------------- +-------------------------------------------------- +JIT compiled ops requires ninja +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- + +--------------------------------------------------JIT compiled ops requires ninja + +JIT compiled ops requires ninja +cpu_adam cpu_adam............... ...............cpu_adamcpu_adam [NO] [NO] .............................. ....... .......[NO][NO] [OKAY].......[OKAY] + ....... +[OKAY] +[OKAY] +fused_adamfused_adam .......................... [NO]fused_adamfused_adam[NO] ........................................ [OKAY][OKAY][NO] +[NO] + .......fused_lamb....... fused_lamb [OKAY] .............[OKAY] +............. + [NO][NO]fused_lamb fused_lamb........................... [OKAY] ............. +[NO][OKAY] +[NO]....... [OKAY]....... + [OKAY] +sparse_attn sparse_attn............ ............[NO] [NO]....... .......[OKAY]sparse_attnsparse_attn + [OKAY]........................transformer + transformer[NO]............[NO] ..........................[NO] [NO][OKAY].......[OKAY] + +.......[OKAY] +transformer[OKAY] +transformer............ stochastic_transformer stochastic_transformer............ [NO] .[NO]........ [NO]....... [OKAY].......[NO][OKAY] + +[OKAY]....... + stochastic_transformer[OKAY] stochastic_transformer + . .[NO] [NO]....... ....... [OKAY][OKAY] + +ninjaninjaninjaninja .................................... .................. ..................[OKAY] [OKAY][OKAY] +[OKAY] + + +------------------------------------------------------------------------------------------------------------------------------------------------------ +-------------------------------------------------- + + +op nameop name op nameop name ................ ................................ ................ installed installedinstalledinstalled ........ compatiblecompatiblecompatiblecompatible + + + +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + + + +cpu_adamcpu_adamcpu_adam ...............cpu_adam ..............................[NO]............... [NO][NO][NO]....... ....... .............. [OKAY] + [OKAY][OKAY][OKAY] + + +fused_adam .............fused_adam fused_adamfused_adam [NO] ............. .......................... ....... [NO][NO] [NO] [OKAY] ....... ....... +ninjaninjaninjaninja .................. .................................... .................. [OKAY][OKAY] + + ....... [OKAY][OKAY][OKAY] +fused_lamb + +[OKAY][OKAY] + +---------------------------------------------------------------------------------------------------- + +----------------------------------------------------------------------------------------------------op nameop name + + .............fused_lamb [NO]fused_lamb............. fused_lamb....... ............. [NO] ............. [OKAY][NO] ....... + [NO][OKAY]....... + op name................op name................ installed................installed ................ .. .. installedinstalled compatible compatible.. + + .......[OKAY] +[OKAY] + ..-------------------------------------------------- --------------------------------------------------compatible +compatible + + +---------------------------------------------------------------------------------------------------- + +sparse_attn ............ [NO] sparse_attn....... ............[OKAY] sparse_attn +cpu_adam ...............cpu_adam [NO]............... cpu_adamcpu_adam .......[NO] [OKAY]..................................... +[OKAY][NO][NO] + .............. [OKAY][OKAY] + +[NO] sparse_attn............transformer ...................[NO]............ [NO][OKAY] ....... [NO] + ....... [OKAY] .......[OKAY] +transformer +fused_adam .............fused_adam [NO]............. fused_adam fused_adam....... [NO] [OKAY]................................. +-------------------------------------------------- +--------------------------------------------------DeepSpeed C++/CUDA extension op report + +-------------------------------------------------- +DeepSpeed C++/CUDA extension op reportNOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + [OKAY]............ + [NO][NO][OKAY] + +---------------------------------------------------------------------------------------------------- + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.JIT compiled ops requires ninja + +---------------------------------------------------------------------------------------------------- +JIT compiled ops requires ninja + +--------------------------------------------------DeepSpeed C++/CUDA extension op report +-------------------------------------------------- + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.DeepSpeed C++/CUDA extension op report + +---------------------------------------------------------------------------------------------------- + +JIT compiled ops requires ninjaNOTE: Ops not installed will be just-in-time (JIT) compiled at +transformer transformer [NO] ........................ stochastic_transformer .......[NO] [NO] .[OKAY]....... +....... [OKAY][NO] +fused_lamb....... ....... ............. [OKAY]fused_lamb [OKAY] + + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +[OKAY] stochastic_transformer +....... stochastic_transformer [OKAY]. +[NO]............. .......[NO]fused_lamb fused_lamb [OKAY].................... +............. [NO][OKAY][NO] +-------------------------------------------------- +JIT compiled ops requires ninja +stochastic_transformer. [NO][NO] ........ [NO] .......[OKAY] +.......[OKAY] +[OKAY] + .............. [OKAY][OKAY] + +sparse_attn ............ [NO]sparse_attn ................... [OKAY][NO] + sparse_attnsparse_attn....... transformer............[OKAY] ............ + ............ [NO]transformer [NO].......[NO]............ .......[OKAY].......[NO] + [OKAY] .......[OKAY] + +transformer[OKAY] +transformer............stochastic_transformer stochastic_transformer............[NO] . .[NO] ....... [NO] [NO]....... [OKAY].............. + [OKAY][OKAY][OKAY] + +stochastic_transformer + stochastic_transformer . [NO]. .......[NO] [OKAY]....... + [OKAY] +ninjaninjaninjaninja .................................... ....................................[OKAY][OKAY] + +[OKAY][OKAY] +-------------------------------------------------- +-------------------------------------------------- +---------------------------------------------------------------------------------------------------- + +op name + op nameop name................op name ................ ................ installed ................ installedinstalled .. .. installed.. compatiblecompatible +.. +compatible ---------------------------------------------------------------------------------------------------- +compatible + + +-------------------------------------------------- +-------------------------------------------------- +cpu_adam cpu_adam...............cpu_adam cpu_adam [NO].............................. ......................[NO][NO] ....... [NO][OKAY]....... + .......[OKAY][OKAY] + +[OKAY] +fused_adam ............. fused_adam[NO]fused_adamfused_adam .............................................. [NO] [NO][OKAY] [NO] +....... ....... ....... [OKAY] [OKAY] +[OKAY] +fused_lamb + fused_lamb.............fused_lamb fused_lamb.............[NO] .......................... [NO].......[NO] [NO] ....... ....... [OKAY][OKAY]....... + + [OKAY][OKAY] + +sparse_attn ............ sparse_attnsparse_attn [NO]............ sparse_attn ............[NO]....... ............[NO][OKAY] ....... +[NO]....... [OKAY]transformer[OKAY]....... + + ............[OKAY]transformer +transformer [NO] transformer............ ............ ....... ............[NO] [NO][NO] [OKAY] .............. + ....... [OKAY] [OKAY] +[OKAY] +stochastic_transformer + stochastic_transformer .stochastic_transformerstochastic_transformer. [NO].[NO] . ....... [NO].......[NO][OKAY] [OKAY] +....... +....... [OKAY][OKAY] + +ninjaninjaninja ninja .................................... .................. .................. [OKAY] [OKAY][OKAY] +[OKAY] + + +------------------------------------------------------------------------------------------------------------------------------------------------------ + +-------------------------------------------------- +op name +op nameop name op name ................................ ................ ................ installedinstalled installedinstalled.... .... compatiblecompatible +compatiblecompatible +-------------------------------------------------- +-------------------------------------------------- + + +---------------------------------------------------------------------------------------------------- + +cpu_adamcpu_adam cpu_adam...............cpu_adam ............... [NO]............... ............... [NO] [NO]....... [NO]....... ....... .......[OKAY] +[OKAY][OKAY][OKAY] + + +fused_adam fused_adam.............fused_adam fused_adam............. [NO] ............. ............. ....... [NO] [NO] [OKAY][NO] .............. +....... [OKAY][OKAY][OKAY] + + +fused_lamb ............. fused_lambfused_lambfused_lamb[NO] ....... .......................... ............. [OKAY] [NO][NO] + [NO] ....... ....... ....... [OKAY] [OKAY] +[OKAY] + +sparse_attn ............ [NO] sparse_attnsparse_attnsparse_attn....... ........................[OKAY] ............ +[NO] [NO][NO]....... .......transformer[OKAY]....... + [OKAY]............ +[OKAY] transformer +[NO] ............transformer.......transformer [NO][OKAY]............ ............ + ....... [NO] [NO] [OKAY] ....... +....... stochastic_transformer [OKAY] [OKAY] +. + stochastic_transformerstochastic_transformer[NO] stochastic_transformer........ . .[OKAY][NO] + [NO][NO]....... ..............[OKAY] +[OKAY][OKAY] + +ninjaninjaninjaninja ........................................................................ [OKAY] [OKAY] +[OKAY][OKAY] + +-------------------------------------------------- +-------------------------------------------------- + +----------------------------------------------------------------------------------------------------op nameop name + + ................op name................op name installed ................installed .................. installed..compatibleinstalled +compatible .. +--------------------------------------------------.. +-------------------------------------------------- compatible + +compatible +-------------------------------------------------- +-------------------------------------------------- +cpu_adam cpu_adam............... ...............[NO]cpu_adam cpu_adam......................[NO] ...............[OKAY].......[NO] + [OKAY].......[NO] + [OKAY]....... + [OKAY] +fused_adam ............. [NO]fused_adam .................... fused_adam [OKAY] fused_adam[NO] + ............. ....... .............fused_lamb [NO][OKAY] [NO] +.................... fused_lamb[NO]....... [OKAY] .............[OKAY]....... + + [NO][OKAY]fused_lamb +fused_lamb....... ..........................[OKAY] + [NO][NO] .............. [OKAY][OKAY] +sparse_attn + ............ [NO] .......sparse_attn [OKAY]............ + [NO] ....... sparse_attntransformersparse_attn[OKAY] +............ ............ ............ [NO]transformer[NO] [NO]....... ............ ....... ....... [OKAY][NO] +[OKAY] [OKAY]....... + +stochastic_transformer [OKAY]transformer. +transformer [NO]............ stochastic_transformer ................... [NO] [NO] .[OKAY] +.............. [NO][OKAY] [OKAY] +....... + [OKAY]stochastic_transformer +stochastic_transformer .. [NO][NO] .............. [OKAY][OKAY] + +ninjaninjaninjaninja .................. ...................................................... [OKAY][OKAY][OKAY][OKAY] + + + +------------------------------------------------------------------------------------------------------------------------------------------------------ +-------------------------------------------------- + + +op nameop nameop name ................op name ................ ................installed................installed .. installed..compatibleinstalled + compatible....-------------------------------------------------- + + compatible--------------------------------------------------compatible + + +-------------------------------------------------- +-------------------------------------------------- +cpu_adam ............... [NO] cpu_adam....... cpu_adam...............[OKAY]cpu_adam [NO].............................. ....... + [NO] [NO] [OKAY] ....... +....... [OKAY][OKAY] + +fused_adam ............. fused_adam .............fused_adamfused_adam [NO].......................... [NO] .......[NO][NO] [OKAY].............. + ....... [OKAY][OKAY]fused_lamb + +[OKAY] +............. fused_lamb[NO] fused_lambfused_lamb............. .................... .............[NO][OKAY] [NO] .......[NO] + .......[OKAY] + .......[OKAY] +[OKAY] +sparse_attn ............ sparse_attn[NO] sparse_attn ............................... sparse_attn[NO] [OKAY][NO] .............. +............ [OKAY] [OKAY] +[NO] + transformer.......transformer transformer ............[OKAY] ............ + ............[NO][NO] transformer[NO]....... ....... ............ ....... [OKAY][NO] [OKAY] +[OKAY] + + .......stochastic_transformerstochastic_transformer [OKAY] stochastic_transformer. . + .[NO] [NO] stochastic_transformer[NO] ....... ........ ....... [OKAY] [OKAY][OKAY] +[NO] + +....... [OKAY] +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report-------------------------------------------------- +---------------------------------------------------------------------------------------------------- + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +DeepSpeed C++/CUDA extension op report +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +-------------------------------------------------- + +--------------------------------------------------JIT compiled ops requires ninjaNOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + + +--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- + + +JIT compiled ops requires ninja-------------------------------------------------- +DeepSpeed C++/CUDA extension op report + +JIT compiled ops requires ninja +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +------------------------------------------------------------------------------------------------------------------------------------------------------ + +DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op report + + +--------------------------------------------------DeepSpeed C++/CUDA extension op report-------------------------------------------------- + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- + +------------------------------------------------------------------------------------------------------------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + + +JIT compiled ops requires ninja +JIT compiled ops requires ninja-------------------------------------------------- + +DeepSpeed C++/CUDA extension op report + +JIT compiled ops requires ninja-------------------------------------------------- + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +----------------------------------------------------------------------------------------------------JIT compiled ops requires ninja + +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +DeepSpeed C++/CUDA extension op report + +---------------------------------------------------------------------------------------------------- +DeepSpeed C++/CUDA extension op report +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +------------------------------------------------------------------------------------------------------------------------------------------------------ + + +JIT compiled ops requires ninjaJIT compiled ops requires ninjaNOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + + +-------------------------------------------------- +JIT compiled ops requires ninja +ninjaninjaninjaninja ........................................................................ [OKAY][OKAY][OKAY][OKAY] + + + +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + + + +op nameop name op nameop name................ ................................installed................ installed ..installed installed .. compatible.. +.. compatiblecompatible --------------------------------------------------compatible + + + +---------------------------------------------------------------------------------------------------- + +-------------------------------------------------- +cpu_adam cpu_adamcpu_adam...............cpu_adam ............... ...............[NO] ............... [NO] ....... [NO] [NO]....... [OKAY]....... + [OKAY].......[OKAY] + +[OKAY] +fused_adam .............fused_adamfused_adam fused_adam.............[NO]............. [NO].............[NO]....... .......[NO].......[OKAY] +[OKAY].......[OKAY] + +[OKAY]fused_lamb +fused_lamb .............fused_lamb............. fused_lamb [NO] .............[NO] ............. ....... .......[NO] [OKAY].......[NO] +[OKAY] +[OKAY]....... + [OKAY] +sparse_attn sparse_attn............ sparse_attnsparse_attn ............ [NO] ............[NO]....... ............ [NO]....... [OKAY] +.......[NO][OKAY] + transformer.......[OKAY] +............ transformer [OKAY][NO]............ +transformer .......transformer[NO] ............ [OKAY] ................... +[NO] [NO][OKAY]....... +stochastic_transformer....... [OKAY][OKAY] +stochastic_transformer. + [NO].stochastic_transformer stochastic_transformer ....... [NO] .[OKAY] . + ....... [NO] [NO] [OKAY] ....... +....... [OKAY][OKAY] + +ninjaninjaninjaninja ...................................................... ..................[OKAY] [OKAY] +[OKAY] +[OKAY]-------------------------------------------------- +-------------------------------------------------- + + +----------------------------------------------------------------------------------------------------op name +op name + op name................op name ................installed ................................installed .. installedinstalled.. compatible + ....compatible -------------------------------------------------- + +compatiblecompatible +-------------------------------------------------- +-------------------------------------------------- + +-------------------------------------------------- +cpu_adam ............... [NO] .......cpu_adam cpu_adam cpu_adam ............... [OKAY]............... + ............... [NO] [NO] [NO] ..................... [OKAY][OKAY] +[OKAY]fused_adam + + ............. [NO] ....... [OKAY] +fused_lamb fused_adam............. fused_adam ............. fused_adam[NO]............. ....... [NO][OKAY].............[NO] +....... [NO] ....... [OKAY] ....... +[OKAY] [OKAY] + +fused_lamb ............. [NO]fused_lamb fused_lambsparse_attn ....... ............. ......................... [OKAY] [NO][NO] + [NO].............. [OKAY].......[OKAY] + +[OKAY] +transformer ............ sparse_attn[NO] ................... [OKAY]sparse_attn[NO] +sparse_attn ............ ....... stochastic_transformer............ [NO] [OKAY] +[NO]........ transformer....... [OKAY][NO] +............[OKAY]....... + transformer[NO][OKAY] ............ +transformer ....... [NO] ............ [OKAY] ....... +[NO] [OKAY]....... +stochastic_transformer [OKAY] +.stochastic_transformer [NO]stochastic_transformer. .......[NO]. .......[NO][OKAY] +[OKAY]....... + [OKAY] +ninjaninjaninjaninja .................................... .................. .................. [OKAY][OKAY] [OKAY] +[OKAY] + +-------------------------------------------------- +-------------------------------------------------- +-------------------------------------------------- +-------------------------------------------------- +op name +op nameop name op name ................................ ................................installed installedinstalled..installed ......compatible compatible +compatible +compatible---------------------------------------------------------------------------------------------------- + + + +---------------------------------------------------------------------------------------------------- + +cpu_adamcpu_adamcpu_adam cpu_adam............... ............... ..............................[NO][NO] [NO][NO]....... .....................[OKAY] +[OKAY][OKAY][OKAY] + + +fused_adam ............. fused_adamfused_adamfused_adam[NO] .............................................. [NO][OKAY][NO][NO] +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- +-------------------------------------------------- + ..................... fused_lamb[OKAY] [OKAY] + +[OKAY]............. + [NO]fused_lambfused_lamb fused_lamb ....... ............. ..........................[OKAY] [NO] +JIT compiled ops requires ninja + +DeepSpeed C++/CUDA extension op report +---------------------------------------------------------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +DeepSpeed C++/CUDA extension op report-------------------------------------------------- + +JIT compiled ops requires ninja-------------------------------------------------- + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +---------------------------------------------------------------------------------------------------- +JIT compiled ops requires ninja + +DeepSpeed C++/CUDA extension op report + [NO][NO]....... ..............[OKAY] + [OKAY][OKAY] + +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +sparse_attn ............ [NO] ....... [OKAY] +sparse_attn sparse_attntransformer............sparse_attn ............[NO] ........................ [NO][NO] .......[NO]....... [OKAY].......[OKAY] + ....... +[OKAY] transformer +transformer[OKAY] +........................stochastic_transformer transformer [NO] . [NO]....... ............ [NO] [OKAY]....... +.......[NO] [OKAY]stochastic_transformer.......[OKAY] + + .[OKAY] +stochastic_transformer[NO] stochastic_transformer....... . [OKAY].[NO] + [NO]....... .......[OKAY] +[OKAY] +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- + +--------------------------------------------------DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +-------------------------------------------------- + + +DeepSpeed C++/CUDA extension op reportJIT compiled ops requires ninjaDeepSpeed C++/CUDA extension op report +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- + + +--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- + + +JIT compiled ops requires ninja--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + + +JIT compiled ops requires ninja-------------------------------------------------- + +JIT compiled ops requires ninja +---------------------------------------------------------------------------------------------------- + +DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op report + +---------------------------------------------------------------------------------------------------- + +----------------------------------------------------------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + + +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +DeepSpeed C++/CUDA extension op report-------------------------------------------------- +JIT compiled ops requires ninja + + +--------------------------------------------------JIT compiled ops requires ninja-------------------------------------------------- + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +---------------------------------------------------------------------------------------------------- + +JIT compiled ops requires ninja +JIT compiled ops requires ninja +ninjaninjaninjaninja ........................................................................ [OKAY] +[OKAY][OKAY][OKAY]-------------------------------------------------- + + + +------------------------------------------------------------------------------------------------------------------------------------------------------op name + + +op name................op name installedop name ................ ................ .................. installed installedcompatible installed .. + .. --------------------------------------------------.. compatible +compatible + +compatible---------------------------------------------------------------------------------------------------- + + +-------------------------------------------------- +cpu_adam ............... [NO] cpu_adamcpu_adam....... cpu_adam .............................. [OKAY] ............... +[NO][NO] [NO].............. [OKAY]....... +[OKAY] +[OKAY] +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +---------------------------------------------------------------------------------------------------- +----------------------------------------------------------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +fused_adam ............. [NO] ....... [OKAY] + + + +DeepSpeed C++/CUDA extension op report--------------------------------------------------DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op report + + + +--------------------------------------------------JIT compiled ops requires ninja---------------------------------------------------------------------------------------------------- + + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + + +fused_adamfused_adam fused_lambfused_adam ............. ....................................... [NO] [NO] [NO][NO] ....... ....... ....... [OKAY]....... [OKAY] +[OKAY] +------------------------------------------------------------------------------------------------------------------------------------------------------ + + +JIT compiled ops requires ninjaJIT compiled ops requires ninjaJIT compiled ops requires ninja + + + +[OKAY] +fused_lambfused_lambfused_lamb ............. ............. ............. [NO] [NO][NO]....... sparse_attn ..............[OKAY] +............[OKAY][OKAY] + +[NO] ....... [OKAY] +transformer ............ [NO] .......sparse_attn sparse_attn sparse_attn[OKAY] ............ ............ +............[NO] [NO][NO]....... stochastic_transformer .............. [OKAY] +.[OKAY][OKAY] + +[NO]transformer transformertransformer ............................... [OKAY]............ +[NO][NO] [NO].............. .......[OKAY][OKAY] + +[OKAY] +stochastic_transformerstochastic_transformer stochastic_transformer .. . [NO] [NO] [NO] ....... ....... ....... [OKAY] [OKAY] +[OKAY] + +---------------------------------------------------------------------------------------------------- + +DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op report + +---------------------------------------------------------------------------------------------------- + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +-------------------------------------------------- +--------------------------------------------------JIT compiled ops requires ninja + +JIT compiled ops requires ninja-------------------------------------------------- + +DeepSpeed C++/CUDA extension op report +---------------------------------------------------------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +-------------------------------------------------- +DeepSpeed C++/CUDA extension op reportJIT compiled ops requires ninja + +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +ninjaninjaninjaninja .................................... .................. ..................[OKAY] [OKAY] +[OKAY] +[OKAY]-------------------------------------------------- + +-------------------------------------------------- + +--------------------------------------------------op name--------------------------------------------------op name + +................................ op nameop name installed installed ................................ .. .. installedcompatiblecompatibleinstalled + +..--------------------------------------------------..-------------------------------------------------- + +compatiblecompatible + +---------------------------------------------------------------------------------------------------- + +cpu_adam cpu_adam ............... [NO] .......cpu_adam ...............cpu_adam[OKAY] ............... + [NO] .......[NO]............... [OKAY]....... +[NO] [OKAY]fused_adam....... + ............. [OKAY][NO] + fused_adam....... .............[OKAY] +[NO] ....... fused_lambfused_adam[OKAY] +..........................fused_adam fused_lamb [NO] [NO] ............. ....... ....... [NO] [OKAY][OKAY] + +............. .......[NO]fused_lamb sparse_attn[OKAY]............. +................... [NO][OKAY][NO] + .............. fused_lamb[OKAY][OKAY]sparse_attn + + ............ [NO]transformer............. [NO]................... [NO][OKAY] ....... +....... sparse_attn [OKAY]transformer[OKAY] + ........................ [NO]stochastic_transformer[NO] + ....... ........[OKAY] +[NO][OKAY] +.......transformer [OKAY] stochastic_transformer +............ [NO]. .......[NO] [OKAY]....... + [OKAY] +stochastic_transformer . [NO]sparse_attn ................... [OKAY] +[NO] ....... [OKAY] +transformer ............ [NO] ....... [OKAY] +stochastic_transformer . [NO] ....... [OKAY] +ninjaninjaninjaninja .................................... .................................... [OKAY] [OKAY] [OKAY] + +[OKAY] +-------------------------------------------------- +-------------------------------------------------- +---------------------------------------------------------------------------------------------------- + +op name +op name op nameop name ................................................................ installed installedinstalled installed .... .. .. compatible compatiblecompatiblecompatible + + + +------------------------------------------------------------------------------------------------------------------------------------------------------ +-------------------------------------------------- + + +cpu_adamcpu_adamcpu_adam cpu_adam ............................................................ [NO][NO][NO] [NO] .............. ....... .......[OKAY] [OKAY] +[OKAY] +[OKAY] + +fused_adamfused_adamfused_adam fused_adam....................................... ............. [NO][NO] [NO] [NO]....... ....... .............. [OKAY] +[OKAY][OKAY][OKAY] + + +fused_lamb .............fused_lamb fused_lambfused_lamb [NO] ....................................... .......[NO][NO][NO] [OKAY]....... + [OKAY] +.............. [OKAY][OKAY] + +sparse_attn ............ [NO]sparse_attn ................... sparse_attn [OKAY] [NO]sparse_attn............ + ...................[NO]transformer [NO][OKAY]............ +....... ....... transformer[NO] [OKAY] [OKAY] +................... + [OKAY]transformertransformer[NO] + ............................... [NO]stochastic_transformer[NO][OKAY] +............... [OKAY][NO]stochastic_transformer +[OKAY] +....... stochastic_transformer.[OKAY] +stochastic_transformer[NO]. .......[NO]. .......[OKAY][NO] + [OKAY]....... + [OKAY] +ninjaninjaninjaninja ...................................................... ..................[OKAY][OKAY][OKAY] +[OKAY] + +-------------------------------------------------- +-------------------------------------------------- +-------------------------------------------------- +--------------------------------------------------op name + op name +................ op name................ op name installed ................ ................installed ..installed .. installedcompatible + ..compatible--------------------------------------------------.. + +--------------------------------------------------compatiblecompatible + + +---------------------------------------------------------------------------------------------------- + +cpu_adam ...............cpu_adam [NO]............... cpu_adam....... cpu_adam [NO] [OKAY] .............................. + ....... [NO] [NO] [OKAY] ....... +....... [OKAY][OKAY] + +fused_adam ............. [NO] .......fused_adam [OKAY]............. + [NO] fused_adamfused_adam....... fused_lamb ............. ............. .............[OKAY] +[NO][NO][NO] ..............fused_lamb .......[OKAY] +[OKAY]............. + [OKAY][NO] +fused_lamb....... [OKAY].............fused_lamb + sparse_attn[NO]............. ...................[NO] [NO][OKAY]....... + .......sparse_attn[OKAY] +[OKAY]............ + [NO] transformer....... ............[OKAY] +sparse_attn[NO] transformer ............ [NO] .............. ............ [OKAY] [OKAY] +[NO]sparse_attn + ................... stochastic_transformer[NO][OKAY]stochastic_transformer . +....... . [NO] [OKAY]transformer [NO] ....... + ............ .......transformer[OKAY] [OKAY] +............[NO] + .......[NO] .......[OKAY] +[OKAY] +stochastic_transformer stochastic_transformer . [NO]. .......[NO] [OKAY]....... + [OKAY] +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +---------------------------------------------------------------------------------------------------- + +DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op report + +-------------------------------------------------- +--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- + +--------------------------------------------------JIT compiled ops requires ninja + +--------------------------------------------------JIT compiled ops requires ninja + +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +ninjaninjaninjaninja ........................................................................ [OKAY][OKAY][OKAY][OKAY] + + + +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + + + +op nameop nameop name op name ................ ................ ................................ installed installed installed.. installed .. ....compatiblecompatible + compatible +compatible +---------------------------------------------------------------------------------------------------- +-------------------------------------------------- + + +-------------------------------------------------- +cpu_adam ...............cpu_adam cpu_adamcpu_adam [NO]............................................. [NO][NO] [NO] ..................... [OKAY][OKAY][OKAY] + +....... + [OKAY] +fused_adam fused_adam............. fused_adam.............[NO] .............[NO]....... [NO]....... [OKAY] ....... +[OKAY] +[OKAY] +fused_lambfused_adamfused_lamb fused_lamb ............. ............. .......................... [NO] [NO][NO][NO] ....... .............. [OKAY] .......[OKAY] +[OKAY] + + [OKAY] +fused_lamb ............. [NO] ....... [OKAY] +sparse_attnsparse_attnsparse_attn .................................... [NO][NO][NO] ..................... [OKAY][OKAY][OKAY] + + +transformertransformer transformer ............ ............ sparse_attn............ [NO] [NO]............[NO] [NO]....... .......[OKAY] ....... +....... [OKAY] [OKAY] +[OKAY] +stochastic_transformer + transformer .............stochastic_transformerstochastic_transformer [NO] [NO]. ........ ....... [NO] [NO] [OKAY] +[OKAY].............. +[OKAY][OKAY] + +stochastic_transformer . [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report-------------------------------------------------- +-------------------------------------------------- + +DeepSpeed C++/CUDA extension op reportNOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +-------------------------------------------------- +--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +JIT compiled ops requires ninja-------------------------------------------------- + +JIT compiled ops requires ninja +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +------------------------------------------------------------------------------------------------------------------------------------------------------ + +DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op report + + +DeepSpeed C++/CUDA extension op report---------------------------------------------------------------------------------------------------- + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + + +--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- + + +--------------------------------------------------JIT compiled ops requires ninja + +JIT compiled ops requires ninjaJIT compiled ops requires ninja + +ninjaninjaninjaninja ........................................................................ [OKAY][OKAY][OKAY][OKAY] + + +-------------------------------------------------- +-------------------------------------------------- +---------------------------------------------------------------------------------------------------- + +op name + op nameop name................op name ................ ................ installed................ installed installed .. .. installed.. compatible +compatible..compatible-------------------------------------------------- + + +----------------------------------------------------------------------------------------------------compatible + + +-------------------------------------------------- +cpu_adam ............... [NO] cpu_adam.......cpu_adam cpu_adam [OKAY] ............................................. + [NO][NO][NO] ..................... [OKAY][OKAY][OKAY] + + +fused_adam ............. [NO] ....... [OKAY]fused_adamfused_adamfused_adam + ....................................... fused_lamb [NO] [NO][NO] .................... ....... ....... [OKAY][NO] [OKAY] +[OKAY] +....... + fused_lamb[OKAY] fused_lamb +fused_lamb............. ..........................[NO] [NO][NO]....... ..............[OKAY] + [OKAY][OKAY] + +sparse_attn ............ [NO] ....... [OKAY] +transformersparse_attn ............sparse_attnsparse_attn [NO].................................... .......[NO][NO] [OKAY][NO] ....... + ....... .......[OKAY] +[OKAY]stochastic_transformer[OKAY] + transformer + .transformertransformer............ [NO]........................ [NO] ....... [NO] [NO].............. [OKAY] [OKAY]....... + [OKAY] + +[OKAY]stochastic_transformer + stochastic_transformer . .[NO]stochastic_transformer [NO]....... ........[OKAY] +[NO][OKAY] +....... [OKAY] +ninjaninjaninjaninja ........................................................................ [OKAY][OKAY][OKAY][OKAY] + + + +------------------------------------------------------------------------------------------------------------------------------------------------------ +-------------------------------------------------- + +op name + op nameop name................ ................op name ................ installed installed installed.. ................ .. compatible ..installed +compatible +--------------------------------------------------..compatible +-------------------------------------------------- + +compatible-------------------------------------------------- + +-------------------------------------------------- +cpu_adam cpu_adam............... cpu_adam............... cpu_adam [NO] ...............[NO] ............... ....... .......[NO] [NO][OKAY] [OKAY] + +.............. [OKAY][OKAY] + +fused_adamfused_adam ..........................fused_adam fused_adam [NO][NO] ............. ............. .............. [NO][NO][OKAY] [OKAY] +....... +....... fused_lamb[OKAY] +[OKAY].............fused_lamb +[NO].............fused_lamb .......[NO]fused_lamb .............[OKAY]....... + .............[NO][OKAY] +[NO]....... .......[OKAY] +[OKAY] +sparse_attn ............ [NO] ....... [OKAY]sparse_attn + ............transformersparse_attnsparse_attn [NO]........................ ............ [NO] ....... [NO][NO]....... [OKAY] .......[OKAY] +....... + [OKAY]transformer[OKAY] + +stochastic_transformer............ transformer[NO]transformer. ...................[NO]............ [OKAY][NO][NO]....... + ..............[OKAY] + stochastic_transformer[OKAY][OKAY] + +. [NO]stochastic_transformerstochastic_transformer ....... .[OKAY]. + [NO][NO] .............. [OKAY][OKAY] + +ninjaninjaninjaninja .................. .................................... .................. [OKAY][OKAY] [OKAY] +[OKAY] + +---------------------------------------------------------------------------------------------------- + +-------------------------------------------------- + +--------------------------------------------------op nameop nameop name + ................op name................ ................ installed ................ installedinstalled .. ..installed .. compatible .. compatiblecompatible + + -------------------------------------------------- +compatible +---------------------------------------------------------------------------------------------------- + + +-------------------------------------------------- +cpu_adam ...............cpu_adam cpu_adam cpu_adam[NO] .................................................... [NO][OKAY][NO] +[NO] ....... .............. [OKAY][OKAY][OKAY] + + +fused_adam ............. [NO] ....... [OKAY] +fused_adamfused_adamfused_adamfused_lamb ............. .......................... ............. [NO] [NO][NO][NO] ....... .............. ....... [OKAY][OKAY] [OKAY] +[OKAY] + + +fused_lamb .............fused_lamb fused_lamb [NO] ................................. [NO][OKAY][NO]sparse_attn + .......................... [OKAY][NO][OKAY] + +....... [OKAY] +sparse_attn transformer............ ............ [NO]sparse_attn[NO] sparse_attn .......................... [OKAY][OKAY][NO] +............ + [NO].......stochastic_transformer transformer [OKAY]....... .............[OKAY] + +[NO][NO] transformer.............. transformer [OKAY]............[OKAY]............ +[NO] + [NO]....... stochastic_transformer....... [OKAY][OKAY] +. + [NO]stochastic_transformer .......stochastic_transformer . [OKAY] +.[NO] [NO]....... .......[OKAY] +[OKAY] +---------------------------------------------------------------------------------------------------- +----------------------------------------------------------------------------------------------------DeepSpeed C++/CUDA extension op report + + +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +DeepSpeed C++/CUDA extension op report +--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- + +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +JIT compiled ops requires ninja-------------------------------------------------- +-------------------------------------------------- + +-------------------------------------------------- +JIT compiled ops requires ninja +JIT compiled ops requires ninja +JIT compiled ops requires ninja + +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +--------------------------------------------------JIT compiled ops requires ninja-------------------------------------------------- + +DeepSpeed C++/CUDA extension op report + +DeepSpeed C++/CUDA extension op report-------------------------------------------------- + +--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- +---------------------------------------------------------------------------------------------------- + + +JIT compiled ops requires ninjaDeepSpeed C++/CUDA extension op reportJIT compiled ops requires ninja + + +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +-------------------------------------------------- +----------------------------------------------------------------------------------------------------DeepSpeed C++/CUDA extension op report + +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- + +DeepSpeed C++/CUDA extension op report--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.---------------------------------------------------------------------------------------------------- + + +--------------------------------------------------JIT compiled ops requires ninja +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +JIT compiled ops requires ninja + +-------------------------------------------------- +JIT compiled ops requires ninja +ninjaninjaninjaninja ........................................................................ [OKAY] [OKAY] +[OKAY][OKAY] + +-------------------------------------------------- +---------------------------------------------------------------------------------------------------- +-------------------------------------------------- + +op name + op nameop nameop name ................ ................ ................................ installed installed installedinstalled.. .. ....compatible + compatiblecompatible--------------------------------------------------compatible + + + +------------------------------------------------------------------------------------------------------------------------------------------------------ + + +cpu_adam ............... [NO] cpu_adamcpu_adam.......cpu_adam [OKAY]............................................. + [NO][NO] .............. [OKAY][OKAY] + +[NO] ....... fused_adam[OKAY] +............. [NO] ....... fused_adamfused_adam[OKAY] +.......................... [NO][NO]fused_lamb ....................fused_adam ....... [NO][OKAY][OKAY] + +.................... [OKAY][NO]fused_lamb +fused_lamb ................................. [NO][NO] .............. [OKAY][OKAY] +[OKAY] +sparse_attn + fused_lamb............ [NO] .................... [NO] [OKAY]....... [OKAY] + +sparse_attnsparse_attn transformer........................ ............[NO] [NO] [NO].............. [OKAY].......[OKAY] +sparse_attn + [OKAY]............ + transformertransformer[NO] ........................ stochastic_transformer[NO][NO] ............... ....... [OKAY][OKAY] + [NO] +[OKAY] ....... +stochastic_transformer transformer[OKAY] +............stochastic_transformer. [NO][NO]. ..............[NO] [OKAY].......[OKAY] + +[OKAY] +stochastic_transformer . [NO] ....... [OKAY] +ninjaninjaninjaninja ...................................................... ..................[OKAY][OKAY][OKAY] + + +--------------------------------------------------[OKAY] +---------------------------------------------------------------------------------------------------- + +op name + --------------------------------------------------op nameop name................ + ................op name................installed installed..................installed ..compatible.. +installed -------------------------------------------------- compatible +compatible +.. + ----------------------------------------------------------------------------------------------------compatible + + +cpu_adam-------------------------------------------------- +............... [NO] cpu_adam.......cpu_adam ............... [OKAY][NO]............... +cpu_adam .......[NO]............... [OKAY].......[NO] + fused_adam[OKAY]....... +............. [OKAY][NO] +....... [OKAY]fused_adam + ............. [NO]fused_adam fused_lamb .......fused_adam .......................... [OKAY] .............[NO][NO] + [NO].......fused_lamb....... [OKAY].............[OKAY]....... + [NO] +[OKAY] fused_lamb....... + .............[OKAY] + fused_lamb[NO] .............sparse_attn....... [OKAY][NO]............ + .......[NO] .......sparse_attn[OKAY] +[OKAY]............ + [NO] .......transformer sparse_attn[OKAY]............ + [NO]............transformer ....... ............sparse_attn [NO][NO] [OKAY]............ + ....... ....... [OKAY][NO]stochastic_transformer[OKAY] + +.......transformer . [OKAY] stochastic_transformer............[NO] + [NO]........transformer [OKAY] +.......[NO]............ .......[OKAY][NO] +[OKAY] +.......stochastic_transformer [OKAY] +. [NO]stochastic_transformer ....... [OKAY]. + [NO] ....... [OKAY] +ninjaninjaninjaninja ...................................................... ..................[OKAY][OKAY][OKAY] + + +-------------------------------------------------- +--------------------------------------------------[OKAY]-------------------------------------------------- +op name + + op nameop name................ --------------------------------------------------................................installed + installedinstalled.. ..op name.. compatiblecompatible................ + + compatible----------------------------------------------------------------------------------------------------installed + + + --------------------------------------------------.. + compatible +-------------------------------------------------- +cpu_adamcpu_adam .............................. cpu_adam [NO][NO]cpu_adam ............................................ [OKAY] [NO][OKAY] +[NO] + .............. [OKAY][OKAY] + +fused_adam .............fused_adam [NO]............. .......[NO]fused_adam fused_adam [OKAY] +....... ............. ............. [OKAY] [NO]fused_lamb[NO] + ........................... [OKAY][NO]fused_lamb[OKAY] +.................... + fused_lamb[OKAY][NO] + fused_lamb.................... .............[OKAY][NO] + [NO]....... .......[OKAY] +[OKAY] +sparse_attn ............ [NO] ....... [OKAY] +sparse_attn ............ transformer[NO]sparse_attn sparse_attn ................... ............ ............[NO][OKAY] + .......[NO][NO] transformer [OKAY] .............. +............ [OKAY][OKAY][NO]stochastic_transformer + + ....... transformertransformer[OKAY]. + ............[NO]stochastic_transformer............ [NO] ........ [NO] [OKAY] .......[NO] +....... .......[OKAY][OKAY] + +[OKAY] +stochastic_transformerstochastic_transformer .. [NO][NO] .............. [OKAY][OKAY] + + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.async_io + ............... [NO] ....... [NO] +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utilstransformer_inference .................... [NO][NO] .............. [OKAY][OKAY] + +quantizer utils.............. ..................[NO] [NO]....... .......[OKAY] +[OKAY] +-------------------------------------------------- +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +transformer_inference .. [NO] ....... async_io[OKAY] + ............... utils[NO] ......................... [NO][NO] +....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +transformer_inference .. [NO]-------------------------------------------------- +....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +transformer_inference ..utils [NO].................. .......[NO] [OKAY]....... + [OKAY] +utils ..................quantizer [NO].............. .......[NO] [OKAY]....... + [OKAY] +quantizer ..............-------------------------------------------------- +[NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.async_io + ............... [NO] ....... [NO] + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +async_io ............... [NO] ....... [NO]transformer_inference + .. [NO] ....... [OKAY] +utils transformer_inference.................. ..[NO] [NO]....... .......[OKAY] +[OKAY] +quantizer utils.............. ..................[NO] .......[NO] [OKAY]....... + [OKAY] +-------------------------------------------------- +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +async_io ............... [NO] ....... [NO] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum [WARNING]  async_io: please install the libaio-devel package with yum + + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_ioasync_io ............... [NO]............... .......[NO] [NO]....... + [NO] +transformer_inference .. [NO] transformer_inference....... ..[OKAY] +[NO] ....... [OKAY] +utils .................. [NO] ....... utils[OKAY] +.................. [NO] ....... quantizer[OKAY] +.............. [NO] ....... quantizer[OKAY] +.............. [NO] .......-------------------------------------------------- +[OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + [WARNING]  async_io: please install the libaio-devel package with yum +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_ioasync_io ............... ...............[NO] [NO]....... .......[NO] +[NO] +transformer_inference ..transformer_inference [NO].. .......[NO] [OKAY]....... + [OKAY] +utils ..................utils [NO].................. [NO]....... .......[OKAY] +[OKAY] +quantizer quantizer.............. ..............[NO] [NO]....... .......[OKAY] +[OKAY] +---------------------------------------------------------------------------------------------------- + + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference [WARNING]  async_io: please install the libaio-devel package with yum .. [NO] ....... + [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +transformer_inference .. [NO] ....... [OKAY] +async_io ...............utils [NO].................. .......[NO] [NO]....... + [OKAY] +quantizer .............. [NO] ....... transformer_inference[OKAY] +.. [NO] ....... --------------------------------------------------[OKAY] + +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +DeepSpeed general environment info:utils .................. +[NO] ....... [OKAY] +torch install path quantizer............... .............. [NO] ....... [OKAY] +['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +-------------------------------------------------- +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: + [WARNING]  async_io: please install the libaio-devel package with yum +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +utils .................. [NO] ....... [OKAY] +async_ioquantizer ............................. [NO][NO] ....... .......[NO] +[OKAY] +-------------------------------------------------- +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +DeepSpeed general environment info:DeepSpeed general environment info: + +transformer_inference .. [NO] ....... [OKAY] +torch install pathtorch install path .............................. ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch']['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] + +torch version ....................torch version 1.8.1.................... + 1.8.1 +torch cuda version ...............torch cuda version 11.1............... + nvcc version11.1 +utils .................. [NO] ....... [OKAY] +.....................nvcc version 11.2..................... + deepspeed install path11.2 +...........deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +quantizer .............. [NO] ....... [OKAY] +deepspeed info['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +...................deepspeed info 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix................... + deepspeed wheel compiled w.0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +......deepspeed wheel compiled w. torch 1.8, cuda 11.1...... + torch 1.8, cuda 11.1 +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +transformer_inference .. [NO] ....... [OKAY] +async_io ............... [NO] ....... [NO] +utils .................. [NO] ....... [OKAY] +transformer_inference .. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.-------------------------------------------------- + +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +[NO] ....... [OKAY] +-------------------------------------------------- +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] + [WARNING]  async_io: please install the libaio-devel package with yum +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum [WARNING]  async_io: please install the libaio-devel package with yum + + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + +async_ioasync_io ............... ...............[NO] [NO]....... .......[NO] +[NO] +transformer_inference transformer_inference.. ..[NO] [NO]....... .......[OKAY] +[OKAY] +utilsutils .................................... [NO][NO] .............. [OKAY][OKAY] + +quantizerquantizer ............................ [NO][NO] .............. [OKAY][OKAY] + +---------------------------------------------------------------------------------------------------- + + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io  [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found................ + [NO] ....... [NO] +transformer_inferenceasync_io .. ...............[NO] .......[NO] [OKAY]....... + [NO] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO]transformer_inference ......... [OKAY][NO] + ....... [OKAY] +-------------------------------------------------- +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +transformer_inference .. [NO] ....... [OKAY] +async_io ............... [NO] ....... [NO] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +transformer_inference .. [NO] ....... [OKAY] +-------------------------------------------------- +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] + [WARNING]  async_io: please install the libaio-devel package with yumquantizer .............. [NO] + ....... [OKAY] +-------------------------------------------------- + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] .......  [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.[OKAY] + +utils .................. [NO] ....... [OKAY] +async_io quantizer............... ..............[NO] [NO]....... .......[NO] [OKAY] + +-------------------------------------------------- +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +DeepSpeed general environment info: +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +torch cuda version ............... 11.1 +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +DeepSpeed general environment info:DeepSpeed general environment info: + +torch install pathtorch install path ............... ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch']['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] + +torch versiontorch version ........................................ 1.8.11.8.1 + +torch cuda version torch cuda version............... ...............11.1 +11.1 +DeepSpeed general environment info: +nvcc version nvcc version..................... .....................11.2 +11.2 +deepspeed install path deepspeed install path........... ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed']['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] + +deepspeed infodeepspeed info ...................................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +0.5.5+57dee5a, 57dee5a, pp_deadlock_fixdeepspeed wheel compiled w. +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 + deepspeed wheel compiled w....... ......torch 1.8, cuda 11.1 +torch 1.8, cuda 11.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + + +DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op report +DeepSpeed C++/CUDA extension op report + +DeepSpeed C++/CUDA extension op report +---------------------------------------------------------------------------------------------------- + +-------------------------------------------------- +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.------------------------------------------------------------------------------------------------------------------------------------------------------ + + +-------------------------------------------------- +JIT compiled ops requires ninja +JIT compiled ops requires ninjaJIT compiled ops requires ninja + +JIT compiled ops requires ninja + +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +DeepSpeed general environment info:DeepSpeed general environment info: + +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +torch install pathtorch install path ............... ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch']['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] + +torch version torch version.................... ....................1.8.1 +1.8.1 +torch cuda version torch cuda version............... ...............11.1 +11.1 +nvcc version nvcc version..................... .....................11.2 +11.2deepspeed install path + deepspeed install path........... ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed']deepspeed info + deepspeed info................... ...................0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +0.5.5+57dee5a, 57dee5a, pp_deadlock_fixdeepspeed wheel compiled w. + deepspeed wheel compiled w....... ......torch 1.8, cuda 11.1 +torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +ninjaninjaninjaninja ........................................................................ [OKAY] [OKAY][OKAY][OKAY] + + + +-------------------------------------------------- +---------------------------------------------------------------------------------------------------- +--------------------------------------------------op name + +op name op name................ ................op name................ ................installed installedinstalled installed.... .. .. compatiblecompatible compatible + +compatible-------------------------------------------------- +-------------------------------------------------- + +-------------------------------------------------- +-------------------------------------------------- + +cpu_adamcpu_adam cpu_adam cpu_adam............... .............................. ...............[NO] [NO][NO]....... [NO] .......[OKAY]....... +....... [OKAY][OKAY][OKAY] + + +fused_adam ............. [NO] .......fused_adam fused_adamfused_adam [OKAY]............. ............. + .............[NO][NO] fused_lamb [NO]........................... .......[OKAY][NO][OKAY] + +.......[OKAY] fused_lamb +[OKAY] fused_lamb +............. .............fused_lamb[NO] [NO] .................... ....... [NO] [OKAY] +[OKAY].......sparse_attn + [OKAY]............ +[NO] ....... [OKAY] +sparse_attntransformer sparse_attn........................sparse_attn [NO] ............ [NO]............ .............. [NO] [OKAY][NO] + [OKAY] stochastic_transformer....... ....... + .[OKAY][OKAY] +transformer +[NO] transformer.......transformer............ ............[NO][OKAY]............ + ....... [NO] [NO] [OKAY] ....... +....... [OKAY][OKAY] +stochastic_transformer + . stochastic_transformerstochastic_transformer[NO] .. ....... [NO][OKAY][NO] + .............. [OKAY][OKAY] + +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ...............DeepSpeed general environment info: +['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch']torch install path + ............... torch version .................... 1.8.1 +['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch']torch cuda version + ............... torch version11.1 +....................nvcc version 1.8.1..................... + 11.2torch cuda version + deepspeed install path............... ...........11.1 +nvcc version ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed']..................... + 11.2deepspeed info + deepspeed install path................... ...........0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w.['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +...... deepspeed infotorch 1.8, cuda 11.1 +................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info:DeepSpeed general environment info: + +torch install pathtorch install path ............... ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version torch version.................... ....................1.8.1 +1.8.1torch cuda version + ...............torch cuda version 11.1............... + nvcc version11.1 +.....................nvcc version 11.2..................... + deepspeed install path11.2 +...........deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +...................deepspeed info 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix................... + 0.5.5+57dee5a, 57dee5a, pp_deadlock_fixdeepspeed wheel compiled w. + ......deepspeed wheel compiled w. torch 1.8, cuda 11.1...... + torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +**** Git info for Megatron: git_hash=829cefd git_branch=main **** + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +**** Git info for Megatron: git_hash=829cefd git_branch=main **** + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +**** Git info for Megatron: git_hash=829cefd git_branch=main **** + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main ******** Git info for Megatron: git_hash=829cefd git_branch=main **** + +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +using world size: 128, data-parallel-size: 1, tensor-model-parallel size: 4, pipeline-model-parallel size: 32 +using torch.float16 for parameters ... +------------------------ arguments ------------------------ + accumulate_allreduce_grads_in_fp32 .............. False + adam_beta1 ...................................... 0.9 + adam_beta2 ...................................... 0.95 + adam_eps ........................................ 1e-08 + adlr_autoresume ................................. False + adlr_autoresume_interval ........................ 1000 + apply_query_key_layer_scaling ................... True + apply_residual_connection_post_layernorm ........ False + attention_dropout ............................... 0.1 + attention_softmax_in_fp32 ....................... False + bert_binary_head ................................ True + bert_load ....................................... None + bf16 ............................................ False + bias_dropout_fusion ............................. True + bias_gelu_fusion ................................ True + biencoder_projection_dim ........................ 0 + biencoder_shared_query_context_model ............ False + block_data_path ................................. None + checkpoint_activations .......................... True + checkpoint_in_cpu ............................... False + checkpoint_num_layers ........................... 1 + clip_grad ....................................... 1.0 + codecarbon_dir .................................. None + consumed_train_samples .......................... 0 + consumed_train_tokens ........................... 0 + consumed_valid_samples .......................... 0 + contigious_checkpointing ........................ False + cpu_optimizer ................................... False + cpu_torch_adam .................................. False + curriculum_learning ............................. False + data_impl ....................................... mmap + data_parallel_size .............................. 1 + data_path ....................................... ['/gpfswork/rech/six/commun/datasets-custom/oscar-en/meg-gpt2_text_document'] + dataloader_type ................................. single + DDP_impl ........................................ local + decoder_seq_length .............................. None + deepscale ....................................... False + deepscale_config ................................ None + deepspeed ....................................... True + deepspeed_activation_checkpointing .............. True + deepspeed_config ................................ ./ds_config.1645939.json + deepspeed_mpi ................................... False + distribute_checkpointed_activations ............. False + distributed_backend ............................. nccl + embedding_path .................................. None + encoder_seq_length .............................. 2048 + eod_mask_loss ................................... False + eval_interval ................................... 1000 + eval_iters ...................................... 5 + evidence_data_path .............................. None + exit_duration_in_mins ........................... 55 + exit_interval ................................... None + ffn_hidden_size ................................. 46400 + finetune ........................................ False + fp16 ............................................ True + fp16_lm_cross_entropy ........................... False + fp32_residual_connection ........................ False + gigaflos_no_embeds .............................. 0 + global_batch_size ............................... 2048 + glu_activation .................................. None + hidden_dropout .................................. 0.1 + hidden_size ..................................... 11600 + hysteresis ...................................... 2 + ict_head_size ................................... None + ict_load ........................................ None + img_dim ......................................... 224 + indexer_batch_size .............................. 128 + indexer_log_interval ............................ 1000 + init_method_std ................................. 0.02 + init_method_xavier_uniform ...................... False + initial_loss_scale .............................. 4294967296 + kv_channels ..................................... 145 + layernorm_epsilon ............................... 1e-05 + lazy_mpu_init ................................... None + load ............................................ /gpfsscratch/rech/six/commun/checkpoints/tr8b-104B/checkpoints + local_rank ...................................... 0 + log_batch_size_to_tensorboard ................... True + log_interval .................................... 1 + log_learning_rate_to_tensorboard ................ True + log_loss_scale_to_tensorboard ................... True + log_num_zeros_in_grad ........................... False + log_params_norm ................................. False + log_timers_to_tensorboard ....................... True + log_validation_ppl_to_tensorboard ............... True + loss_on_targets_only ............................ False + loss_scale ...................................... 12.0 + loss_scale_window ............................... 1000 + lr .............................................. 6e-05 + lr_decay_iters .................................. None + lr_decay_samples ................................ None + lr_decay_style .................................. cosine + lr_decay_tokens ................................. 260000000000 + lr_warmup_fraction .............................. None + lr_warmup_iters ................................. 0 + lr_warmup_samples ............................... 216320 + make_vocab_size_divisible_by .................... 128 + mask_prob ....................................... 0.15 + masked_softmax_fusion ........................... True + max_position_embeddings ......................... 2048 + memory_centric_tiled_linear ..................... False + merge_file ...................................... /gpfswork/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed-tr8b-104B/data/gpt2-merges.txt + micro_batch_size ................................ 1 + min_loss_scale .................................. 1.0 + min_lr .......................................... 6e-06 + mmap_warmup ..................................... False + no_load_optim ................................... None + no_load_rng ..................................... None + no_save_optim ................................... None + no_save_rng ..................................... None + num_attention_heads ............................. 80 + num_channels .................................... 3 + num_classes ..................................... 1000 + num_layers ...................................... 64 + num_layers_per_virtual_pipeline_stage ........... None + num_workers ..................................... 2 + onnx_safe ....................................... None + openai_gelu ..................................... False + optimizer ....................................... adam + override_lr_scheduler ........................... False + params_dtype .................................... torch.float16 + partition_activations ........................... False + patch_dim ....................................... 16 + pipeline_model_parallel_size .................... 32 + position_embedding_type ......................... PositionEmbeddingType.absolute + profile_backward ................................ False + query_in_block_prob ............................. 0.1 + rampup_batch_size ............................... None + rank ............................................ 0 + remote_device ................................... none + reset_attention_mask ............................ False + reset_position_ids .............................. False + retriever_report_topk_accuracies ................ [] + retriever_score_scaling ......................... False + retriever_seq_length ............................ 256 + sample_rate ..................................... 1.0 + save ............................................ /gpfsscratch/rech/six/commun/checkpoints/tr8b-104B/checkpoints + save_interval ................................... 300 + scatter_gather_tensors_in_pipeline .............. True + scattered_embeddings ............................ False + seed ............................................ 43 + seq_length ...................................... 2048 + sgd_momentum .................................... 0.9 + short_seq_prob .................................. 0.1 + split ........................................... 949,50,1 + split_transformers .............................. False + synchronize_each_layer .......................... False + tensor_model_parallel_size ...................... 4 + tensorboard_dir ................................. /gpfsscratch/rech/six/commun/checkpoints/tr8b-104B/tr8b-104B-logs/tensorboard + tensorboard_log_interval ........................ 1 + tensorboard_queue_size .......................... 5 + tile_factor ..................................... 1 + titles_data_path ................................ None + tokenizer_name_or_path .......................... None + tokenizer_type .................................. GPT2BPETokenizer + train_iters ..................................... None + train_samples ................................... 600000000 + train_tokens .................................... 300000000000 + use_bnb_optimizer ............................... False + use_checkpoint_lr_scheduler ..................... False + use_contiguous_buffers_in_ddp ................... False + use_cpu_initialization .......................... None + use_one_sent_docs ............................... False + use_pin_memory .................................. False + virtual_pipeline_model_parallel_size ............ None + vocab_extra_ids ................................. 0 + vocab_file ...................................... /gpfswork/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed-tr8b-104B/data/gpt2-vocab.json + weight_decay .................................... 0.1 + world_size ...................................... 128 + zero_allgather_bucket_size ...................... 0.0 + zero_contigious_gradients ....................... False + zero_reduce_bucket_size ......................... 0.0 + zero_reduce_scatter ............................. False + zero_stage ...................................... 1 +-------------------- end of arguments --------------------- +setting number of micro-batches to constant 2048 +> building GPT2BPETokenizer tokenizer ... +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main ******** Git info for Megatron: git_hash=829cefd git_branch=main **** + +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.-------------------------------------------------- + +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** + > padded vocab (size: 50257) with 431 dummy tokens (new size: 50688) +> initializing torch distributed ... +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main ******** Git info for Megatron: git_hash=829cefd git_branch=main **** + +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +DeepSpeed general environment info: +torch install path DeepSpeed general environment info:............... +torch install path['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] + ...............torch version .................... 1.8.1 +torch cuda version['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +............... 11.1torch version + nvcc version.................... .....................1.8.1 +11.2 +deepspeed install pathtorch cuda version .......................... 11.1 +['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed']nvcc version + deepspeed info..................... ...................11.2 +0.5.5+57dee5a, 57dee5a, pp_deadlock_fixdeepspeed install path + deepspeed wheel compiled w............ ...... torch 1.8, cuda 11.1 +['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+57dee5a, 57dee5a, pp_deadlock_fix +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +> setting tensorboard ... +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +**** Git info for Megatron: git_hash=829cefd git_branch=main **** +> initializing tensor model parallel with size 4 +> initializing pipeline model parallel with size 32 +> setting random seeds to 43 ... +[2021-10-22 06:34:46,025] [INFO] [checkpointing.py:226:model_parallel_cuda_manual_seed] > initializing model parallel cuda seeds on global rank 0, model parallel rank 0, and data parallel rank 0 with model parallel seed: 2761 and data parallel seed: 43 +> compiling dataset index builder ... +make: Entering directory '/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed/megatron/data' +g++ -O3 -Wall -shared -std=c++11 -fPIC -fdiagnostics-color -I/gpfswork/rech/six/commun/conda/cutting-edge/include/python3.8 -I/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/pybind11/include helpers.cpp -o helpers.cpython-38-x86_64-linux-gnu.so +make: Leaving directory '/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed/megatron/data' +>>> done with dataset index builder. Compilation time: 6.052 seconds +> compiling and loading fused kernels ... +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +Detected CUDA files, patching ldflags +Emitting ninja build file /gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed/megatron/fused_kernels/build/build.ninja... +Building extension module scaled_upper_triang_masked_softmax_cuda... +Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) +[1/3] c++ -MMD -MF scaled_upper_triang_masked_softmax.o.d -DTORCH_EXTENSION_NAME=scaled_upper_triang_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/include -isystem /gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/include/TH -isystem /gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/include/THC -isystem /gpfslocalsys/cuda/11.2/include -isystem /gpfswork/rech/six/commun/conda/cutting-edge/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -c /gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax.cpp -o scaled_upper_triang_masked_softmax.o +[2/3] /gpfslocalsys/cuda/11.2/bin/nvcc --generate-dependencies-with-compile --dependency-output scaled_upper_triang_masked_softmax_cuda.cuda.o.d -DTORCH_EXTENSION_NAME=scaled_upper_triang_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/include -isystem /gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/include/TH -isystem /gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/include/THC -isystem /gpfslocalsys/cuda/11.2/include -isystem /gpfswork/rech/six/commun/conda/cutting-edge/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_70,code=compute_70 -gencode=arch=compute_70,code=sm_70 --compiler-options '-fPIC' -O3 -gencode arch=compute_70,code=sm_70 --use_fast_math -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda -gencode arch=compute_80,code=sm_80 -std=c++14 -c /gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax_cuda.cu -o scaled_upper_triang_masked_softmax_cuda.cuda.o +[3/3] c++ scaled_upper_triang_masked_softmax.o scaled_upper_triang_masked_softmax_cuda.cuda.o -shared -L/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda_cu -ltorch_cuda_cpp -ltorch -ltorch_python -L/gpfslocalsys/cuda/11.2/lib64 -lcudart -o scaled_upper_triang_masked_softmax_cuda.so +Loading extension module scaled_upper_triang_masked_softmax_cuda... +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +Detected CUDA files, patching ldflags +Emitting ninja build file /gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed/megatron/fused_kernels/build/build.ninja... +Building extension module scaled_masked_softmax_cuda... +Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) +[1/3] c++ -MMD -MF scaled_masked_softmax.o.d -DTORCH_EXTENSION_NAME=scaled_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/include -isystem /gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/include/TH -isystem /gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/include/THC -isystem /gpfslocalsys/cuda/11.2/include -isystem /gpfswork/rech/six/commun/conda/cutting-edge/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -c /gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax.cpp -o scaled_masked_softmax.o +[2/3] /gpfslocalsys/cuda/11.2/bin/nvcc --generate-dependencies-with-compile --dependency-output scaled_masked_softmax_cuda.cuda.o.d -DTORCH_EXTENSION_NAME=scaled_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/include -isystem /gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/include/TH -isystem /gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/include/THC -isystem /gpfslocalsys/cuda/11.2/include -isystem /gpfswork/rech/six/commun/conda/cutting-edge/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_70,code=compute_70 -gencode=arch=compute_70,code=sm_70 --compiler-options '-fPIC' -O3 -gencode arch=compute_70,code=sm_70 --use_fast_math -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda -gencode arch=compute_80,code=sm_80 -std=c++14 -c /gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_cuda.cu -o scaled_masked_softmax_cuda.cuda.o +/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax.h(320): warning: variable "batch_count" was declared but never referenced + +/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax.h(320): warning: variable "batch_count" was declared but never referenced + +[3/3] c++ scaled_masked_softmax.o scaled_masked_softmax_cuda.cuda.o -shared -L/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda_cu -ltorch_cuda_cpp -ltorch -ltorch_python -L/gpfslocalsys/cuda/11.2/lib64 -lcudart -o scaled_masked_softmax_cuda.so +Loading extension module scaled_masked_softmax_cuda... +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +Detected CUDA files, patching ldflags +Emitting ninja build file /gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed/megatron/fused_kernels/build/build.ninja... +Building extension module fused_mix_prec_layer_norm_cuda... +Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) +[1/3] c++ -MMD -MF layer_norm_cuda.o.d -DTORCH_EXTENSION_NAME=fused_mix_prec_layer_norm_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/include -isystem /gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/include/TH -isystem /gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/include/THC -isystem /gpfslocalsys/cuda/11.2/include -isystem /gpfswork/rech/six/commun/conda/cutting-edge/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -c /gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed/megatron/fused_kernels/layer_norm_cuda.cpp -o layer_norm_cuda.o +[2/3] /gpfslocalsys/cuda/11.2/bin/nvcc --generate-dependencies-with-compile --dependency-output layer_norm_cuda_kernel.cuda.o.d -DTORCH_EXTENSION_NAME=fused_mix_prec_layer_norm_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/include -isystem /gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/include/TH -isystem /gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/include/THC -isystem /gpfslocalsys/cuda/11.2/include -isystem /gpfswork/rech/six/commun/conda/cutting-edge/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_70,code=compute_70 -gencode=arch=compute_70,code=sm_70 --compiler-options '-fPIC' -O3 -gencode arch=compute_70,code=sm_70 --use_fast_math -maxrregcount=50 -gencode arch=compute_80,code=sm_80 -std=c++14 -c /gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed/megatron/fused_kernels/layer_norm_cuda_kernel.cu -o layer_norm_cuda_kernel.cuda.o +[3/3] c++ layer_norm_cuda.o layer_norm_cuda_kernel.cuda.o -shared -L/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda_cu -ltorch_cuda_cpp -ltorch -ltorch_python -L/gpfslocalsys/cuda/11.2/lib64 -lcudart -o fused_mix_prec_layer_norm_cuda.so +Loading extension module fused_mix_prec_layer_norm_cuda... +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +>>> done with compiling and loading fused kernels. Compilation time: 162.654 seconds +time to initialize megatron (seconds): 142.840 +[after megatron is initialized] datetime: 2021-10-22 06:37:34 +building GPT model ... +[2021-10-22 06:37:34,906] [INFO] [utils.py:806:see_memory_usage] Before Building Model +[2021-10-22 06:37:34,907] [INFO] [utils.py:807:see_memory_usage] MA 0.0 GB Max_MA 0.0 GB CA 0.0 GB Max_CA 0 GB +[2021-10-22 06:37:34,908] [INFO] [utils.py:815:see_memory_usage] CPU Virtual Memory: used = 39.18 GB, percent = 20.9% +SEED_LAYERS=False BASE_SEED=1234 SEED_FN=None +Using topology: {ProcessCoord(pipe=0, data=0, model=0): 0, ProcessCoord(pipe=0, data=0, model=1): 1, ProcessCoord(pipe=0, data=0, model=2): 2, ProcessCoord(pipe=0, data=0, model=3): 3, ProcessCoord(pipe=1, data=0, model=0): 4, ProcessCoord(pipe=1, data=0, model=1): 5, ProcessCoord(pipe=1, data=0, model=2): 6, ProcessCoord(pipe=1, data=0, model=3): 7, ProcessCoord(pipe=2, data=0, model=0): 8, ProcessCoord(pipe=2, data=0, model=1): 9, ProcessCoord(pipe=2, data=0, model=2): 10, ProcessCoord(pipe=2, data=0, model=3): 11, ProcessCoord(pipe=3, data=0, model=0): 12, ProcessCoord(pipe=3, data=0, model=1): 13, ProcessCoord(pipe=3, data=0, model=2): 14, ProcessCoord(pipe=3, data=0, model=3): 15, ProcessCoord(pipe=4, data=0, model=0): 16, ProcessCoord(pipe=4, data=0, model=1): 17, ProcessCoord(pipe=4, data=0, model=2): 18, ProcessCoord(pipe=4, data=0, model=3): 19, ProcessCoord(pipe=5, data=0, model=0): 20, ProcessCoord(pipe=5, data=0, model=1): 21, ProcessCoord(pipe=5, data=0, model=2): 22, ProcessCoord(pipe=5, data=0, model=3): 23, ProcessCoord(pipe=6, data=0, model=0): 24, ProcessCoord(pipe=6, data=0, model=1): 25, ProcessCoord(pipe=6, data=0, model=2): 26, ProcessCoord(pipe=6, data=0, model=3): 27, ProcessCoord(pipe=7, data=0, model=0): 28, ProcessCoord(pipe=7, data=0, model=1): 29, ProcessCoord(pipe=7, data=0, model=2): 30, ProcessCoord(pipe=7, data=0, model=3): 31, ProcessCoord(pipe=8, data=0, model=0): 32, ProcessCoord(pipe=8, data=0, model=1): 33, ProcessCoord(pipe=8, data=0, model=2): 34, ProcessCoord(pipe=8, data=0, model=3): 35, ProcessCoord(pipe=9, data=0, model=0): 36, ProcessCoord(pipe=9, data=0, model=1): 37, ProcessCoord(pipe=9, data=0, model=2): 38, ProcessCoord(pipe=9, data=0, model=3): 39, ProcessCoord(pipe=10, data=0, model=0): 40, ProcessCoord(pipe=10, data=0, model=1): 41, ProcessCoord(pipe=10, data=0, model=2): 42, ProcessCoord(pipe=10, data=0, model=3): 43, ProcessCoord(pipe=11, data=0, model=0): 44, ProcessCoord(pipe=11, data=0, model=1): 45, ProcessCoord(pipe=11, data=0, model=2): 46, ProcessCoord(pipe=11, data=0, model=3): 47, ProcessCoord(pipe=12, data=0, model=0): 48, ProcessCoord(pipe=12, data=0, model=1): 49, ProcessCoord(pipe=12, data=0, model=2): 50, ProcessCoord(pipe=12, data=0, model=3): 51, ProcessCoord(pipe=13, data=0, model=0): 52, ProcessCoord(pipe=13, data=0, model=1): 53, ProcessCoord(pipe=13, data=0, model=2): 54, ProcessCoord(pipe=13, data=0, model=3): 55, ProcessCoord(pipe=14, data=0, model=0): 56, ProcessCoord(pipe=14, data=0, model=1): 57, ProcessCoord(pipe=14, data=0, model=2): 58, ProcessCoord(pipe=14, data=0, model=3): 59, ProcessCoord(pipe=15, data=0, model=0): 60, ProcessCoord(pipe=15, data=0, model=1): 61, ProcessCoord(pipe=15, data=0, model=2): 62, ProcessCoord(pipe=15, data=0, model=3): 63, ProcessCoord(pipe=16, data=0, model=0): 64, ProcessCoord(pipe=16, data=0, model=1): 65, ProcessCoord(pipe=16, data=0, model=2): 66, ProcessCoord(pipe=16, data=0, model=3): 67, ProcessCoord(pipe=17, data=0, model=0): 68, ProcessCoord(pipe=17, data=0, model=1): 69, ProcessCoord(pipe=17, data=0, model=2): 70, ProcessCoord(pipe=17, data=0, model=3): 71, ProcessCoord(pipe=18, data=0, model=0): 72, ProcessCoord(pipe=18, data=0, model=1): 73, ProcessCoord(pipe=18, data=0, model=2): 74, ProcessCoord(pipe=18, data=0, model=3): 75, ProcessCoord(pipe=19, data=0, model=0): 76, ProcessCoord(pipe=19, data=0, model=1): 77, ProcessCoord(pipe=19, data=0, model=2): 78, ProcessCoord(pipe=19, data=0, model=3): 79, ProcessCoord(pipe=20, data=0, model=0): 80, ProcessCoord(pipe=20, data=0, model=1): 81, ProcessCoord(pipe=20, data=0, model=2): 82, ProcessCoord(pipe=20, data=0, model=3): 83, ProcessCoord(pipe=21, data=0, model=0): 84, ProcessCoord(pipe=21, data=0, model=1): 85, ProcessCoord(pipe=21, data=0, model=2): 86, ProcessCoord(pipe=21, data=0, model=3): 87, ProcessCoord(pipe=22, data=0, model=0): 88, ProcessCoord(pipe=22, data=0, model=1): 89, ProcessCoord(pipe=22, data=0, model=2): 90, ProcessCoord(pipe=22, data=0, model=3): 91, ProcessCoord(pipe=23, data=0, model=0): 92, ProcessCoord(pipe=23, data=0, model=1): 93, ProcessCoord(pipe=23, data=0, model=2): 94, ProcessCoord(pipe=23, data=0, model=3): 95, ProcessCoord(pipe=24, data=0, model=0): 96, ProcessCoord(pipe=24, data=0, model=1): 97, ProcessCoord(pipe=24, data=0, model=2): 98, ProcessCoord(pipe=24, data=0, model=3): 99, ProcessCoord(pipe=25, data=0, model=0): 100, ProcessCoord(pipe=25, data=0, model=1): 101, ProcessCoord(pipe=25, data=0, model=2): 102, ProcessCoord(pipe=25, data=0, model=3): 103, ProcessCoord(pipe=26, data=0, model=0): 104, ProcessCoord(pipe=26, data=0, model=1): 105, ProcessCoord(pipe=26, data=0, model=2): 106, ProcessCoord(pipe=26, data=0, model=3): 107, ProcessCoord(pipe=27, data=0, model=0): 108, ProcessCoord(pipe=27, data=0, model=1): 109, ProcessCoord(pipe=27, data=0, model=2): 110, ProcessCoord(pipe=27, data=0, model=3): 111, ProcessCoord(pipe=28, data=0, model=0): 112, ProcessCoord(pipe=28, data=0, model=1): 113, ProcessCoord(pipe=28, data=0, model=2): 114, ProcessCoord(pipe=28, data=0, model=3): 115, ProcessCoord(pipe=29, data=0, model=0): 116, ProcessCoord(pipe=29, data=0, model=1): 117, ProcessCoord(pipe=29, data=0, model=2): 118, ProcessCoord(pipe=29, data=0, model=3): 119, ProcessCoord(pipe=30, data=0, model=0): 120, ProcessCoord(pipe=30, data=0, model=1): 121, ProcessCoord(pipe=30, data=0, model=2): 122, ProcessCoord(pipe=30, data=0, model=3): 123, ProcessCoord(pipe=31, data=0, model=0): 124, ProcessCoord(pipe=31, data=0, model=1): 125, ProcessCoord(pipe=31, data=0, model=2): 126, ProcessCoord(pipe=31, data=0, model=3): 127} +[2021-10-22 06:37:36,589] [INFO] [module.py:365:_partition_layers] Partitioning pipeline stages with method type:transformer +stage=0 layers=5 + 0: _to_float16 + 1: EmbeddingPipe + 2: + 3: ParallelTransformerLayerPipe + 4: ParallelTransformerLayerPipe +stage=1 layers=2 + 5: ParallelTransformerLayerPipe + 6: ParallelTransformerLayerPipe +stage=2 layers=2 + 7: ParallelTransformerLayerPipe + 8: ParallelTransformerLayerPipe +stage=3 layers=2 + 9: ParallelTransformerLayerPipe + 10: ParallelTransformerLayerPipe +stage=4 layers=2 + 11: ParallelTransformerLayerPipe + 12: ParallelTransformerLayerPipe +stage=5 layers=2 + 13: ParallelTransformerLayerPipe + 14: ParallelTransformerLayerPipe +stage=6 layers=2 + 15: ParallelTransformerLayerPipe + 16: ParallelTransformerLayerPipe +stage=7 layers=2 + 17: ParallelTransformerLayerPipe + 18: ParallelTransformerLayerPipe +stage=8 layers=2 + 19: ParallelTransformerLayerPipe + 20: ParallelTransformerLayerPipe +stage=9 layers=2 + 21: ParallelTransformerLayerPipe + 22: ParallelTransformerLayerPipe +stage=10 layers=2 + 23: ParallelTransformerLayerPipe + 24: ParallelTransformerLayerPipe +stage=11 layers=2 + 25: ParallelTransformerLayerPipe + 26: ParallelTransformerLayerPipe +stage=12 layers=2 + 27: ParallelTransformerLayerPipe + 28: ParallelTransformerLayerPipe +stage=13 layers=2 + 29: ParallelTransformerLayerPipe + 30: ParallelTransformerLayerPipe +stage=14 layers=2 + 31: ParallelTransformerLayerPipe + 32: ParallelTransformerLayerPipe +stage=15 layers=2 + 33: ParallelTransformerLayerPipe + 34: ParallelTransformerLayerPipe +stage=16 layers=2 + 35: ParallelTransformerLayerPipe + 36: ParallelTransformerLayerPipe +stage=17 layers=2 + 37: ParallelTransformerLayerPipe + 38: ParallelTransformerLayerPipe +stage=18 layers=2 + 39: ParallelTransformerLayerPipe + 40: ParallelTransformerLayerPipe +stage=19 layers=2 + 41: ParallelTransformerLayerPipe + 42: ParallelTransformerLayerPipe +stage=20 layers=2 + 43: ParallelTransformerLayerPipe + 44: ParallelTransformerLayerPipe +stage=21 layers=2 + 45: ParallelTransformerLayerPipe + 46: ParallelTransformerLayerPipe +stage=22 layers=2 + 47: ParallelTransformerLayerPipe + 48: ParallelTransformerLayerPipe +stage=23 layers=2 + 49: ParallelTransformerLayerPipe + 50: ParallelTransformerLayerPipe +stage=24 layers=2 + 51: ParallelTransformerLayerPipe + 52: ParallelTransformerLayerPipe +stage=25 layers=2 + 53: ParallelTransformerLayerPipe + 54: ParallelTransformerLayerPipe +stage=26 layers=2 + 55: ParallelTransformerLayerPipe + 56: ParallelTransformerLayerPipe +stage=27 layers=2 + 57: ParallelTransformerLayerPipe + 58: ParallelTransformerLayerPipe +stage=28 layers=2 + 59: ParallelTransformerLayerPipe + 60: ParallelTransformerLayerPipe +stage=29 layers=2 + 61: ParallelTransformerLayerPipe + 62: ParallelTransformerLayerPipe +stage=30 layers=2 + 63: ParallelTransformerLayerPipe + 64: ParallelTransformerLayerPipe +stage=31 layers=6 + 65: ParallelTransformerLayerPipe + 66: ParallelTransformerLayerPipe + 67: + 68: MixedFusedLayerNorm + 69: EmbeddingPipe + 70: float16_to_fp32 + loss: CrossEntropy +Traceback (most recent call last): + File "/gpfswork/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed-tr8b-104B/pretrain_gpt.py", line 237, in + pretrain(train_valid_test_datasets_provider, model_provider, forward_step, + File "/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed/megatron/training.py", line 134, in pretrain + model, optimizer, lr_scheduler = setup_model_and_optimizer(model_provider) + File "/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed/megatron/training.py", line 342, in setup_model_and_optimizer + model = get_model(model_provider_func) + File "/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed/megatron/training.py", line 240, in get_model + model = model_provider_func( + File "/gpfswork/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed-tr8b-104B/pretrain_gpt.py", line 68, in model_provider + model = GPTModelPipe( + File "/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed/megatron/model/gpt_model.py", line 279, in __init__ + super().__init__(layers=self.specs, + File "/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed/runtime/pipe/module.py", line 200, in __init__ + self._build() + File "/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed/runtime/pipe/module.py", line 248, in _build + module = layer.build() + File "/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed/runtime/pipe/module.py", line 70, in build + return self.typename(*self.module_args, **self.module_kwargs) + File "/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed/megatron/model/transformer.py", line 441, in __init__ + self.self_attention = ParallelAttention( + File "/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed/megatron/model/transformer.py", line 151, in __init__ + self.query_key_value = mpu.ColumnParallelLinear( + File "/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed/megatron/mpu/layers.py", line 259, in __init__ + self.weight = Parameter(torch.empty( +RuntimeError: CUDA out of memory. Tried to allocate 194.00 MiB (GPU 2; 31.75 GiB total capacity; 12.00 MiB already allocated; 157.00 MiB free; 18.00 MiB reserved in total by PyTorch) +Killing subprocess 2924509 +Killing subprocess 2924510 +Killing subprocess 2924511 +Killing subprocess 2924512 +Traceback (most recent call last): + File "/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/runpy.py", line 194, in _run_module_as_main + return _run_code(code, main_globals, None, + File "/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/runpy.py", line 87, in _run_code + exec(code, run_globals) + File "/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/distributed/launch.py", line 340, in + main() + File "/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/distributed/launch.py", line 326, in main + sigkill_handler(signal.SIGTERM, None) # not coming back + File "/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/distributed/launch.py", line 301, in sigkill_handler + raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd) +subprocess.CalledProcessError: Command '['/gpfswork/rech/six/commun/conda/cutting-edge/bin/python', '-u', '/gpfswork/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed-tr8b-104B/pretrain_gpt.py', '--local_rank=3', '--tensor-model-parallel-size', '4', '--pipeline-model-parallel-size', '32', '--num-layers', '64', '--hidden-size', '11600', '--num-attention-heads', '80', '--seq-length', '2048', '--max-position-embeddings', '2048', '--micro-batch-size', '1', '--global-batch-size', '2048', '--train-samples', '600_000_000', '--train-tokens', '300_000_000_000', '--vocab-file', '/gpfswork/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed-tr8b-104B/data/gpt2-vocab.json', '--merge-file', '/gpfswork/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed-tr8b-104B/data/gpt2-merges.txt', '--loss-scale', '12', '--fp16', '--checkpoint-activations', '--seed', '43', '--optimizer', 'adam', '--adam-beta1', '0.9', '--adam-beta2', '0.95', '--adam-eps', '1e-8', '--lr', '6e-5', '--min-lr', '6e-6', '--lr-warmup-samples', '216_320', '--lr-decay-tokens', '260000000000', '--lr-decay-style', 'cosine', '--clip-grad', '1.0', '--weight-decay', '1e-1', '--exit-duration-in-mins', '55', '--log-interval', '1', '--save-interval', '300', '--eval-interval', '1000', '--eval-iters', '5', '--tensorboard-dir', '/gpfsscratch/rech/six/commun/checkpoints/tr8b-104B/tr8b-104B-logs/tensorboard', '--tensorboard-queue-size', '5', '--log-timers-to-tensorboard', '--log-batch-size-to-tensorboard', '--log-validation-ppl-to-tensorboard', '--save', '/gpfsscratch/rech/six/commun/checkpoints/tr8b-104B/checkpoints', '--load', '/gpfsscratch/rech/six/commun/checkpoints/tr8b-104B/checkpoints', '--data-path', '/gpfswork/rech/six/commun/datasets-custom/oscar-en/meg-gpt2_text_document', '--data-impl', 'mmap', '--split', '949,50,1', '--distributed-backend', 'nccl', '--deepspeed', '--deepspeed_config', './ds_config.1645939.json', '--zero-stage', '1', '--deepspeed-activation-checkpointing']' returned non-zero exit status 1. +srun: error: r8i2n4: task 22: Exited with exit code 1 +srun: Terminating job step 1645939.0 +slurmstepd: error: *** STEP 1645939.0 ON r6i3n3 CANCELLED AT 2021-10-22T06:37:39 *** +Killing subprocess 2330903 +Killing subprocess 2330904 +Killing subprocess 2330905 +Killing subprocess 2330906 +Killing subprocess 2151458 +Main process received SIGTERM, exiting +Killing subprocess 1978795 +Killing subprocess 2151459 +Killing subprocess 2151460 +Killing subprocess 2151462 +Main process received SIGTERM, exiting +Killing subprocess 1978796 +Killing subprocess 195931 +Killing subprocess 1978797 +Killing subprocess 1978798 +Main process received SIGTERM, exiting +Killing subprocess 2190049 +Killing subprocess 195932 +Killing subprocess 2190050 +Killing subprocess 2190051 +Killing subprocess 2190053 +Killing subprocess 195933 +Killing subprocess 195934 +Main process received SIGTERM, exiting +Main process received SIGTERM, exiting +Killing subprocess 518488 +Killing subprocess 518489 +Killing subprocess 518490 +Killing subprocess 518491 +Main process received SIGTERM, exiting +Killing subprocess 2778649 +Killing subprocess 604324 +Killing subprocess 3025916 +Killing subprocess 604325 +Killing subprocess 2778650 +Killing subprocess 3080690 +Killing subprocess 767148 +Killing subprocess 3025917 +Killing subprocess 2778651 +Killing subprocess 2789903 +Killing subprocess 3080691 +Killing subprocess 1230242 +Killing subprocess 3025918 +Killing subprocess 604326 +Killing subprocess 767149 +Killing subprocess 2174109 +Killing subprocess 2789904 +Killing subprocess 610053 +Killing subprocess 2778652 +Killing subprocess 1230243 +Killing subprocess 2174110 +Killing subprocess 767150 +Killing subprocess 2789905 +Killing subprocess 610054 +Killing subprocess 604327 +Killing subprocess 2171255 +Killing subprocess 1230244 +Killing subprocess 767151 +Main process received SIGTERM, exiting +Killing subprocess 3080692 +Killing subprocess 610055 +Killing subprocess 3080693 +Killing subprocess 1000808 +Killing subprocess 2171256 +Main process received SIGTERM, exiting +Killing subprocess 1230245 +Killing subprocess 3025919 +Main process received SIGTERM, exiting +Killing subprocess 610056 +Killing subprocess 1000809 +Killing subprocess 2174111 +Main process received SIGTERM, exiting +Killing subprocess 2174112 +Main process received SIGTERM, exiting +Main process received SIGTERM, exiting +Main process received SIGTERM, exiting +Killing subprocess 2789907 +Killing subprocess 2171257 +Main process received SIGTERM, exiting +Killing subprocess 2171258 +Killing subprocess 1000810 +Main process received SIGTERM, exiting +Main process received SIGTERM, exiting +Killing subprocess 1000811 +Main process received SIGTERM, exiting +Killing subprocess 3527695 +Killing subprocess 3527696 +Killing subprocess 3527697 +Killing subprocess 3527698 +Main process received SIGTERM, exiting +Killing subprocess 1931279 +Killing subprocess 1931280 +Killing subprocess 1931281 +Killing subprocess 1931282 +Main process received SIGTERM, exiting +Killing subprocess 865870 +Killing subprocess 865871 +Killing subprocess 865872 +Killing subprocess 865873 +Main process received SIGTERM, exiting +Killing subprocess 966479 +Killing subprocess 966480 +Killing subprocess 966481 +Killing subprocess 966482 +Main process received SIGTERM, exiting +Killing subprocess 2004240 +Killing subprocess 2399152 +Killing subprocess 2004241 +Killing subprocess 2399153 +Killing subprocess 2004242 +Killing subprocess 2004243 +Main process received SIGTERM, exiting +Killing subprocess 2399154 +Killing subprocess 2278198 +Killing subprocess 2278199 +Killing subprocess 2399155 +Main process received SIGTERM, exiting +Killing subprocess 2278200 +Killing subprocess 2278202 +Main process received SIGTERM, exiting +Killing subprocess 1900979 +Killing subprocess 1900980 +Killing subprocess 1900981 +Killing subprocess 1900982 +Main process received SIGTERM, exiting +Killing subprocess 3130893 +Killing subprocess 3130894 +Killing subprocess 3130895 +Killing subprocess 3130896 +Main process received SIGTERM, exiting +Killing subprocess 2132490 +Killing subprocess 2132491 +Killing subprocess 2132492 +Killing subprocess 2132494 +Main process received SIGTERM, exiting +Killing subprocess 2027567 +Killing subprocess 2027568 +Killing subprocess 1923123 +Killing subprocess 1923124 +Killing subprocess 1923125 +Killing subprocess 2027569 +Killing subprocess 2027570 +Main process received SIGTERM, exiting +Killing subprocess 1923126 +Main process received SIGTERM, exiting +Killing subprocess 2024004 +Killing subprocess 2024005 +Killing subprocess 2024006 +Killing subprocess 2024007 +Main process received SIGTERM, exiting +Killing subprocess 1934525 +Killing subprocess 1934526 +Killing subprocess 1934527 +Killing subprocess 1934528 +Main process received SIGTERM, exiting +srun: error: r6i4n1: task 1: Exited with exit code 1 +srun: error: r8i7n8: task 27: Exited with exit code 1 +srun: error: r6i4n4: task 4: Exited with exit code 1 +srun: error: r9i0n0: task 28: Exited with exit code 1 +srun: error: r9i0n1: task 29: Exited with exit code 1 +srun: error: r6i6n1: task 5: Exited with exit code 1 +srun: error: r6i4n2: task 2: Exited with exit code 1 +srun: error: r6i4n3: task 3: Exited with exit code 1 +srun: error: r8i2n5: task 23: Exited with exit code 1 +srun: error: r8i2n7: task 25: Exited with exit code 1 +srun: error: r8i2n2: task 20: Exited with exit code 1 +srun: error: r8i2n3: task 21: Exited with exit code 1 +srun: error: r8i2n8: task 26: Exited with exit code 1 +srun: error: r8i2n0: task 18: Exited with exit code 1 +srun: error: r8i1n2: task 11: Exited with exit code 1 +srun: error: r7i1n4: task 6: Exited with exit code 1 +srun: error: r8i2n1: task 19: Exited with exit code 1 +srun: error: r9i0n3: task 30: Exited with exit code 1 +srun: error: r8i2n6: task 24: Exited with exit code 1 +srun: error: r8i1n4: task 13: Exited with exit code 1 +srun: error: r7i1n6: task 7: Exited with exit code 1 +srun: error: r8i1n3: task 12: Exited with exit code 1 +srun: error: r6i3n3: task 0: Exited with exit code 1 +srun: error: r7i4n3: task 8: Exited with exit code 1 +srun: error: r8i1n7: task 16: Exited with exit code 1 +srun: error: r8i1n6: task 15: Exited with exit code 1 +srun: error: r8i1n8: task 17: Exited with exit code 1 +srun: error: r7i6n5: task 9: Exited with exit code 1 +srun: error: r8i0n7: task 10: Exited with exit code 1 +srun: error: r9i6n0: task 31: Exited with exit code 1 +srun: error: r8i1n5: task 14: Exited with exit code 1