diff --git "a/logs/main_log.txt" "b/logs/main_log.txt" --- "a/logs/main_log.txt" +++ "b/logs/main_log.txt" @@ -48830,3 +48830,5896 @@ srun: error: r7i6n5: task 9: Exited with exit code 1 srun: error: r8i0n7: task 10: Exited with exit code 1 srun: error: r9i6n0: task 31: Exited with exit code 1 srun: error: r8i1n5: task 14: Exited with exit code 1 +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + + +DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op report + + +DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op report---------------------------------------------------------------------------------------------------- + + + +--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +---------------------------------------------------------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- + + + +JIT compiled ops requires ninjaJIT compiled ops requires ninjaJIT compiled ops requires ninja + +-------------------------------------------------- + +JIT compiled ops requires ninja +ninjaninja .................................... [OKAY][OKAY] + +---------------------------------------------------------------------------------------------------- + +op nameop name ................................ installedinstalled .... compatiblecompatible + +---------------------------------------------------------------------------------------------------- + +cpu_adamcpu_adam .............................. [NO][NO] .............. [OKAY][OKAY] + +fused_adamfused_adam .......................... [NO][NO] .............. [OKAY][OKAY] + +fused_lamb .............fused_lamb [NO]............. .......[NO] [OKAY]....... + [OKAY] +sparse_attn sparse_attn............ ............[NO] [NO]....... ....... [OKAY][OKAY] + +transformertransformer ........................ [NO][NO] .............. [OKAY][OKAY] + +stochastic_transformerstochastic_transformer .. [NO][NO] .............. [OKAY][OKAY] + +ninja .................. [OKAY] +-------------------------------------------------- +op name ninja................ ..................installed [OKAY] +.. --------------------------------------------------compatible + +op name-------------------------------------------------- ................ + installed .. compatible +-------------------------------------------------- +cpu_adam ............... [NO] ....... [OKAY] +cpu_adam ............... [NO] ....... [OKAY] +fused_adam ............. [NO] ....... [OKAY] +fused_lamb ............. [NO] fused_adam....... .............[OKAY] +[NO] ....... [OKAY] +fused_lamb ............. [NO] .......sparse_attn ............ [NO][OKAY] +....... [OKAY] +transformer ............ [NO] ....... [OKAY] +stochastic_transformer . [NO]sparse_attn ................... [OKAY][NO] + ....... [OKAY] +transformer ............ [NO] ....... [OKAY] +stochastic_transformer . [NO] ....... [OKAY] +------------------------------------------------------------------------------------------------------------------------------------------------------ + + +DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op report + + +---------------------------------------------------------------------------------------------------- +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +---------------------------------------------------------------------------------------------------- + +JIT compiled ops requires ninjaJIT compiled ops requires ninja + +-------------------------------------------------- + +DeepSpeed C++/CUDA extension op reportJIT compiled ops requires ninja + +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +ninjaninjaninjaninja ...................................................... .................. [OKAY][OKAY] [OKAY] + +[OKAY] +-------------------------------------------------- +-------------------------------------------------- +-------------------------------------------------- +-------------------------------------------------- +op nameop name +op name ................op name................ ................................installed installed installed installed.. .. .. ..compatible compatible compatible + +compatible +---------------------------------------------------------------------------------------------------- +-------------------------------------------------- + +-------------------------------------------------- + +cpu_adamcpu_adam cpu_adamcpu_adam ............... ............... ..............................[NO] [NO][NO][NO]....... .......[OKAY].............. + [OKAY][OKAY] +[OKAY] + +fused_adam ............. fused_adam[NO]fused_adamfused_adam ....... ............. .......................... [OKAY] + [NO][NO][NO] fused_lamb .................................. [OKAY] [OKAY] +[NO][OKAY] + +....... fused_lamb[OKAY] fused_lamb............. +fused_lamb [NO].......................... .......[NO][NO] [OKAY] +.............. sparse_attn[OKAY][OKAY] + +............ [NO] ....... [OKAY] +sparse_attntransformer ........................ sparse_attn[NO][NO]sparse_attn ....... ................... ............[OKAY] [OKAY][NO] +[NO] + .......stochastic_transformer.......transformer [OKAY] ............. +[OKAY] [NO][NO] + transformer.............. [OKAY]transformer[OKAY] +............ + ............[NO] [NO].......stochastic_transformer [OKAY]....... + .[OKAY] +[NO]stochastic_transformer .......stochastic_transformer .[OKAY] +[NO]. .......[NO] [OKAY]....... + [OKAY] +------------------------------------------------------------------------------------------------------------------------------------------------------ + + +DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op report +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + + +JIT compiled ops requires ninja-------------------------------------------------- +-------------------------------------------------- +--------------------------------------------------JIT compiled ops requires ninja + + +JIT compiled ops requires ninjaDeepSpeed C++/CUDA extension op report + +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +------------------------------------------------------------------------------------------------------------------------------------------------------ + +DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op report + + +DeepSpeed C++/CUDA extension op report---------------------------------------------------------------------------------------------------- + + +--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.---------------------------------------------------------------------------------------------------- +-------------------------------------------------- + +--------------------------------------------------JIT compiled ops requires ninja + +JIT compiled ops requires ninjaJIT compiled ops requires ninja + +DeepSpeed C++/CUDA extension op report + +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +ninjaninjaninjaninja ........................................................................ [OKAY][OKAY][OKAY][OKAY] + + + +------------------------------------------------------------------------------------------------------------------------------------------------------ +-------------------------------------------------- + + +op name op nameop nameop name................ ................................................installed installedinstalled.. installed .. ..compatible ..compatible + +compatible--------------------------------------------------compatible-------------------------------------------------- + + + +-------------------------------------------------- +-------------------------------------------------- +cpu_adamcpu_adam cpu_adamcpu_adam.............................. ............... ...............[NO] [NO][NO] [NO] .............. .............. [OKAY][OKAY] + [OKAY] +[OKAY] + +fused_adam .............fused_adamfused_adam fused_adam [NO] .............................................. [OKAY][NO][NO][NO] + ..................... fused_lamb [OKAY][OKAY] + [OKAY]............. + + fused_lamb[NO] ....................fused_lambfused_lamb [NO] [OKAY] .......................... + ....... [NO][OKAY][NO] + .............. [OKAY][OKAY] + +sparse_attn ............ [NO] .......sparse_attn [OKAY] +............ sparse_attnsparse_attn[NO] transformer ........................................... [NO] [NO] [OKAY] [NO] +.............. transformer .......[OKAY] [OKAY] + +............[OKAY] +[NO]stochastic_transformer transformer........ transformer ............ [OKAY][NO] ............ + [NO] ....... [NO] ....... [OKAY]stochastic_transformer +....... [OKAY] +[OKAY]. + stochastic_transformer[NO] .......stochastic_transformer [OKAY]. + . [NO][NO] .............. [OKAY][OKAY] + +---------------------------------------------------------------------------------------------------- + +DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op report-------------------------------------------------- + + +----------------------------------------------------------------------------------------------------DeepSpeed C++/CUDA extension op report + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +-------------------------------------------------- +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +JIT compiled ops requires ninja +--------------------------------------------------JIT compiled ops requires ninja +-------------------------------------------------- + + +DeepSpeed C++/CUDA extension op reportJIT compiled ops requires ninja + +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +ninjaninjaninjaninja .................................... .................. .................. [OKAY][OKAY][OKAY] + + +[OKAY]------------------------------------------------------------------------------------------------------------------------------------------------------ + + + +op name-------------------------------------------------- +................op nameop name op nameinstalled ................ ................ .. installed ................ compatible + ..installedinstalled-------------------------------------------------- compatible .. +.. + compatible--------------------------------------------------compatible + + +-------------------------------------------------- +--------------------------------------------------cpu_adam + ............... [NO] .......cpu_adam [OKAY] cpu_adam +............... cpu_adam...............[NO] ...............[NO]....... [OKAY].......fused_adam +[NO] .............[OKAY] ....... +[NO] [OKAY]....... +[OKAY] +fused_adam .............fused_lamb [NO].............fused_adam .......[NO] ............. fused_adam.......[OKAY] + [NO][OKAY]............. fused_lamb + [NO].................... .......[NO][OKAY] +[OKAY]....... + [OKAY]fused_lamb +sparse_attn fused_lamb ......................... .............[NO][NO] [NO].............. [OKAY].......[OKAY] +sparse_attn + [OKAY]............ +transformer [NO]............ .......[NO] [OKAY]....... + [OKAY] +transformersparse_attn ............ ............stochastic_transformer[NO] sparse_attn [NO] ........ ............ .......[OKAY][NO] +[NO].......[OKAY] +[OKAY]stochastic_transformer....... + transformer [OKAY]. + ............[NO] transformer [NO] ....... ...................[OKAY] +[NO][OKAY] +....... [OKAY] +stochastic_transformer .stochastic_transformer [NO] ........ [OKAY][NO] + ....... [OKAY] +ninjaninjaninjaninja ........................................................................ [OKAY] [OKAY][OKAY] + +[OKAY] +------------------------------------------------------------------------------------------------------------------------------------------------------ + + + +--------------------------------------------------op nameop name + op name................op name................ ................installedinstalled................ installed .. .. installed..compatible +compatible..compatible +-------------------------------------------------- + +----------------------------------------------------------------------------------------------------compatible + + +-------------------------------------------------- +cpu_adam ...............cpu_adam cpu_adam[NO] cpu_adam...................... ............... ...............[OKAY][NO][NO] + ....... [NO].......[OKAY] +[OKAY]....... + fused_adam[OKAY] +............. [NO] ....... [OKAY] +fused_adam fused_adam.............fused_lamb .......................... [NO] fused_adam[NO]....... .............[NO] [OKAY] ....... +....... [NO] [OKAY] [OKAY] +fused_lamb....... + .............[OKAY]fused_lamb + [NO]............. fused_lamb....... [OKAY][NO] +............. sparse_attn ....... [NO] ............[OKAY] +.......[NO] [OKAY]....... +[OKAY]sparse_attn + ............ [NO] transformer....... ............[OKAY]sparse_attn +[NO] ...................transformer sparse_attn [NO] [OKAY] ............................... + [NO][NO]stochastic_transformer[OKAY] ....... +....... . [OKAY]transformer [OKAY] + +[NO]............ transformerstochastic_transformer.......[NO] ...................[OKAY] +. [OKAY][NO][NO] + .............. [OKAY] stochastic_transformer +[OKAY] +. [NO] .......stochastic_transformer [OKAY] +. [NO] ....... [OKAY] +------------------------------------------------------------------------------------------------------------------------------------------------------ + +DeepSpeed C++/CUDA extension op report +DeepSpeed C++/CUDA extension op report + +DeepSpeed C++/CUDA extension op report-------------------------------------------------- +---------------------------------------------------------------------------------------------------- +-------------------------------------------------- + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + + + +------------------------------------------------------------------------------------------------------------------------------------------------------DeepSpeed C++/CUDA extension op report + + + +JIT compiled ops requires ninjaJIT compiled ops requires ninjaJIT compiled ops requires ninja-------------------------------------------------- + + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +ninjaninjaninjaninja ...................................................... ..................[OKAY] [OKAY][OKAY][OKAY] + + + +---------------------------------------------------------------------------------------------------- +---------------------------------------------------------------------------------------------------- + +op nameop name +op name op name................................................ installedinstalled ................installed .. ....installedcompatible +compatible compatible-------------------------------------------------- +.. + +-------------------------------------------------- -------------------------------------------------- +compatible + +-------------------------------------------------- +cpu_adam ............... [NO]cpu_adam cpu_adam .......cpu_adam............... [OKAY]..............................[NO] + [NO].......[NO] .......[OKAY]....... + [OKAY][OKAY] + +fused_adam ............. [NO] ....... [OKAY] +fused_adamfused_lamb ..........................fused_adamfused_adam [NO][NO]............. ............. ....... ....... [NO][OKAY][NO][OKAY] + +.............. fused_lamb[OKAY][OKAY] + +............. [NO] fused_lamb.......fused_lamb .............[OKAY]sparse_attn +.............[NO] ............ [NO] ....... [NO] .......[OKAY]....... + [OKAY][OKAY] +sparse_attn + ............ [NO]transformer ................... [NO][OKAY] sparse_attn +....... transformersparse_attn ............[OKAY] +............[NO]............ stochastic_transformer [NO] [NO]....... . ....... .......[OKAY][NO] +[OKAY] [OKAY].......transformer + + [OKAY]............ +transformer stochastic_transformer [NO] ............ . .......[NO][NO] [OKAY].............. + [OKAY][OKAY] + +stochastic_transformer . stochastic_transformer[NO] ........ [OKAY][NO] + ....... [OKAY] +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report-------------------------------------------------- +-------------------------------------------------- + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.DeepSpeed C++/CUDA extension op report + +-------------------------------------------------- +----------------------------------------------------------------------------------------------------JIT compiled ops requires ninja-------------------------------------------------- + + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op report + + +------------------------------------------------------------------------------------------------------------------------------------------------------ + + +JIT compiled ops requires ninjaNOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + + +---------------------------------------------------------------------------------------------------- + +JIT compiled ops requires ninja +JIT compiled ops requires ninja +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +---------------------------------------------------------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +--------------------------------------------------DeepSpeed C++/CUDA extension op report + + +DeepSpeed C++/CUDA extension op report--------------------------------------------------JIT compiled ops requires ninja + + +--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- + + +DeepSpeed C++/CUDA extension op report--------------------------------------------------JIT compiled ops requires ninja + + +JIT compiled ops requires ninja-------------------------------------------------- + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +ninjaninjaninjaninja ........................................................................ [OKAY][OKAY][OKAY][OKAY] + + + +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + + + +op nameop nameop nameop name ................................................................ installed installed installedinstalled.... ..compatible .. + compatiblecompatible + -------------------------------------------------- +compatible +---------------------------------------------------------------------------------------------------- + + +-------------------------------------------------- +cpu_adamcpu_adamcpu_adam ...............cpu_adam.............................. [NO] [NO] ...............[NO]....... .......[NO] [OKAY] ....... +[OKAY] ....... +[OKAY] +[OKAY] +fused_adam ............. fused_adam[NO] fused_adam fused_adam.................... ............. [OKAY].............[NO] + [NO][NO]....... fused_lamb [OKAY]........................... + [NO]fused_lamb [OKAY][OKAY] +.................... + [NO][OKAY]fused_lamb +fused_lamb....... .............[OKAY]............. + [NO][NO] .............. [OKAY][OKAY] + +sparse_attn ............ [NO] ....... sparse_attn[OKAY] +............ [NO] .......sparse_attn transformersparse_attn[OKAY] + .................................... transformer[NO] [NO] [NO]............ ..............[NO]....... [OKAY][OKAY][OKAY]....... + + +[OKAY]stochastic_transformertransformer + transformer ............. ............[NO]stochastic_transformer [NO][NO] ...................... [OKAY][NO][OKAY][OKAY] + + +....... [OKAY] +stochastic_transformerstochastic_transformer .. [NO][NO] .............. [OKAY][OKAY] + +ninjaninjaninjaninja ........................................................................ [OKAY][OKAY][OKAY] + + +[OKAY]---------------------------------------------------------------------------------------------------- + + +--------------------------------------------------op nameop name-------------------------------------------------- + + ................op name................ op nameinstalled................ installed ................installed.... ..compatible compatibleinstalled +compatible + --------------------------------------------------.. + +-------------------------------------------------- -------------------------------------------------- +compatible + +-------------------------------------------------- +cpu_adam cpu_adam...............cpu_adam [NO] cpu_adam ............... ...................... ............... [NO][OKAY][NO] + [NO].............. .......[OKAY][OKAY] + +[OKAY] +fused_adam ............. fused_adamfused_adam[NO] fused_adam ............. ................................. [NO][OKAY] [NO] +[NO] ....... ....... ....... [OKAY]fused_lamb[OKAY] +[OKAY] + +............. fused_lambfused_lamb[NO]fused_lamb .......................... ....... [NO][NO]............. .......[OKAY].......[NO] + [OKAY][OKAY]....... + + [OKAY] +sparse_attnsparse_attn sparse_attn............ sparse_attn ............[NO]........................ [NO][NO].......[NO] ....... [OKAY]....... ....... +[OKAY] [OKAY] +transformer[OKAY] + +............ transformertransformer[NO] transformer ............ ...............................[NO] [OKAY].......[NO][NO] + [OKAY].............. + [OKAY][OKAY] +stochastic_transformer + stochastic_transformer .stochastic_transformer. stochastic_transformer [NO] [NO] . ....... ........ [NO] [OKAY] [NO] +....... [OKAY] ....... +[OKAY] +[OKAY] +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- +-------------------------------------------------- + +JIT compiled ops requires ninjaDeepSpeed C++/CUDA extension op report + +---------------------------------------------------------------------------------------------------- + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +--------------------------------------------------JIT compiled ops requires ninja + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +------------------------------------------------------------------------------------------------------------------------------------------------------ +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op report +-------------------------------------------------- + +DeepSpeed C++/CUDA extension op report +---------------------------------------------------------------------------------------------------- + +JIT compiled ops requires ninja +--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.---------------------------------------------------------------------------------------------------- + + +--------------------------------------------------JIT compiled ops requires ninja +JIT compiled ops requires ninjaJIT compiled ops requires ninja + + +------------------------------------------------------------------------------------------------------------------------------------------------------ + +-------------------------------------------------- +DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op report +DeepSpeed C++/CUDA extension op report + + +----------------------------------------------------------------------------------------------------DeepSpeed C++/CUDA extension op report-------------------------------------------------- + + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + + + +------------------------------------------------------------------------------------------------------------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + + + +JIT compiled ops requires ninjaJIT compiled ops requires ninjaJIT compiled ops requires ninja-------------------------------------------------- + + + +JIT compiled ops requires ninja +ninjaninjaninjaninja .................................... .................................... [OKAY][OKAY] [OKAY] + +[OKAY] + +------------------------------------------------------------------------------------------------------------------------------------------------------ +-------------------------------------------------- + + +op name op name................op nameop name ................ installed................ ................ ..installed installed compatibleinstalled .. .. +compatible.. -------------------------------------------------- + + compatiblecompatible +-------------------------------------------------- + +---------------------------------------------------------------------------------------------------- + +cpu_adam ............... [NO] cpu_adam....... cpu_adam...............cpu_adam [OKAY] ...............[NO] +............... [NO][NO]....... ....... [OKAY] ....... +[OKAY] +[OKAY]fused_adam + ............. [NO] ....... [OKAY] +fused_adam fused_adam.............fused_lambfused_adam ..........................[NO]............. [NO].......[NO][NO] .....................[OKAY] [OKAY] +[OKAY] +[OKAY] + +fused_lamb fused_lambfused_lamb............. ............. .............[NO] [NO][NO]....... ..............sparse_attn[OKAY] +[OKAY]............[OKAY] + +[NO] ....... [OKAY] +transformer ............ [NO]sparse_attn .......sparse_attn............ sparse_attn[OKAY]............ [NO] +............ [NO] ....... stochastic_transformer.......[NO][OKAY] [OKAY] +....... +. [OKAY][NO] +transformer transformer ....... ............transformer ............ [OKAY][NO][NO] + ............ .............. [NO][OKAY][OKAY] + +....... [OKAY]stochastic_transformer +stochastic_transformer .stochastic_transformer. [NO][NO] . ....... ....... [NO] [OKAY] [OKAY] +....... + [OKAY] +ninjaninjaninjaninja ........................................................................ [OKAY][OKAY][OKAY][OKAY] + + + +------------------------------------------------------------------------------------------------------------------------------------------------------ +-------------------------------------------------- + + +ninjaninjaninjaninja .................. ......................................................[OKAY] +[OKAY][OKAY][OKAY] +op nameop name op nameop name................................ installed................................ installed ..installed installed .. compatible..compatible +.. +---------------------------------------------------------------------------------------------------- + +-------------------------------------------------- + + +---------------------------------------------------------------------------------------------------- +op name-------------------------------------------------- + op name +................op name op name................installed ................ .................. installed installed compatibleinstalled +compatiblecompatible + +---------------------------------------------------------------------------------------------------- + +.. -------------------------------------------------- .. +.. compatible compatible +compatible +---------------------------------------------------------------------------------------------------- + + +--------------------------------------------------cpu_adam +cpu_adam cpu_adam............... cpu_adam...............cpu_adam[NO] [NO]..................................... ....... [NO][OKAY][NO] + [OKAY].............. + ............... [NO] ....... cpu_adam[OKAY]cpu_adam + [OKAY][OKAY] + + ...............cpu_adam ............... [NO]............... .......[NO][NO] [OKAY]fused_adam....... + ....... .............[OKAY] +fused_adam ............. [NO] fused_adam....... .............fused_adamfused_adam [OKAY] [NO] +[OKAY][NO] + ....... fused_adam[OKAY] +............. ............. .......[NO] fused_lamb [NO][OKAY] +............. [NO] fused_lamb....... .............[OKAY]fused_adam +........................... [OKAY]fused_lamb[OKAY][NO] + +.................... [OKAY][NO] + fused_adam[NO] .............fused_lamb.................... [NO][OKAY]............. [NO] +[NO]....... ..............[OKAY] +fused_lamb.......fused_lamb [OKAY].......................... + [OKAY][OKAY] + + [NO][NO] .............. [OKAY][OKAY] + +fused_lamb ............. [NO]fused_lamb .................... sparse_attn [OKAY] [NO] +sparse_attn ............ [NO] .......sparse_attn [OKAY]............ +............ sparse_attn[NO]....... ................... [OKAY] [NO] + [NO] transformer....... sparse_attnsparse_attn ............ [OKAY] ............[NO] +[OKAY] +.......sparse_attn [OKAY]transformer + ............ [NO] ....... [NO] transformer.......[OKAY] +...................[OKAY] +[NO][OKAY]stochastic_transformer transformer + ........................ transformer[NO][NO] ............ sparse_attn....... ....... [NO] [OKAY] ............[OKAY] +....... + ................... . transformer[OKAY][NO] + [NO][OKAY]stochastic_transformertransformer + [NO]................... stochastic_transformer .......[NO][OKAY] [OKAY] +........ + [NO][OKAY] stochastic_transformer....... + [OKAY] + .................... stochastic_transformer [NO][OKAY] [NO] +.stochastic_transformer [NO] ........ [OKAY][NO] + ....... [OKAY] +....... . .......transformer[NO] [OKAY] [OKAY]....... +............ + [OKAY] +[NO]stochastic_transformer ....... [OKAY]. + [NO] ....... stochastic_transformer[OKAY] +. [NO] ....... [OKAY] +-------------------------------------------------- +----------------------------------------------------------------------------------------------------DeepSpeed C++/CUDA extension op report + + +DeepSpeed C++/CUDA extension op report-------------------------------------------------- +DeepSpeed C++/CUDA extension op report + +--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- + +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + + + +----------------------------------------------------------------------------------------------------JIT compiled ops requires ninjaDeepSpeed C++/CUDA extension op report + + + +JIT compiled ops requires ninjaJIT compiled ops requires ninja-------------------------------------------------- + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +ninjaninjaninjaninja ...................................................... .................. [OKAY] [OKAY] +[OKAY] +-------------------------------------------------- + +[OKAY]-------------------------------------------------- + +op name-------------------------------------------------- op name +-------------------------------------------------- ................op name................ + installedinstalledop name ................ .. ..................installedcompatible +compatible --------------------------------------------------installed +.. + -------------------------------------------------- .. +compatible +compatible +-------------------------------------------------- +-------------------------------------------------- +cpu_adam ...............cpu_adam [NO]............... .......cpu_adam[NO] [OKAY]cpu_adam...................... + [OKAY]...............[NO] + .......[NO] [OKAY]....... + [OKAY] +fused_adam ............. fused_adam[NO] .................... [NO][OKAY]fused_adam +fused_adam ....... ............. fused_lamb[OKAY] ............. + [NO][NO].............fused_lamb ...........................[NO] [OKAY][NO] + .......[OKAY]....... +[OKAY]fused_lamb + [OKAY]fused_lamb +............. .............[NO] [NO]....... .......[OKAY] +[OKAY] +sparse_attn ............ [NO] sparse_attn....... ............[OKAY] +[NO]sparse_attn sparse_attn.......transformer ............ ........................ [NO] [OKAY][NO] [NO] + ..................... transformer[OKAY] [OKAY] +[OKAY]............ + + transformer[NO]transformer stochastic_transformer............................... [OKAY][NO].[NO] + [NO].............. stochastic_transformer .......[OKAY][OKAY] +[OKAY] +. + stochastic_transformer[NO]stochastic_transformer ........ . [OKAY][NO] + [NO]....... .......[OKAY] +[OKAY] +---------------------------------------------------------------------------------------------------- + +DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op report + +---------------------------------------------------------------------------------------------------- + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +---------------------------------------------------------------------------------------------------- + +JIT compiled ops requires ninja +DeepSpeed C++/CUDA extension op report +JIT compiled ops requires ninja + +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +ninjaninjaninjaninja ........................................................................ [OKAY][OKAY][OKAY] +[OKAY] + +-------------------------------------------------- +-------------------------------------------------- +-------------------------------------------------- +--------------------------------------------------op name + +op name op nameop name ................ ................installed................................ installed installed.. installed ..compatible +.... compatible--------------------------------------------------compatible +compatible + + +------------------------------------------------------------------------------------------------------------------------------------------------------ + + +cpu_adam ............... [NO] cpu_adam.......cpu_adamcpu_adam [OKAY].............................. +............... [NO][NO][NO] ..................... [OKAY] [OKAY] +[OKAY] +fused_adam + ............. [NO] ....... [OKAY] +fused_adam fused_adam.............fused_lambfused_adam [NO].......................... ............. .......[NO] [NO][NO] [OKAY] .............. + ....... [OKAY] fused_lamb[OKAY][OKAY] + + +............. [NO] fused_lamb.......fused_lamb [OKAY].......................... + [NO][NO] .............. [OKAY][OKAY]sparse_attn + +............ [NO] ....... [OKAY]sparse_attn + ............ [NO] transformer....... sparse_attn ............sparse_attn [OKAY] [NO] +........................ .......transformer[NO][NO] ...................[OKAY] ....... + [NO] [OKAY] [OKAY]....... +stochastic_transformer + [OKAY]transformer +transformer. ............ stochastic_transformer............ [NO] [NO][NO] ........ .......[OKAY][NO] +....... [OKAY][OKAY]....... + + [OKAY] +stochastic_transformerstochastic_transformer .. [NO] [NO]....... .......[OKAY] +[OKAY] +-------------------------------------------------- +--------------------------------------------------DeepSpeed C++/CUDA extension op report +---------------------------------------------------------------------------------------------------- +-------------------------------------------------- + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.DeepSpeed C++/CUDA extension op report +DeepSpeed C++/CUDA extension op report + + +DeepSpeed C++/CUDA extension op report------------------------------------------------------------------------------------------------------------------------------------------------------ + + + +JIT compiled ops requires ninjaNOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + + + +--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja + +--------------------------------------------------JIT compiled ops requires ninja + +JIT compiled ops requires ninja +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +---------------------------------------------------------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +--------------------------------------------------DeepSpeed C++/CUDA extension op report-------------------------------------------------- + +JIT compiled ops requires ninja-------------------------------------------------- + + +DeepSpeed C++/CUDA extension op reportNOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +---------------------------------------------------------------------------------------------------- +JIT compiled ops requires ninja + +--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- + +JIT compiled ops requires ninjaDeepSpeed C++/CUDA extension op report + +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +ninjaninjaninjaninja .................. ...................................................... [OKAY][OKAY][OKAY] +ninjaninjaninjaninja .................................... .................. .................. [OKAY][OKAY][OKAY][OKAY] + + + + +--------------------------------------------------[OKAY] + +-------------------------------------------------- + +op name---------------------------------------------------------------------------------------------------- op name +---------------------------------------------------------------------------------------------------- + +----------------------------------------------------------------------------------------------------op nameop name + +................ +op name ................installed op name installed .. ................................ .. compatibleinstalledcompatibleinstalled + +--------------------------------------------------....-------------------------------------------------- + op name................................op name installed ................ installed..................installed compatible.. installed +..compatible -------------------------------------------------- +.. +compatible-------------------------------------------------- + +compatible-------------------------------------------------- + + compatiblecompatible + +---------------------------------------------------------------------------------------------------- + + +-------------------------------------------------- +cpu_adam ............... [NO]cpu_adam ...................... cpu_adam cpu_adam[OKAY][NO] +cpu_adam ............... cpu_adam[NO] ......................cpu_adam cpu_adam [NO][OKAY] +...................... ............... [NO][OKAY] +[NO]....... .......[OKAY] +fused_adam[OKAY] +.............................. ....... [NO] [NO] [OKAY] ....... +............. [NO] .......fused_adam [OKAY]............. +....... [OKAY][OKAY]fused_adam + + fused_adam[NO] fused_adam....................fused_lamb [OKAY] .......................... + ............. [NO] ....... [OKAY]fused_adam +[NO] [NO][NO] .......fused_lamb....... .............[OKAY].......[OKAY] + +[NO][OKAY] + ............. [NO]fused_lamb ....... fused_adamfused_adam ............. [OKAY] .......................... +[NO] fused_lamb[NO].......[NO] .............[OKAY].............. +.......fused_lamb [OKAY].............fused_lamb + [OKAY][NO][OKAY] + +....... [OKAY]fused_lambfused_lamb + .............[NO] [NO]....... sparse_attn ....... [OKAY]............ + .......................... [NO][NO]sparse_attn .......................... [OKAY][NO][OKAY] +sparse_attn + [OKAY][NO] + ....... [OKAY] + ................... [OKAY][NO] + ....... transformer[OKAY] +sparse_attntransformer ........................ [NO][NO] sparse_attn .......sparse_attn....... [OKAY]............[OKAY]............ + +............ [NO]transformer ...................sparse_attnsparse_attn [OKAY][NO]............ + [NO] [NO]....... transformer ....... stochastic_transformer[OKAY] ............ [OKAY] + + ............ ....... [NO] stochastic_transformer[NO] [OKAY] .............. + . [OKAY] [OKAY]stochastic_transformer[NO] + +.[NO] transformertransformer.......[NO] ............................... [OKAY] [OKAY] + + .......transformer. transformer[OKAY]............ +[NO][NO] .............. stochastic_transformer[OKAY][OKAY] + +[NO]............[NO] ....... [NO] ....... [OKAY] +.......[OKAY] +. stochastic_transformerstochastic_transformer[NO] ........ .[OKAY] +[OKAY] +[NO][NO] .............. [OKAY][OKAY] + +stochastic_transformer stochastic_transformer . .[NO] [NO]....... .......[OKAY] +[OKAY] + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + [WARNING]  async_io: please install the libaio-devel package with yum +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.-------------------------------------------------- + + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + +async_ioasync_io .............................. [NO][NO] .............. [NO][NO] + +transformer_inferencetransformer_inference .... [NO][NO] .............. [OKAY][OKAY] + +utilsutils .................................... [NO][NO] .............. [OKAY][OKAY] + +quantizerquantizer .............. ..............[NO] [NO]....... ....... [OKAY][OKAY] + +-------------------------------------------------- +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] + [WARNING]  async_io: please install the libaio-devel package with yum +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info:DeepSpeed general environment info: + +torch install pathtorch install path .............................. ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch']['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] + +torch versiontorch version ........................................ 1.8.11.8.1 + +torch cuda versiontorch cuda version ............... ............... 11.111.1 + +nvcc versionnvcc version .......................................... 11.211.2 + +deepspeed install pathdeepspeed install path ...................... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed']['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] + +deepspeed infodeepspeed info ...................................... 0.5.5+29bee73, 29bee73, master0.5.5+29bee73, 29bee73, master + +deepspeed wheel compiled w.deepspeed wheel compiled w. ............ torch 1.8, cuda 11.1torch 1.8, cuda 11.1 + +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + [WARNING]  async_io: please install the libaio-devel package with yumasync_io + ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +[NO] +transformer_inferenceasync_io .. [NO]............... .......[NO] [OKAY]....... + [NO] +utils .................. [NO] ....... [OKAY] +DeepSpeed general environment info: +quantizer ..............transformer_inference [NO].. .......[NO] [OKAY]....... + [OKAY] +-------------------------------------------------- +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +-------------------------------------------------- +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +transformer_inference .. [NO]utils ......................... [OKAY][NO] + ....... [OKAY] +utils .................. [NO] quantizer....... ..............[OKAY] +[NO] ....... [OKAY] +quantizer .............. [NO]-------------------------------------------------- +....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_ioasync_io ............... ...............[NO] .......[NO] [NO]....... + [NO] +transformer_inferencetransformer_inference .... [NO][NO] .............. [OKAY][OKAY] + +utils utils.................. ..................[NO] [NO]....... [OKAY]....... + [OKAY] +quantizer .............. [NO]quantizer ..................... [OKAY][NO] + ....... [OKAY] +-------------------------------------------------- +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +transformer_inference .. utils[NO] ......................... [NO][OKAY] +....... [OKAY] +utils .................. quantizer[NO] ..................... [NO][OKAY] +....... [OKAY] +quantizer .............. [NO]-------------------------------------------------- +....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum [WARNING]  async_io: please install the libaio-devel package with yum + + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + +async_io async_io............... [NO]............... .......[NO] [NO]....... + [NO] +transformer_inference .. [NO]transformer_inference ......... [OKAY][NO] + ....... [OKAY] +utils .................. [NO]utils ......................... [OKAY][NO] + ....... [OKAY] +quantizer .............. [NO]quantizer ..................... [OKAY][NO] +....... [OKAY] +-------------------------------------------------- +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info:DeepSpeed general environment info: + +torch install pathtorch install path .............................. ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch']['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] + +torch versiontorch version ........................................ 1.8.11.8.1 + +torch cuda versiontorch cuda version .............................. 11.111.1 + +nvcc versionnvcc version .......................................... 11.2 +11.2deepspeed install path + deepspeed install path........... ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed']deepspeed info + deepspeed info................... ...................0.5.5+29bee73, 29bee73, master +0.5.5+29bee73, 29bee73, masterdeepspeed wheel compiled w. + deepspeed wheel compiled w....... torch 1.8, cuda 11.1...... + torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +DeepSpeed general environment info:DeepSpeed general environment info: + +torch install pathtorch install path .............................. ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch']['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] + +torch versiontorch version ........................................ 1.8.11.8.1 + +torch cuda versiontorch cuda version .............................. 11.111.1 + +nvcc versionnvcc version .......................................... 11.211.2 + +deepspeed install pathdeepspeed install path ...................... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed']['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] + +deepspeed info deepspeed info................... ...................0.5.5+29bee73, 29bee73, master +0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. deepspeed wheel compiled w....... ......torch 1.8, cuda 11.1 +torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +DeepSpeed general environment info: +torch version .................... 1.8.1 +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +torch version .................... 1.8.1 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install pathDeepSpeed general environment info: ............... + torch install path ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch']............... + torch version .................... 1.8.1 +['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch cuda version ............... torch version11.1 +.................... nvcc version1.8.1 +..................... 11.2torch cuda version + deepspeed install path............... ...........11.1 +nvcc version ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed']..................... + 11.2deepspeed info + deepspeed install path................... ...........0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed']...... + torch 1.8, cuda 11.1deepspeed info + ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info:DeepSpeed general environment info: + +torch install pathtorch install path .............................. ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch']['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] + +torch versiontorch version ........................................ 1.8.11.8.1 + +torch cuda versiontorch cuda version .............................. 11.111.1 + +nvcc versionnvcc version .......................................... 11.211.2 + +deepspeed install pathdeepspeed install path ...................... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed']['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] + +deepspeed infodeepspeed info ...................................... 0.5.5+29bee73, 29bee73, master0.5.5+29bee73, 29bee73, master + +deepspeed wheel compiled w.deepspeed wheel compiled w. ............ torch 1.8, cuda 11.1torch 1.8, cuda 11.1 + +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +DeepSpeed general environment info: +DeepSpeed general environment info: +torch install path ...............torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +.................... 1.8.1 +torch version torch cuda version.................... ...............1.8.1 +11.1 +torch cuda versionnvcc version .................................... 11.211.1 + +deepspeed install pathnvcc version ................................ 11.2 +['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed']deepspeed install path + ...........deepspeed info ................... 0.5.5+29bee73, 29bee73, master +['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed wheel compiled w. deepspeed info...... ...................torch 1.8, cuda 11.1 +0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info:DeepSpeed general environment info: + +torch install pathtorch install path .............................. ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch']['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] + +torch versiontorch version ........................................ 1.8.11.8.1 + +torch cuda versiontorch cuda version .............................. 11.111.1 + +nvcc versionnvcc version .......................................... 11.211.2 + +deepspeed install pathdeepspeed install path ...................... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed']['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] + +deepspeed infodeepspeed info ...................................... 0.5.5+29bee73, 29bee73, master0.5.5+29bee73, 29bee73, master + +deepspeed wheel compiled w.deepspeed wheel compiled w. ............ torch 1.8, cuda 11.1torch 1.8, cuda 11.1 + + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +---------------------------------------------------------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +DeepSpeed C++/CUDA extension op report +------------------------------------------------------------------------------------------------------------------------------------------------------ + +JIT compiled ops requires ninja +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +-------------------------------------------------- +--------------------------------------------------JIT compiled ops requires ninja + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.DeepSpeed C++/CUDA extension op report + +---------------------------------------------------------------------------------------------------- + +JIT compiled ops requires ninjaNOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +-------------------------------------------------- +JIT compiled ops requires ninja +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +---------------------------------------------------------------------------------------------------- + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.DeepSpeed C++/CUDA extension op report + +------------------------------------------------------------------------------------------------------------------------------------------------------ + + +JIT compiled ops requires ninjaDeepSpeed C++/CUDA extension op reportNOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + + +---------------------------------------------------------------------------------------------------- + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.JIT compiled ops requires ninja + +-------------------------------------------------- +JIT compiled ops requires ninja +ninjaninjaninjaninja .................................... .................. ..................[OKAY] [OKAY] [OKAY] + +[OKAY] +---------------------------------------------------------------------------------------------------- + +-------------------------------------------------- +--------------------------------------------------op name +op name + op name................op name ................installed ................ .................. installed installedcompatible +..installed.. --------------------------------------------------compatible +compatible +..-------------------------------------------------- + + --------------------------------------------------compatible + +-------------------------------------------------- +cpu_adam cpu_adam............... [NO]...............cpu_adam cpu_adam....... [NO] .............................. [OKAY] ....... +[NO][NO] [OKAY].............. + [OKAY][OKAY] + +fused_adam ............. [NO] .......fused_adam [OKAY]fused_adam + .............fused_adam............. fused_lamb[NO] ..........................[NO] ..............[NO][NO] [OKAY] [OKAY]....... +....... + [OKAY][OKAY]fused_lamb +fused_lamb + .......................... fused_lamb [NO][NO] .................... ....... [NO] [OKAY] sparse_attn[OKAY] + ....... +............ [OKAY][NO] + ....... [OKAY] +sparse_attntransformer ........................sparse_attn [NO] [NO]sparse_attn ................... ................... [NO][OKAY][OKAY] [NO] +....... + .......[OKAY] stochastic_transformer +[OKAY]transformer + transformer............. [NO]transformer............[NO] ..........................[NO] [OKAY] + [NO][OKAY]....... + .......[OKAY] +[OKAY] +stochastic_transformer .stochastic_transformerstochastic_transformer [NO] ......... [OKAY][NO][NO] + .............. [OKAY][OKAY] + +ninjaninjaninja ...................................................... ninja [OKAY][OKAY][OKAY] + +.................. +---------------------------------------------------------------------------------------------------- -------------------------------------------------- + +[OKAY] +op name +op name op name--------------------------------------------------................ + installed................op name ................ .. ................installed installed compatible installed +.... -------------------------------------------------- ..compatible + compatible +compatible +---------------------------------------------------------------------------------------------------- + + +-------------------------------------------------- +cpu_adam ............... [NO] cpu_adam.......cpu_adam cpu_adam...............[OKAY] ............... +............... [NO][NO][NO] ..................... [OKAY]fused_adam[OKAY] + [OKAY] +............. + [NO] ....... [OKAY] +fused_lambfused_adam .............fused_adam.............fused_adam [NO] [NO]............. ............. .............. [NO] [NO] [OKAY][OKAY] ....... + +....... [OKAY][OKAY] +fused_lamb + .............fused_lamb [NO] .............fused_lamb....... sparse_attn ............. [OKAY][NO] ............ + [NO] .......[NO]....... [OKAY].......[OKAY] + +[OKAY] +transformer sparse_attn............ ............[NO] [NO]....... .......[OKAY] sparse_attn +sparse_attn[OKAY] +stochastic_transformer........................ transformer[NO][NO]. [NO] ................... ....... ....... [NO] [OKAY] [OKAY][OKAY] + +....... +transformer [OKAY]............ +transformer [NO]............ .......[NO] stochastic_transformer[OKAY]....... + [OKAY]. + stochastic_transformer[NO] stochastic_transformer....... . [OKAY].[NO] + [NO]....... .......[OKAY] +[OKAY] +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +--------------------------------------------------JIT compiled ops requires ninja + +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +ninjaninjaninjaninja .................................... .................. ..................[OKAY][OKAY][OKAY] + + +[OKAY]-------------------------------------------------- +---------------------------------------------------------------------------------------------------- + +op name-------------------------------------------------- + +op name................op name ................installedop name ................ installed.. ................ installed..compatible +installedcompatible..-------------------------------------------------- + +..--------------------------------------------------compatible + +compatible +-------------------------------------------------- +-------------------------------------------------- +cpu_adam ...............cpu_adam [NO]............... cpu_adamcpu_adam.......[NO] ............... ............... [OKAY] ....... +[NO] [NO] [OKAY] ....... +....... [OKAY] +[OKAY] +fused_adam ............. fused_adam[NO] fused_adam .............fused_adam....... [NO].............[OKAY] ............. +....... [NO] [NO] fused_lamb [OKAY]....... ....... + ............. [OKAY] [OKAY] +fused_lamb +[NO] ....................fused_lamb [NO]fused_lamb[OKAY] +....... ............. .............[OKAY] +[NO][NO] .............. [OKAY][OKAY] + +sparse_attn ............ [NO] .......sparse_attn [OKAY]............ + [NO]sparse_attnsparse_attn ....... transformer........................ [OKAY]............[NO] +[NO] ....... transformer[NO] ....... [OKAY]............ ....... + [OKAY] [NO] +[OKAY] transformer +....... transformer............[OKAY] +stochastic_transformer............[NO] [NO]stochastic_transformer ........ ....... [OKAY].[NO] + [NO][OKAY]....... +stochastic_transformer....... [OKAY][OKAY] +stochastic_transformer +. [NO] ........ [NO][OKAY] +....... [OKAY] +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +using world size: 128, data-parallel-size: 1, tensor-model-parallel size: 4, pipeline-model-parallel size: 32 +using torch.float16 for parameters ... +------------------------ arguments ------------------------ + accumulate_allreduce_grads_in_fp32 .............. False + adam_beta1 ...................................... 0.9 + adam_beta2 ...................................... 0.95 + adam_eps ........................................ 1e-08 + adlr_autoresume ................................. False + adlr_autoresume_interval ........................ 1000 + apply_query_key_layer_scaling ................... True + apply_residual_connection_post_layernorm ........ False + attention_dropout ............................... 0.1 + attention_softmax_in_fp32 ....................... False + bert_binary_head ................................ True + bert_load ....................................... None + bf16 ............................................ False + bias_dropout_fusion ............................. True + bias_gelu_fusion ................................ True + biencoder_projection_dim ........................ 0 + biencoder_shared_query_context_model ............ False + block_data_path ................................. None + checkpoint_activations .......................... True + checkpoint_in_cpu ............................... False + checkpoint_num_layers ........................... 1 + clip_grad ....................................... 1.0 + codecarbon_dir .................................. None + consumed_train_samples .......................... 0 + consumed_train_tokens ........................... 0 + consumed_valid_samples .......................... 0 + contigious_checkpointing ........................ False + cpu_optimizer ................................... False + cpu_torch_adam .................................. False + curriculum_learning ............................. False + data_impl ....................................... mmap + data_parallel_size .............................. 1 + data_path ....................................... ['/gpfswork/rech/six/commun/datasets-custom/oscar-en/meg-gpt2_text_document'] + dataloader_type ................................. single + DDP_impl ........................................ local + decoder_seq_length .............................. None + deepscale ....................................... False + deepscale_config ................................ None + deepspeed ....................................... True + deepspeed_activation_checkpointing .............. True + deepspeed_config ................................ ./ds_config.1655850.json + deepspeed_mpi ................................... False + distribute_checkpointed_activations ............. False + distributed_backend ............................. nccl + embedding_path .................................. None + encoder_seq_length .............................. 2048 + eod_mask_loss ................................... False + eval_interval ................................... 1000 + eval_iters ...................................... 5 + evidence_data_path .............................. None + exit_duration_in_mins ........................... 55 + exit_interval ................................... None + ffn_hidden_size ................................. 46400 + finetune ........................................ False + fp16 ............................................ True + fp16_lm_cross_entropy ........................... False + fp32_residual_connection ........................ False + gigaflos_no_embeds .............................. 0 + global_batch_size ............................... 2048 + glu_activation .................................. None + hidden_dropout .................................. 0.1 + hidden_size ..................................... 11600 + hysteresis ...................................... 2 + ict_head_size ................................... None + ict_load ........................................ None + img_dim ......................................... 224 + indexer_batch_size .............................. 128 + indexer_log_interval ............................ 1000 + init_method_std ................................. 0.02 + init_method_xavier_uniform ...................... False + initial_loss_scale .............................. 4294967296 + kv_channels ..................................... 145 + layernorm_epsilon ............................... 1e-05 + lazy_mpu_init ................................... None + load ............................................ /gpfsscratch/rech/six/commun/checkpoints/tr8b-104B/checkpoints + local_rank ...................................... 0 + log_batch_size_to_tensorboard ................... True + log_interval .................................... 1 + log_learning_rate_to_tensorboard ................ True + log_loss_scale_to_tensorboard ................... True + log_num_zeros_in_grad ........................... False + log_params_norm ................................. False + log_timers_to_tensorboard ....................... True + log_validation_ppl_to_tensorboard ............... True + loss_on_targets_only ............................ False + loss_scale ...................................... 12.0 + loss_scale_window ............................... 1000 + lr .............................................. 6e-05 + lr_decay_iters .................................. None + lr_decay_samples ................................ None + lr_decay_style .................................. cosine + lr_decay_tokens ................................. 260000000000 + lr_warmup_fraction .............................. None + lr_warmup_iters ................................. 0 + lr_warmup_samples ............................... 216320 + make_vocab_size_divisible_by .................... 128 + mask_prob ....................................... 0.15 + masked_softmax_fusion ........................... True + max_position_embeddings ......................... 2048 + memory_centric_tiled_linear ..................... False + merge_file ...................................... /gpfswork/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed-tr8b-104B/data/gpt2-merges.txt + micro_batch_size ................................ 1 + min_loss_scale .................................. 1.0 + min_lr .......................................... 6e-06 + mmap_warmup ..................................... False + no_load_optim ................................... None + no_load_rng ..................................... None + no_save_optim ................................... None + no_save_rng ..................................... None + num_attention_heads ............................. 80 + num_channels .................................... 3 + num_classes ..................................... 1000 + num_layers ...................................... 64 + num_layers_per_virtual_pipeline_stage ........... None + num_workers ..................................... 2 + onnx_safe ....................................... None + openai_gelu ..................................... False + optimizer ....................................... adam + override_lr_scheduler ........................... False + params_dtype .................................... torch.float16 + partition_activations ........................... False + patch_dim ....................................... 16 + pipeline_model_parallel_size .................... 32 + position_embedding_type ......................... PositionEmbeddingType.absolute + profile_backward ................................ False + query_in_block_prob ............................. 0.1 + rampup_batch_size ............................... None + rank ............................................ 0 + remote_device ................................... none + reset_attention_mask ............................ False + reset_position_ids .............................. False + retriever_report_topk_accuracies ................ [] + retriever_score_scaling ......................... False + retriever_seq_length ............................ 256 + sample_rate ..................................... 1.0 + save ............................................ /gpfsscratch/rech/six/commun/checkpoints/tr8b-104B/checkpoints + save_interval ................................... 300 + scatter_gather_tensors_in_pipeline .............. True + scattered_embeddings ............................ False + seed ............................................ 43 + seq_length ...................................... 2048 + sgd_momentum .................................... 0.9 + short_seq_prob .................................. 0.1 + split ........................................... 949,50,1 + split_transformers .............................. False + synchronize_each_layer .......................... False + tensor_model_parallel_size ...................... 4 + tensorboard_dir ................................. /gpfsscratch/rech/six/commun/checkpoints/tr8b-104B/tr8b-104B-logs/tensorboard + tensorboard_log_interval ........................ 1 + tensorboard_queue_size .......................... 5 + tile_factor ..................................... 1 + titles_data_path ................................ None + tokenizer_name_or_path .......................... None + tokenizer_type .................................. GPT2BPETokenizer + train_iters ..................................... None + train_samples ................................... 600000000 + train_tokens .................................... 300000000000 + use_bnb_optimizer ............................... False + use_checkpoint_lr_scheduler ..................... False + use_contiguous_buffers_in_ddp ................... False + use_cpu_initialization .......................... None + use_one_sent_docs ............................... False + use_pin_memory .................................. False + virtual_pipeline_model_parallel_size ............ None + vocab_extra_ids ................................. 0 + vocab_file ...................................... /gpfswork/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed-tr8b-104B/data/gpt2-vocab.json + weight_decay .................................... 0.1 + world_size ...................................... 128 + zero_allgather_bucket_size ...................... 0.0 + zero_contigious_gradients ....................... False + zero_reduce_bucket_size ......................... 0.0 + zero_reduce_scatter ............................. False + zero_stage ...................................... 1 +-------------------- end of arguments --------------------- +setting number of micro-batches to constant 2048 +> building GPT2BPETokenizer tokenizer ... +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +--------------------------------------------------JIT compiled ops requires ninja-------------------------------------------------- + +DeepSpeed C++/CUDA extension op report +DeepSpeed C++/CUDA extension op report + +---------------------------------------------------------------------------------------------------- + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +------------------------------------------------------------------------------------------------------------------------------------------------------ + + +JIT compiled ops requires ninjaJIT compiled ops requires ninjaDeepSpeed C++/CUDA extension op report + + +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +ninjaninjaninjaninja ........................................................................ [OKAY] [OKAY] +[OKAY][OKAY] + +-------------------------------------------------- +---------------------------------------------------------------------------------------------------- + +-------------------------------------------------- + +op nameop nameop nameop name ................................ ................ ................ installedinstalledinstalled installed .... .. compatible..compatiblecompatible + + +--------------------------------------------------compatible---------------------------------------------------------------------------------------------------- + + + +-------------------------------------------------- +cpu_adamcpu_adamcpu_adam ...............cpu_adam ............... ............... ...............[NO] [NO] .......[NO][NO]....... [OKAY].......[OKAY]....... + + [OKAY][OKAY] + +fused_adamfused_adamfused_adam fused_adam .......................... ............. [NO]............. [NO] [NO] .......[NO] ....... ....... [OKAY]....... [OKAY] +[OKAY] +[OKAY] + +fused_lambfused_lambfused_lambfused_lamb .................................................... [NO][NO][NO][NO] ............................ [OKAY][OKAY][OKAY][OKAY] + + + +sparse_attn sparse_attn............sparse_attnsparse_attn ............[NO]........................ [NO] ....... [NO][NO] [OKAY]....... +....... ....... [OKAY] [OKAY] +transformer[OKAY] + +............transformer transformer transformer [NO] ............ ........................ ....... [NO] [NO][NO] [OKAY] +..................... [OKAY][OKAY]stochastic_transformer[OKAY] + + + .stochastic_transformerstochastic_transformer stochastic_transformer [NO] . ......... [NO][OKAY][NO][NO] + ..................... [OKAY][OKAY][OKAY] + + +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main ******** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** + +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +---------------------------------------------------------------------------------------------------- + +DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op report + +------------------------------------------------------------------------------------------------------------------------------------------------------ +-------------------------------------------------- + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +DeepSpeed C++/CUDA extension op report +DeepSpeed C++/CUDA extension op report +---------------------------------------------------------------------------------------------------- + +-------------------------------------------------- +--------------------------------------------------JIT compiled ops requires ninja +JIT compiled ops requires ninja + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +---------------------------------------------------------------------------------------------------- + +JIT compiled ops requires ninjaJIT compiled ops requires ninja + +ninjaninjaninjaninja ........................................................................ [OKAY][OKAY][OKAY] + + +[OKAY]-------------------------------------------------- +---------------------------------------------------------------------------------------------------- + +-------------------------------------------------- + +op nameop nameop name op name ................ ................ ................................installed installedinstalled installed.. .... compatible compatible +compatible.. +-------------------------------------------------- + +---------------------------------------------------------------------------------------------------- + +compatible +cpu_adam-------------------------------------------------- +...............cpu_adam cpu_adam[NO]............... ......................[NO] cpu_adam [NO] .......[OKAY] ............... +.......[OKAY] +[OKAY] + [NO]fused_adam ............. fused_adam[NO]fused_adam....... ............. ....... ............. [NO][OKAY][OKAY] + .......[NO] ....... +fused_lamb [OKAY] [OKAY]............. + + [NO] ....... fused_lambfused_lamb[OKAY] +fused_adam .......................... [NO][NO] ........................... [OKAY][NO][OKAY] + +sparse_attn ................... [NO] ....... [OKAY][OKAY] + +transformersparse_attn sparse_attnfused_lamb ............ ............ ............ [NO] [NO][NO]....... ............. .............. [OKAY] [OKAY] +[OKAY][NO] + +stochastic_transformertransformertransformer ......................... [NO][NO][NO]....... ..................... [OKAY][OKAY] + +[OKAY][OKAY] + +stochastic_transformer . [NO]stochastic_transformer ....... [OKAY]. + [NO] sparse_attn....... ............[OKAY] + [NO] ....... [OKAY] +transformer ............ [NO] ....... [OKAY] +stochastic_transformer . [NO] ....... [OKAY] +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main ******** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** + + > padded vocab (size: 50257) with 431 dummy tokens (new size: 50688) +> initializing torch distributed ... +---------------------------------------------------------------------------------------------------- +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- + + +DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op reportNOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- + + +-------------------------------------------------- +--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- + + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +JIT compiled ops requires ninja + +JIT compiled ops requires ninja +-------------------------------------------------- + +--------------------------------------------------JIT compiled ops requires ninja + +JIT compiled ops requires ninja +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report-------------------------------------------------- +---------------------------------------------------------------------------------------------------- + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.DeepSpeed C++/CUDA extension op report + + +--------------------------------------------------DeepSpeed C++/CUDA extension op report---------------------------------------------------------------------------------------------------- + + + +--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.DeepSpeed C++/CUDA extension op reportJIT compiled ops requires ninja + + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.---------------------------------------------------------------------------------------------------- + + +--------------------------------------------------JIT compiled ops requires ninjaNOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + + +JIT compiled ops requires ninja +-------------------------------------------------- +JIT compiled ops requires ninja +ninjaninjaninjaninja .................. .................. [OKAY].................................... + [OKAY][OKAY]--------------------------------------------------[OKAY] + + + +--------------------------------------------------op name---------------------------------------------------------------------------------------------------- + + +................op name op name op nameinstalled................................ ................ installed .. installed .. compatible installed +compatible .. +--------------------------------------------------.. -------------------------------------------------- compatible + +compatible + +---------------------------------------------------------------------------------------------------- + +cpu_adamcpu_adam .............................. [NO][NO] .......cpu_adam cpu_adam....... [OKAY] ............... +............... [OKAY] [NO][NO] + .............. [OKAY][OKAY] + +fused_adam ............. [NO] fused_adam....... .............[OKAY] +[NO]fused_adam .......fused_adamfused_lamb............. .............[OKAY][NO]............. + [NO].......[NO] fused_lamb.......[OKAY]....... + .............[OKAY][OKAY] + +[NO]fused_lamb ....................fused_lamb [NO][OKAY]............. + .......[NO] [OKAY] sparse_attn +....... ............[OKAY] +[NO] .......sparse_attn [OKAY]............ + [NO] transformer....... ............[OKAY] +sparse_attn[NO] sparse_attntransformer................... ............[NO]............[OKAY] [NO] +[NO]....... .......[OKAY]....... + stochastic_transformer [OKAY] [OKAY] +transformer + ............. transformerstochastic_transformer[NO] [NO] ................... ........ [OKAY][NO][NO] [OKAY] + ....... +....... [OKAY][OKAY] + +stochastic_transformer . stochastic_transformer[NO] ........ [OKAY][NO] + ....... [OKAY] +ninjaninjaninjaninja .................................... .................. ..................[OKAY][OKAY] + +[OKAY][OKAY] +-------------------------------------------------- + +--------------------------------------------------op name---------------------------------------------------------------------------------------------------- + + +................op nameop name op name installed................ ................ ................ ..installed installed installed.. compatible .. + .. compatible--------------------------------------------------compatible + + +compatible-------------------------------------------------- + +-------------------------------------------------- +-------------------------------------------------- +cpu_adam ............... cpu_adam[NO]cpu_adamcpu_adam .................................................... [OKAY][NO][NO][NO] + ..................... [OKAY][OKAY][OKAY] + + +fused_adam ............. [NO] .......fused_adamfused_adam fused_adam [OKAY] ....................................... + [NO][NO][NO] fused_lamb ....... ....... .................... [OKAY][OKAY] [OKAY] + +[NO] + .......fused_lambfused_lamb fused_lamb [OKAY] +....................................... [NO][NO][NO] ..................... [OKAY][OKAY][OKAY] + + +sparse_attn ............ [NO] ....... [OKAY] +transformersparse_attnsparse_attn sparse_attn ................................................ [NO][NO][NO][NO] ....... ..................... [OKAY][OKAY][OKAY] +[OKAY] + + +transformer transformertransformerstochastic_transformer............ ........................[NO] .[NO][NO] ....... ....... .......[NO][OKAY] [OKAY]....... +[OKAY] + +[OKAY] +stochastic_transformerstochastic_transformer stochastic_transformer .. . [NO] [NO] [NO] ....... ....... ....... [OKAY] +[OKAY][OKAY] + +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +---------------------------------------------------------------------------------------------------- +DeepSpeed C++/CUDA extension op report + +--------------------------------------------------DeepSpeed C++/CUDA extension op report + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.--------------------------------------------------JIT compiled ops requires ninja + +-------------------------------------------------- + +JIT compiled ops requires ninja +DeepSpeed C++/CUDA extension op report +---------------------------------------------------------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +--------------------------------------------------DeepSpeed C++/CUDA extension op report + +JIT compiled ops requires ninja-------------------------------------------------- + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +ninjaninjaninjaninja ........................................................................ [OKAY][OKAY][OKAY][OKAY] + + + +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + + + +op nameop name op nameop name ................ ................ ................................ installed installedinstalledinstalled .... .... compatible compatiblecompatible + +compatible +---------------------------------------------------------------------------------------------------- + +-------------------------------------------------- + +-------------------------------------------------- +cpu_adam ...............cpu_adamcpu_adam ...............[NO]cpu_adam............... ....... [NO] ...............[NO] [OKAY] +.......[NO]....... [OKAY][OKAY]....... + + [OKAY] +fused_adam ............. [NO] ....... fused_adam[OKAY] fused_adam +............. fused_adam.............[NO] ............. fused_lamb [NO] [NO] .................... ....... ....... [NO][OKAY] [OKAY] +....... +[OKAY] +[OKAY] +fused_lambfused_lamb fused_lamb.......................... [NO].............[NO] .......[NO]....... sparse_attn[OKAY]....... +............[OKAY] +[OKAY][NO] + ....... [OKAY] +transformer ............ [NO]sparse_attn ................... [OKAY]sparse_attn[NO] + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + ............sparse_attn....... stochastic_transformer [NO][OKAY]............ + ........[NO] [OKAY] transformer + [NO]................... .......transformer[OKAY][NO] + [OKAY]................... + transformer [NO][OKAY] +................... [OKAY][NO]stochastic_transformer + ....... .[OKAY] stochastic_transformer +[NO] ........ stochastic_transformer[OKAY][NO] +........ [OKAY] +[NO] ....... [OKAY] + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +---------------------------------------------------------------------------------------------------- +--------------------------------------------------DeepSpeed C++/CUDA extension op report + + +DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op report-------------------------------------------------- + +-------------------------------------------------- +----------------------------------------------------------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.DeepSpeed C++/CUDA extension op reportNOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- + + + +------------------------------------------------------------------------------------------------------------------------------------------------------JIT compiled ops requires ninja + + +JIT compiled ops requires ninja +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +JIT compiled ops requires ninja + +-------------------------------------------------- +JIT compiled ops requires ninja +ninjaninjaninjaninja .................. .................. .................................... [OKAY] [OKAY][OKAY] +[OKAY] + +-------------------------------------------------- +---------------------------------------------------------------------------------------------------- + + +op name-------------------------------------------------- op nameop name................ + installedop name................ ................ .. ................ installedinstalled compatible installed.. + .. --------------------------------------------------.. + compatiblecompatiblecompatible + + +------------------------------------------------------------------------------------------------------------------------------------------------------ +cpu_adam + + ............... [NO] ....... [OKAY] +cpu_adamcpu_adamcpu_adam ............................................. [NO][NO][NO] ....... ....... .......fused_adam[OKAY] +[OKAY][OKAY]............. + +[NO] ....... [OKAY] +fused_lamb fused_adam............. fused_adamfused_adam............. [NO] ............. .............[NO]....... .......[NO][OKAY] [NO] +[OKAY] +.............. fused_lamb[OKAY] [OKAY] +............. + [NO] .......fused_lambsparse_attnfused_lamb [OKAY] ............. +......................... [NO][NO][NO] ..................... [OKAY][OKAY][OKAY] + + +sparse_attn transformer............ ............[NO] [NO]....... .......[OKAY] +[OKAY] +sparse_attnsparse_attntransformerstochastic_transformer ............ ........................[NO] . [NO] [NO][NO] ............................ [OKAY][OKAY][OKAY] +[OKAY] + + +stochastic_transformer transformertransformer. ........................[NO] [NO][NO]....... ....... [OKAY] ....... +[OKAY] +[OKAY] +stochastic_transformer stochastic_transformer. [NO]. .......[NO] [OKAY]....... + [OKAY] +------------------------------------------------------------------------------------------------------------------------------------------------------ + + +--------------------------------------------------DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op report + + + +----------------------------------------------------------------------------------------------------DeepSpeed C++/CUDA extension op report +-------------------------------------------------- + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- + + + +----------------------------------------------------------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- + + + +JIT compiled ops requires ninjaJIT compiled ops requires ninjaJIT compiled ops requires ninja +-------------------------------------------------- + + +JIT compiled ops requires ninja +ninjaninjaninja ninja .................. .................................... .................. [OKAY] +[OKAY][OKAY][OKAY] +-------------------------------------------------- + + +-------------------------------------------------- +--------------------------------------------------op name +-------------------------------------------------- op name +op name ................ op name................................installed installed ................installed.. ....installed compatible compatible + +compatible.. +---------------------------------------------------------------------------------------------------- +-------------------------------------------------- +compatible + +-------------------------------------------------- +cpu_adam cpu_adam...............cpu_adam [NO]...............cpu_adam ............... [NO] ....... ............... ....... [NO][OKAY] [NO] +[OKAY] ....... +....... [OKAY][OKAY] + +fused_adam ............. [NO]fused_adam ....... fused_adam[OKAY] +.............fused_adam............. [NO] .............[NO]fused_lamb ....... .................... [NO][OKAY] [OKAY] + +[NO]....... ....... fused_lamb[OKAY]fused_lamb[OKAY] + +............. .............[NO] fused_lamb[NO]....... ....................[OKAY] +[NO][OKAY] +....... [OKAY] +sparse_attn ............ [NO] ....... [OKAY] +sparse_attnsparse_attntransformer ........................sparse_attn ............ [NO] ............[NO][NO]....... .......[NO].......[OKAY] +[OKAY].......[OKAY] + +transformer[OKAY] +............transformer stochastic_transformer transformer............[NO] .............[NO] ....... [NO][NO]....... .......[OKAY].......[OKAY] + +[OKAY] +[OKAY] +stochastic_transformerstochastic_transformer .stochastic_transformer . [NO] [NO]........ [NO].......[OKAY] +.......[OKAY] +[OKAY] + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +-------------------------------------------------- +--------------------------------------------------DeepSpeed C++/CUDA extension op report +-------------------------------------------------- + +--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.DeepSpeed C++/CUDA extension op report + +-------------------------------------------------- +-------------------------------------------------- +DeepSpeed C++/CUDA extension op reportJIT compiled ops requires ninja + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +-------------------------------------------------- +--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +JIT compiled ops requires ninja-------------------------------------------------- + +JIT compiled ops requires ninja + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io  [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found................ + [NO] ....... [NO] +async_io ............... [NO] .......transformer_inference [NO].. + [NO] ....... [OKAY] +utils .................. [NO] .......transformer_inference [OKAY].. + [NO] ....... [OKAY]quantizer + .............. [NO]utils ......................... [OKAY][NO] + ....... [OKAY] +-------------------------------------------------- +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +ninjaninjaninjaninja ........................................................................ [OKAY][OKAY][OKAY][OKAY] + + + +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + + + +op nameop name op nameop name ................................ ................ installed................ installed .. installed.. installed .. compatible +compatible..--------------------------------------------------compatible + + +--------------------------------------------------compatible +-------------------------------------------------- + +-------------------------------------------------- +cpu_adam ............... [NO] ....... cpu_adamcpu_adam[OKAY]cpu_adam ............... + ............... ............... [NO] [NO] [NO] ....... ....... ....... [OKAY] [OKAY]fused_adam +[OKAY] +............. + [NO] ....... [OKAY] +fused_adam fused_lambfused_adam............. fused_adam ............. [NO] .............[NO] ............. [NO]..............[NO] [OKAY] + .......[OKAY]....... +[OKAY] +[OKAY]fused_lamb + .............fused_lamb [NO]fused_lambsparse_attn............. ................................[NO] [OKAY] [NO] +.......[NO] [OKAY].............. + [OKAY][OKAY] + +transformer ............ [NO]sparse_attn ....... ............[OKAY] +sparse_attn[NO]sparse_attn ...................stochastic_transformer ............ [OKAY]. [NO] + [NO] [NO] transformer....... ....... ................... [OKAY][NO][OKAY] +....... +[OKAY] +[OKAY]transformertransformer + ........................ [NO]stochastic_transformer [NO] .............. . [OKAY] [OKAY] +[NO] + ....... [OKAY]stochastic_transformerstochastic_transformer + .. [NO][NO] .............. [OKAY][OKAY] + + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +------------------------------------------------------------------------------------------------------------------------------------------------------ + +--------------------------------------------------DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op report + + + +DeepSpeed C++/CUDA extension op report----------------------------------------------------------------------------------------------------DeepSpeed C++/CUDA extension op report + + + +--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- + + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- + +-------------------------------------------------- +JIT compiled ops requires ninja-------------------------------------------------- +JIT compiled ops requires ninja + +JIT compiled ops requires ninja +JIT compiled ops requires ninja + + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. [WARNING]  async_io: please install the libaio-devel package with yum + +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +ninjaninjaninja .................. ....................................ninja[OKAY] [OKAY].................. +[OKAY] +-------------------------------------------------- +[OKAY]-------------------------------------------------- + +-------------------------------------------------- +op name-------------------------------------------------- op name + +................ op nameop name ................ ................................ installed installedinstalledinstalled ........ compatiblecompatiblecompatible +compatible + +-------------------------------------------------- +---------------------------------------------------------------------------------------------------- + +-------------------------------------------------- + +cpu_adamcpu_adamcpu_adam cpu_adam .............................. ............... ...............[NO] [NO] [NO] [NO] .............. ....... ....... [OKAY][OKAY] +[OKAY][OKAY] + + + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +fused_adamfused_adam fused_adamfused_adam ............. ............. ............. [NO][NO].............[NO] .....................[NO] [OKAY][OKAY] [OKAY] + +....... + [OKAY]fused_lamb +fused_lamb fused_lamb ............. .............fused_lamb ............. [NO][NO].............[NO] .....................[NO] [OKAY] + [OKAY][OKAY]....... + + [OKAY] +sparse_attn ............ [NO]sparse_attnsparse_attn sparse_attn ....... ............ ........................ [OKAY][NO] [NO] +[NO] ....... .......transformer ....... [OKAY] [OKAY]............ +[OKAY] + +[NO]transformer transformertransformer ............................... [NO][NO][OKAY]............ +....... ....... [NO][OKAY][OKAY] +.......stochastic_transformer +[OKAY] +. stochastic_transformer[NO] stochastic_transformer stochastic_transformer........ . [NO]. [OKAY] [NO] +[NO] ....... ....... ....... [OKAY] [OKAY] +[OKAY] + +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +---------------------------------------------------------------------------------------------------- +-------------------------------------------------- +JIT compiled ops requires ninjaDeepSpeed C++/CUDA extension op report +-------------------------------------------------- + +DeepSpeed C++/CUDA extension op report-------------------------------------------------- + + +DeepSpeed C++/CUDA extension op reportNOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- + + +----------------------------------------------------------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.JIT compiled ops requires ninja +-------------------------------------------------- +-------------------------------------------------- + +JIT compiled ops requires ninjaJIT compiled ops requires ninja + +ninjaninjaninjaninja .................................... .................. ..................[OKAY][OKAY][OKAY] + +[OKAY] +-------------------------------------------------- +---------------------------------------------------------------------------------------------------- + +-------------------------------------------------- +op nameop nameop name + ................op name................................ ................installedinstalledinstalled ..installed.... compatible +compatiblecompatible..-------------------------------------------------- + + +--------------------------------------------------compatible-------------------------------------------------- + + +-------------------------------------------------- +cpu_adam ............... cpu_adamcpu_adam .............................. [NO][NO] .............. [OKAY][OKAY] + +[NO]cpu_adam ...................... fused_adam[NO] fused_adam ............. ....... [OKAY]............. [NO] + [OKAY] [NO] +....... .......[OKAY] +[OKAY] +fused_lamb fused_lamb............. .............[NO] fused_adam [NO] ....... ............. ....... [OKAY] [NO] +fused_adam[OKAY] +....... [OKAY] + fused_lamb............. .............sparse_attn [NO]............sparse_attn .......[NO]............ [NO][OKAY][NO] +....... [OKAY] +transformer....... ................... [NO] .......sparse_attn[OKAY] +............[OKAY][OKAY] + +[NO] fused_lamb....... transformer ............. [OKAY]stochastic_transformer............ + [NO] transformer........ ............[NO] [OKAY][NO] + ....... [OKAY] +stochastic_transformer [NO]........ stochastic_transformer .......[OKAY][NO]. + .......[NO] [OKAY]....... + [OKAY][OKAY] + +sparse_attn ............ [NO] ....... [OKAY] +transformer ............ [NO] ....... [OKAY] +stochastic_transformer . [NO] ....... [OKAY] +---------------------------------------------------------------------------------------------------- +--------------------------------------------------DeepSpeed C++/CUDA extension op report + + +DeepSpeed C++/CUDA extension op report--------------------------------------------------DeepSpeed C++/CUDA extension op report + + +--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + + +--------------------------------------------------JIT compiled ops requires ninja--------------------------------------------------DeepSpeed C++/CUDA extension op report + + + +JIT compiled ops requires ninjaJIT compiled ops requires ninja +-------------------------------------------------- + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +ninja .................. [OKAY]ninja + --------------------------------------------------.................. + [OKAY]op name + ................-------------------------------------------------- +installedop name .................. compatibleinstalled + ..-------------------------------------------------- +compatible +-------------------------------------------------- +cpu_adam ...............cpu_adam [NO]............... [NO]....... ....... [OKAY][OKAY] + +fused_adamfused_adam .......................... [NO][NO] .............. [OKAY][OKAY] + +fused_lambfused_lamb .......................... [NO][NO] .............. [OKAY] +[OKAY] +sparse_attnsparse_attn ............ ............[NO] .......[NO] [OKAY]....... + [OKAY]transformer + ............ [NO]transformer ................... [OKAY] +[NO] ....... [OKAY]stochastic_transformer + . stochastic_transformer[NO] ....... .[OKAY] + [NO] ....... [OKAY] + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +ninja .................. [OKAY] +-------------------------------------------------- +op name ................ installed .. compatible +-------------------------------------------------- +cpu_adam ............... [NO] ....... [OKAY] +ninjafused_adam ............................... [NO][OKAY] +.......-------------------------------------------------- +[OKAY] +op name ................ installedfused_lamb ............... compatible +[NO] --------------------------------------------------....... + [OKAY] +cpu_adam ............... [NO] .......sparse_attn [OKAY]............ + [NO] ....... [OKAY] +transformer ............ [NO] .......fused_adam [OKAY]............. + [NO] ....... [OKAY]stochastic_transformer + .fused_lamb [NO]............. [NO]....... .......[OKAY] +[OKAY] +sparse_attn ............ [NO] ....... [OKAY] +transformer ............ [NO] ....... [OKAY] +stochastic_transformer . [NO] ....... [OKAY] +---------------------------------------------------------------------------------------------------- + +DeepSpeed C++/CUDA extension op report +DeepSpeed C++/CUDA extension op report +-------------------------------------------------- +--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- +------------------------------------------------------------------------------------------------------------------------------------------------------ + + +DeepSpeed C++/CUDA extension op report + +JIT compiled ops requires ninjaJIT compiled ops requires ninjaDeepSpeed C++/CUDA extension op report + +-------------------------------------------------- + +--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.-------------------------------------------------- + +--------------------------------------------------JIT compiled ops requires ninja + +JIT compiled ops requires ninja +ninjaninjaninjaninja ........................................................................ [OKAY][OKAY][OKAY] +[OKAY] + +---------------------------------------------------------------------------------------------------- + +-------------------------------------------------- +-------------------------------------------------- + +op nameop nameop name op name ................................ ................installed................ installed installed installed.. .. ..compatible..compatible + + compatible----------------------------------------------------------------------------------------------------compatible + + + +---------------------------------------------------------------------------------------------------- + +cpu_adamcpu_adam ..............................cpu_adamcpu_adam [NO] [NO].............................. .......[NO][NO]....... ....... [OKAY].......[OKAY] + +[OKAY][OKAY] + +fused_adamfused_adam .............fused_adam............. fused_adam [NO]............. [NO] ............. .......[NO].......[NO] .......[OKAY]....... +[OKAY] [OKAY] +fused_lamb[OKAY] + +............. [NO]fused_lambfused_lamb fused_lamb....... ............. .............[OKAY] ............. + [NO] [NO] [NO]....... ..............[OKAY] +[OKAY][OKAY] + +sparse_attn ............ [NO] ....... [OKAY]sparse_attn +DeepSpeed general environment info: + ............sparse_attntransformersparse_attn [NO] ............ ............................... [NO] .......[NO][OKAY][NO] + [OKAY].............. +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +transformer [OKAY][OKAY]............ +stochastic_transformer +torch cuda version ............... 11.1 + [NO]transformer.transformer .......[NO]........................ ....... [NO][OKAY][NO] +nvcc version ..................... 11.2 + [OKAY] ....... + .......stochastic_transformer[OKAY] +[OKAY] +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +. [NO] stochastic_transformer.......stochastic_transformer [OKAY]. +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +. [NO][NO] .............. [OKAY][OKAY] + +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... DeepSpeed general environment info:1.8.1 + +torch cuda version ............... 11.1 +torch install pathnvcc version .................................... 11.2 +deepspeed install path ........... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed infotorch version ....................................... 0.5.5+29bee73, 29bee73, master1.8.1 + +deepspeed wheel compiled w. ......torch cuda version torch 1.8, cuda 11.1............... + 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +transformer_inference .. [NO] ....... [OKAY] +async_io ...............utils .................. [NO] ....... [OKAY] + [NO]quantizer .............. [NO] ....... [OKAY] +....... [NO] +-------------------------------------------------- +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum [WARNING]  async_io: please install the libaio-devel package with yum + + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_ioasync_io .............................. [NO][NO]async_io ....... ....... ............... [NO][NO] + +[NO] ....... [NO] +transformer_inferencetransformer_inference ..transformer_inference.. [NO] ..[NO] .......[NO]....... [OKAY].......[OKAY] + +[OKAY] +utils utils..................utils ..................[NO].................. [NO][NO]....... ..............[OKAY] +[OKAY][OKAY] + +quantizer ..............quantizer quantizer [NO] .............. .............. ....... [NO] [NO] [OKAY] +.............. [OKAY][OKAY] + +-------------------------------------------------- +-------------------------------------------------- +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO]async_io ....... ...............[NO] +[NO] ....... [NO] +transformer_inference ..transformer_inference [NO].. .......[NO] [OKAY]....... + [OKAY] +utils ..................utils [NO].................. .......[NO] [OKAY]....... + [OKAY] [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +quantizer + .............. quantizer[NO] ..................... [NO][OKAY] +....... [OKAY] +-------------------------------------------------- +async_io-------------------------------------------------- +............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + +DeepSpeed general environment info:DeepSpeed general environment info: + +torch install pathtorch install path .............................. ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch']['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] + +torch versiontorch version ........................................ 1.8.11.8.1 + +torch cuda versiontorch cuda version .............................. 11.111.1 + +nvcc versionnvcc version .......................................... 11.211.2 + +deepspeed install pathdeepspeed install path ...................... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed']['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1DeepSpeed general environment info: +nvcc version +..................... 11.2 +deepspeed install path torch install path........... ............... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] + +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1torch version + .................... 1.8.1 +torch cuda version ............... 11.1 +DeepSpeed general environment info:nvcc version ..................... + 11.2 +deepspeed install path ...........torch install path ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed']............... + deepspeed info ................... 0.5.5+29bee73, 29bee73, master +['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch']deepspeed wheel compiled w. + ...... torch versiontorch 1.8, cuda 11.1 +.................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +-------------------------------------------------- +DeepSpeed C++/CUDA extension op report +---------------------------------------------------------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + +------------------------------------------------------------------------------------------------------------------------------------------------------DeepSpeed C++/CUDA extension op report + +JIT compiled ops requires ninja + + +--------------------------------------------------DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op report + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.---------------------------------------------------------------------------------------------------- + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.--------------------------------------------------NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + + +--------------------------------------------------JIT compiled ops requires ninja-------------------------------------------------- + + +JIT compiled ops requires ninjaJIT compiled ops requires ninja + +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +ninjaninjaninja ninja.................................... [OKAY]..................[OKAY].................. + + --------------------------------------------------[OKAY][OKAY]-------------------------------------------------- + + + +op name-------------------------------------------------- +op name--------------------------------------------------................ op name + ................installedop name................ installed ..................installed ..compatibleinstalled +.. -------------------------------------------------- compatible.. +compatible + +compatible---------------------------------------------------------------------------------------------------- + + +-------------------------------------------------- +cpu_adam ............... [NO] .......cpu_adam cpu_adam cpu_adam[OKAY] .............................. + ...............[NO][NO] [NO]....... ....... fused_adam[OKAY] +.......[OKAY] +[OKAY]............. + [NO] ....... [OKAY] +fused_adam fused_lamb.............fused_adam fused_adam.............[NO] ................................. [NO][NO] [OKAY] +[NO].............. [OKAY] + fused_lamb.......[OKAY] +.............[OKAY] +[NO]fused_lamb ....... fused_lamb.............[OKAY] sparse_attn + ............. [NO] ............ [NO] ....... [NO] ....... [OKAY] ....... + [OKAY]sparse_attn[OKAY] + +............ [NO]transformer ................... [OKAY][NO] +....... sparse_attntransformer[OKAY] sparse_attn +........................ [NO] ............[NO].......stochastic_transformer ....... [NO] [OKAY][OKAY] . + + .......[NO]transformer stochastic_transformer.......[OKAY]............ + . [OKAY][NO]transformer[NO] ....... + ............ ....... [OKAY][NO][OKAY] + +....... [OKAY]stochastic_transformer + . [NO] stochastic_transformer....... [OKAY] +. [NO] ....... [OKAY] + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +DeepSpeed general environment info: +torch install pathDeepSpeed general environment info: +............... torch install path ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch']............... + torch version .................... 1.8.1['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] + +torch cuda version torch version............... ....................11.1 +1.8.1nvcc version + .....................torch cuda version 11.2............... + deepspeed install path11.1 +...........nvcc version ..................... 11.2['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] + +deepspeed install pathdeepspeed info .............................. 0.5.5+29bee73, 29bee73, master +['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed']deepspeed wheel compiled w. + ......deepspeed info torch 1.8, cuda 11.1................... + 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +async_io ...............async_io [NO] ...................... [NO][NO] +....... [NO] +transformer_inferencetransformer_inference .... [NO][NO] .............. [OKAY][OKAY] + +utilsutils .................................... [NO][NO] .............. [OKAY][OKAY] + +quantizer quantizer.............. ..............[NO] [NO]....... .......[OKAY] +[OKAY] +-------------------------------------------------- +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +---------------------------------------------------------------------------------------------------- + +DeepSpeed C++/CUDA extension op reportDeepSpeed C++/CUDA extension op report + +------------------------------------------------------------------------------------------------------------------------------------------------------ + + +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op.DeepSpeed C++/CUDA extension op report + + +------------------------------------------------------------------------------------------------------------------------------------------------------ + +JIT compiled ops requires ninja +--------------------------------------------------JIT compiled ops requires ninja +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. + + +--------------------------------------------------DeepSpeed C++/CUDA extension op report + +JIT compiled ops requires ninja +-------------------------------------------------- +NOTE: Ops not installed will be just-in-time (JIT) compiled at + runtime if needed. Op compatibility means that your system + meet the required dependencies to JIT install the op. +-------------------------------------------------- +JIT compiled ops requires ninja +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum [WARNING]  async_io: please install the libaio-devel package with yum + + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_ioasync_io ............... ...............[NO] [NO]....... .......[NO] +[NO] +transformer_inferencetransformer_inference .... [NO][NO] .............. [OKAY][OKAY] + +utils ..................utils [NO].................. .......[NO] [OKAY]....... + [OKAY] +quantizer .............. [NO]quantizer ..................... [OKAY][NO] + ....... [OKAY] +-------------------------------------------------- +-------------------------------------------------- +ninjaninjaninjaninja .................................... .................................... [OKAY] [OKAY][OKAY] + +[OKAY] + +------------------------------------------------------------------------------------------------------------------------------------------------------ + +-------------------------------------------------- +op nameop name +op name op name ................................ ................ ................ installedinstalledinstalled .... installed .. compatiblecompatible .. +compatible + +----------------------------------------------------------------------------------------------------compatible-------------------------------------------------- + + + +-------------------------------------------------- +cpu_adamcpu_adamcpu_adam ..............................cpu_adam............... [NO][NO][NO] ............... ....... ....... [NO] .......[OKAY][OKAY] + +.......[OKAY] +[OKAY] +fused_adamfused_adam .......................... [NO]fused_adam[NO] fused_adam........................... [OKAY].............[NO][OKAY] + +.......[NO] fused_lamb [OKAY] fused_lamb....... +............. .............[OKAY][NO] +fused_lamb [NO] .......fused_lamb.................... [OKAY] .............[NO][OKAY] +[NO] +....... .......[OKAY] + [OKAY] +sparse_attnsparse_attn ........................ [NO][NO] .............. [OKAY]sparse_attn[OKAY] +sparse_attn + ............transformer............ transformer [NO] ............ ............ ....... [NO] [NO][NO].......[OKAY] + ..............[OKAY] +transformer[OKAY][OKAY] transformer + +............ ............[NO] [NO]stochastic_transformer.......stochastic_transformer [OKAY] ....... + ..[OKAY] +[NO][NO]stochastic_transformer .............. .stochastic_transformer[OKAY][OKAY] +[NO] + ....... .[OKAY] +[NO] ....... [OKAY] + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + +async_ioasync_io .............................. [NO][NO] .............. [NO][NO] + +transformer_inference .. transformer_inference[NO] ......... [NO][OKAY] +....... [OKAY] +utils ..................utils [NO].................. .......[NO] [OKAY]....... + [OKAY] +quantizer quantizer.............. ..............[NO] [NO]....... .......[OKAY] +[OKAY] +---------------------------------------------------------------------------------------------------- + + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + + [WARNING]  async_io: please install the libaio-devel package with yum [WARNING]  async_io: please install the libaio-devel package with yum + + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + +async_io ............... [NO]async_io ....... [NO]............... + [NO] ....... [NO] +transformer_inference .. [NO] ....... transformer_inference[OKAY] +.. [NO] ....... utils[OKAY] +.................. [NO] ....... utils[OKAY] +.................. [NO] quantizer....... ..............[OKAY] +[NO] ....... [OKAY]quantizer + .............. [NO] --------------------------------------------------....... + [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [WARNING]  async_io: please install the libaio-devel package with yum [OKAY] + +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version DeepSpeed general environment info:..................... 11.2 + +deepspeed install path ........... torch install path['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] + deepspeed info............... ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ......['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] torch 1.8, cuda 11.1 + +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +DeepSpeed general environment info:DeepSpeed general environment info: + +torch install pathtorch install path .............................. ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch']['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] + +torch version torch version.................... ....................1.8.1 +1.8.1 +torch cuda version torch cuda version............... ...............11.1 +11.1nvcc version + nvcc version..................... .....................11.2 +11.2deepspeed install path + deepspeed install path........... ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed']deepspeed info + deepspeed info................... ...................0.5.5+29bee73, 29bee73, master +0.5.5+29bee73, 29bee73, masterdeepspeed wheel compiled w. + deepspeed wheel compiled w....... ......torch 1.8, cuda 11.1 +torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ...............DeepSpeed general environment info: +torch install path['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] + ............... torch version .................... 1.8.1 +['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch']torch cuda version + ............... torch version11.1 +....................nvcc version 1.8.1..................... + 11.2 +torch cuda version deepspeed install path............... ...........11.1 +nvcc version ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed']..................... + 11.2deepspeed info + deepspeed install path................... ...........0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed']...... + deepspeed infotorch 1.8, cuda 11.1 +................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... async_io[NO] + ............... [NO] ....... [NO] + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.transformer_inference + .. [NO] ....... [OKAY] +transformer_inference ..utils [NO]async_io.................. ....... [NO] ............... [OKAY] ....... +[NO] [OKAY]....... + [NO]utils + quantizer.................. ..............[NO] [NO]....... .......[OKAY] +[OKAY] +quantizer ..............--------------------------------------------------transformer_inference + [NO] ......... [NO][OKAY] +....... [OKAY] +-------------------------------------------------- +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info:DeepSpeed general environment info: + +torch install pathtorch install path .............................. ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch']['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] + +torch versiontorch version ........................................ 1.8.11.8.1 + +torch cuda versiontorch cuda version .............................. 11.111.1 + +nvcc versionnvcc version .......................................... 11.211.2 + +deepspeed install pathdeepspeed install path ...................... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed']['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] + +deepspeed infodeepspeed info ...................................... 0.5.5+29bee73, 29bee73, master0.5.5+29bee73, 29bee73, master + +deepspeed wheel compiled w.deepspeed wheel compiled w. ............ torch 1.8, cuda 11.1torch 1.8, cuda 11.1 + + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + ............... [NO] ....... [NO] +async_iotransformer_inference ................. [NO][NO] .............. [OKAY][NO] + +utils .................. [NO] ....... [OKAY] +quantizer ..............transformer_inference [NO].. .......[NO] [OKAY]....... + [OKAY] +-------------------------------------------------- +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum [WARNING]  async_io: please install the libaio-devel package with yum + + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO]async_io ....... ...............[NO] +[NO] ....... [NO] +transformer_inference ..transformer_inference [NO].. .......[NO] [OKAY]....... + [OKAY] +utils ..................utils [NO] ......................... [NO][OKAY] +....... [OKAY] +quantizer .............. [NO]quantizer ..................... [OKAY][NO] + ....... [OKAY] +-------------------------------------------------- +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +....... [NO] +async_io transformer_inference............... ..[NO] [NO]....... .......[NO] +[OKAY] +utils .................. [NO] ....... [OKAY]transformer_inference + .. [NO]quantizer ..................... [OKAY][NO] + ....... [OKAY] +utils ..................-------------------------------------------------- +[NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info:DeepSpeed general environment info:DeepSpeed general environment info: + + +torch install pathtorch install pathtorch install path ............................................. ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch']['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch']['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] + + +torch versiontorch versiontorch version ............................................................ 1.8.11.8.11.8.1 + + +torch cuda versiontorch cuda versiontorch cuda version ............................................. 11.111.111.1 + + +nvcc versionnvcc versionnvcc version ............................................................... 11.211.211.2 + + +deepspeed install pathdeepspeed install pathdeepspeed install path ................................. ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed']['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed']['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] + + +deepspeed infodeepspeed infodeepspeed info ......................................................... 0.5.5+29bee73, 29bee73, master0.5.5+29bee73, 29bee73, master0.5.5+29bee73, 29bee73, master + + +deepspeed wheel compiled w.deepspeed wheel compiled w.deepspeed wheel compiled w. .................. torch 1.8, cuda 11.1torch 1.8, cuda 11.1torch 1.8, cuda 11.1 + + +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info:DeepSpeed general environment info: + +torch install pathtorch install path .............................. ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch']['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] + +torch versiontorch version ........................................ 1.8.11.8.1 + +torch cuda versiontorch cuda version .............................. 11.111.1 + +nvcc versionnvcc version .......................................... 11.211.2 + +deepspeed install pathdeepspeed install path ...................... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed']['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] + +deepspeed infodeepspeed info ...................................... 0.5.5+29bee73, 29bee73, master0.5.5+29bee73, 29bee73, master + +deepspeed wheel compiled w.deepspeed wheel compiled w. ............ torch 1.8, cuda 11.1torch 1.8, cuda 11.1 + + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info:DeepSpeed general environment info: + +torch install pathtorch install path ............... ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version ....................torch version 1.8.1.................... + 1.8.1 +torch cuda version ...............torch cuda version 11.1............... + 11.1nvcc version + .....................nvcc version 11.2..................... + deepspeed install path11.2 +...........deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed']deepspeed info + ...................deepspeed info 0.5.5+29bee73, 29bee73, master................... + 0.5.5+29bee73, 29bee73, masterdeepspeed wheel compiled w. + ......deepspeed wheel compiled w. torch 1.8, cuda 11.1...... + torch 1.8, cuda 11.1 + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +DeepSpeed general environment info: +torch install path ............... ['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] +torch version .................... 1.8.1 +torch cuda version ............... 11.1 +nvcc version ..................... 11.2 +deepspeed install path ........... ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. +async_io ............... [NO] ....... [NO] +transformer_inference .. [NO] ....... [OKAY] +utils .................. [NO] ....... [OKAY] +quantizer .............. [NO] ....... [OKAY] +-------------------------------------------------- +DeepSpeed general environment info: +torch install path ............... DeepSpeed general environment info:['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch'] + +torch version ....................torch install path 1.8.1 + ............... torch cuda version ............... 11.1 +['/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch']nvcc version + ..................... torch version11.2 +....................deepspeed install path 1.8.1........... + torch cuda version ['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed']............... + deepspeed info11.1 +...................nvcc version 0.5.5+29bee73, 29bee73, master..................... + 11.2deepspeed wheel compiled w. + deepspeed install path...... ...........torch 1.8, cuda 11.1 +['/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/DeepSpeed/deepspeed'] +deepspeed info ................... 0.5.5+29bee73, 29bee73, master +deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1 +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main ******** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** + +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main ******** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** + +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main ******** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** + +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +**** Git info for Megatron: git_hash=bdc6ad6 git_branch=main **** +> setting tensorboard ... +> initializing tensor model parallel with size 4 +> initializing pipeline model parallel with size 32 +> setting random seeds to 43 ... +[2021-10-22 18:28:35,800] [INFO] [checkpointing.py:226:model_parallel_cuda_manual_seed] > initializing model parallel cuda seeds on global rank 0, model parallel rank 0, and data parallel rank 0 with model parallel seed: 2761 and data parallel seed: 43 +> compiling dataset index builder ... +make: Entering directory '/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed/megatron/data' +make: Nothing to be done for 'default'. +make: Leaving directory '/gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed/megatron/data' +>>> done with dataset index builder. Compilation time: 0.303 seconds +> compiling and loading fused kernels ... +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +Detected CUDA files, patching ldflags +Emitting ninja build file /gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed/megatron/fused_kernels/build/build.ninja... +Building extension module scaled_upper_triang_masked_softmax_cuda... +Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) +ninja: no work to do. +Loading extension module scaled_upper_triang_masked_softmax_cuda... +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +Detected CUDA files, patching ldflags +Emitting ninja build file /gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed/megatron/fused_kernels/build/build.ninja... +Building extension module scaled_masked_softmax_cuda... +Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) +ninja: no work to do. +Loading extension module scaled_masked_softmax_cuda... +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +Detected CUDA files, patching ldflags +Emitting ninja build file /gpfsssd/worksf/projects/rech/six/commun/code/tr8b-104B/Megatron-DeepSpeed/megatron/fused_kernels/build/build.ninja... +Building extension module fused_mix_prec_layer_norm_cuda... +Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) +ninja: no work to do. +Loading extension module fused_mix_prec_layer_norm_cuda... +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +/gpfswork/rech/six/commun/conda/cutting-edge/lib/python3.8/site-packages/torch/utils/cpp_extension.py:283: UserWarning: + + !! WARNING !! + +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +Your compiler (c++) is not compatible with the compiler Pytorch was +built with for this platform, which is g++ on linux. Please +use g++ to to compile your extension. Alternatively, you may +compile PyTorch from source using c++, and then you can also use +c++ to compile your extension. + +See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help +with compiling PyTorch from source. +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + + !! WARNING !! + + warnings.warn(WRONG_COMPILER_WARNING.format( +>>> done with compiling and loading fused kernels. Compilation time: 25.888 seconds +time to initialize megatron (seconds): 78.047 +[after megatron is initialized] datetime: 2021-10-22 18:29:02 +building GPT model ... +[2021-10-22 18:29:02,120] [INFO] [utils.py:806:see_memory_usage] Before Building Model +[2021-10-22 18:29:02,121] [INFO] [utils.py:807:see_memory_usage] MA 0.0 GB Max_MA 0.0 GB CA 0.0 GB Max_CA 0 GB +[2021-10-22 18:29:02,121] [INFO] [utils.py:815:see_memory_usage] CPU Virtual Memory: used = 40.03 GB, percent = 21.4% +SEED_LAYERS=False BASE_SEED=1234 SEED_FN=None +Using topology: {ProcessCoord(pipe=0, data=0, model=0): 0, ProcessCoord(pipe=0, data=0, model=1): 1, ProcessCoord(pipe=0, data=0, model=2): 2, ProcessCoord(pipe=0, data=0, model=3): 3, ProcessCoord(pipe=1, data=0, model=0): 4, ProcessCoord(pipe=1, data=0, model=1): 5, ProcessCoord(pipe=1, data=0, model=2): 6, ProcessCoord(pipe=1, data=0, model=3): 7, ProcessCoord(pipe=2, data=0, model=0): 8, ProcessCoord(pipe=2, data=0, model=1): 9, ProcessCoord(pipe=2, data=0, model=2): 10, ProcessCoord(pipe=2, data=0, model=3): 11, ProcessCoord(pipe=3, data=0, model=0): 12, ProcessCoord(pipe=3, data=0, model=1): 13, ProcessCoord(pipe=3, data=0, model=2): 14, ProcessCoord(pipe=3, data=0, model=3): 15, ProcessCoord(pipe=4, data=0, model=0): 16, ProcessCoord(pipe=4, data=0, model=1): 17, ProcessCoord(pipe=4, data=0, model=2): 18, ProcessCoord(pipe=4, data=0, model=3): 19, ProcessCoord(pipe=5, data=0, model=0): 20, ProcessCoord(pipe=5, data=0, model=1): 21, ProcessCoord(pipe=5, data=0, model=2): 22, ProcessCoord(pipe=5, data=0, model=3): 23, ProcessCoord(pipe=6, data=0, model=0): 24, ProcessCoord(pipe=6, data=0, model=1): 25, ProcessCoord(pipe=6, data=0, model=2): 26, ProcessCoord(pipe=6, data=0, model=3): 27, ProcessCoord(pipe=7, data=0, model=0): 28, ProcessCoord(pipe=7, data=0, model=1): 29, ProcessCoord(pipe=7, data=0, model=2): 30, ProcessCoord(pipe=7, data=0, model=3): 31, ProcessCoord(pipe=8, data=0, model=0): 32, ProcessCoord(pipe=8, data=0, model=1): 33, ProcessCoord(pipe=8, data=0, model=2): 34, ProcessCoord(pipe=8, data=0, model=3): 35, ProcessCoord(pipe=9, data=0, model=0): 36, ProcessCoord(pipe=9, data=0, model=1): 37, ProcessCoord(pipe=9, data=0, model=2): 38, ProcessCoord(pipe=9, data=0, model=3): 39, ProcessCoord(pipe=10, data=0, model=0): 40, ProcessCoord(pipe=10, data=0, model=1): 41, ProcessCoord(pipe=10, data=0, model=2): 42, ProcessCoord(pipe=10, data=0, model=3): 43, ProcessCoord(pipe=11, data=0, model=0): 44, ProcessCoord(pipe=11, data=0, model=1): 45, ProcessCoord(pipe=11, data=0, model=2): 46, ProcessCoord(pipe=11, data=0, model=3): 47, ProcessCoord(pipe=12, data=0, model=0): 48, ProcessCoord(pipe=12, data=0, model=1): 49, ProcessCoord(pipe=12, data=0, model=2): 50, ProcessCoord(pipe=12, data=0, model=3): 51, ProcessCoord(pipe=13, data=0, model=0): 52, ProcessCoord(pipe=13, data=0, model=1): 53, ProcessCoord(pipe=13, data=0, model=2): 54, ProcessCoord(pipe=13, data=0, model=3): 55, ProcessCoord(pipe=14, data=0, model=0): 56, ProcessCoord(pipe=14, data=0, model=1): 57, ProcessCoord(pipe=14, data=0, model=2): 58, ProcessCoord(pipe=14, data=0, model=3): 59, ProcessCoord(pipe=15, data=0, model=0): 60, ProcessCoord(pipe=15, data=0, model=1): 61, ProcessCoord(pipe=15, data=0, model=2): 62, ProcessCoord(pipe=15, data=0, model=3): 63, ProcessCoord(pipe=16, data=0, model=0): 64, ProcessCoord(pipe=16, data=0, model=1): 65, ProcessCoord(pipe=16, data=0, model=2): 66, ProcessCoord(pipe=16, data=0, model=3): 67, ProcessCoord(pipe=17, data=0, model=0): 68, ProcessCoord(pipe=17, data=0, model=1): 69, ProcessCoord(pipe=17, data=0, model=2): 70, ProcessCoord(pipe=17, data=0, model=3): 71, ProcessCoord(pipe=18, data=0, model=0): 72, ProcessCoord(pipe=18, data=0, model=1): 73, ProcessCoord(pipe=18, data=0, model=2): 74, ProcessCoord(pipe=18, data=0, model=3): 75, ProcessCoord(pipe=19, data=0, model=0): 76, ProcessCoord(pipe=19, data=0, model=1): 77, ProcessCoord(pipe=19, data=0, model=2): 78, ProcessCoord(pipe=19, data=0, model=3): 79, ProcessCoord(pipe=20, data=0, model=0): 80, ProcessCoord(pipe=20, data=0, model=1): 81, ProcessCoord(pipe=20, data=0, model=2): 82, ProcessCoord(pipe=20, data=0, model=3): 83, ProcessCoord(pipe=21, data=0, model=0): 84, ProcessCoord(pipe=21, data=0, model=1): 85, ProcessCoord(pipe=21, data=0, model=2): 86, ProcessCoord(pipe=21, data=0, model=3): 87, ProcessCoord(pipe=22, data=0, model=0): 88, ProcessCoord(pipe=22, data=0, model=1): 89, ProcessCoord(pipe=22, data=0, model=2): 90, ProcessCoord(pipe=22, data=0, model=3): 91, ProcessCoord(pipe=23, data=0, model=0): 92, ProcessCoord(pipe=23, data=0, model=1): 93, ProcessCoord(pipe=23, data=0, model=2): 94, ProcessCoord(pipe=23, data=0, model=3): 95, ProcessCoord(pipe=24, data=0, model=0): 96, ProcessCoord(pipe=24, data=0, model=1): 97, ProcessCoord(pipe=24, data=0, model=2): 98, ProcessCoord(pipe=24, data=0, model=3): 99, ProcessCoord(pipe=25, data=0, model=0): 100, ProcessCoord(pipe=25, data=0, model=1): 101, ProcessCoord(pipe=25, data=0, model=2): 102, ProcessCoord(pipe=25, data=0, model=3): 103, ProcessCoord(pipe=26, data=0, model=0): 104, ProcessCoord(pipe=26, data=0, model=1): 105, ProcessCoord(pipe=26, data=0, model=2): 106, ProcessCoord(pipe=26, data=0, model=3): 107, ProcessCoord(pipe=27, data=0, model=0): 108, ProcessCoord(pipe=27, data=0, model=1): 109, ProcessCoord(pipe=27, data=0, model=2): 110, ProcessCoord(pipe=27, data=0, model=3): 111, ProcessCoord(pipe=28, data=0, model=0): 112, ProcessCoord(pipe=28, data=0, model=1): 113, ProcessCoord(pipe=28, data=0, model=2): 114, ProcessCoord(pipe=28, data=0, model=3): 115, ProcessCoord(pipe=29, data=0, model=0): 116, ProcessCoord(pipe=29, data=0, model=1): 117, ProcessCoord(pipe=29, data=0, model=2): 118, ProcessCoord(pipe=29, data=0, model=3): 119, ProcessCoord(pipe=30, data=0, model=0): 120, ProcessCoord(pipe=30, data=0, model=1): 121, ProcessCoord(pipe=30, data=0, model=2): 122, ProcessCoord(pipe=30, data=0, model=3): 123, ProcessCoord(pipe=31, data=0, model=0): 124, ProcessCoord(pipe=31, data=0, model=1): 125, ProcessCoord(pipe=31, data=0, model=2): 126, ProcessCoord(pipe=31, data=0, model=3): 127} +[2021-10-22 18:29:03,802] [INFO] [module.py:365:_partition_layers] Partitioning pipeline stages with method type:transformer +stage=0 layers=5 + 0: _to_float16 + 1: EmbeddingPipe + 2: + 3: ParallelTransformerLayerPipe + 4: ParallelTransformerLayerPipe +stage=1 layers=2 + 5: ParallelTransformerLayerPipe + 6: ParallelTransformerLayerPipe +stage=2 layers=2 + 7: ParallelTransformerLayerPipe + 8: ParallelTransformerLayerPipe +stage=3 layers=2 + 9: ParallelTransformerLayerPipe + 10: ParallelTransformerLayerPipe +stage=4 layers=2 + 11: ParallelTransformerLayerPipe + 12: ParallelTransformerLayerPipe +stage=5 layers=2 + 13: ParallelTransformerLayerPipe + 14: ParallelTransformerLayerPipe +stage=6 layers=2 + 15: ParallelTransformerLayerPipe + 16: ParallelTransformerLayerPipe +stage=7 layers=2 + 17: ParallelTransformerLayerPipe + 18: ParallelTransformerLayerPipe +stage=8 layers=2 + 19: ParallelTransformerLayerPipe + 20: ParallelTransformerLayerPipe +stage=9 layers=2 + 21: ParallelTransformerLayerPipe + 22: ParallelTransformerLayerPipe +stage=10 layers=2 + 23: ParallelTransformerLayerPipe + 24: ParallelTransformerLayerPipe +stage=11 layers=2 + 25: ParallelTransformerLayerPipe + 26: ParallelTransformerLayerPipe +stage=12 layers=2 + 27: ParallelTransformerLayerPipe + 28: ParallelTransformerLayerPipe +stage=13 layers=2 + 29: ParallelTransformerLayerPipe + 30: ParallelTransformerLayerPipe +stage=14 layers=2 + 31: ParallelTransformerLayerPipe + 32: ParallelTransformerLayerPipe +stage=15 layers=2 + 33: ParallelTransformerLayerPipe + 34: ParallelTransformerLayerPipe +stage=16 layers=2 + 35: ParallelTransformerLayerPipe + 36: ParallelTransformerLayerPipe +stage=17 layers=2 + 37: ParallelTransformerLayerPipe + 38: ParallelTransformerLayerPipe +stage=18 layers=2 + 39: ParallelTransformerLayerPipe + 40: ParallelTransformerLayerPipe +stage=19 layers=2 + 41: ParallelTransformerLayerPipe + 42: ParallelTransformerLayerPipe +stage=20 layers=2 + 43: ParallelTransformerLayerPipe + 44: ParallelTransformerLayerPipe +stage=21 layers=2 + 45: ParallelTransformerLayerPipe + 46: ParallelTransformerLayerPipe +stage=22 layers=2 + 47: ParallelTransformerLayerPipe + 48: ParallelTransformerLayerPipe +stage=23 layers=2 + 49: ParallelTransformerLayerPipe + 50: ParallelTransformerLayerPipe +stage=24 layers=2 + 51: ParallelTransformerLayerPipe + 52: ParallelTransformerLayerPipe +stage=25 layers=2 + 53: ParallelTransformerLayerPipe + 54: ParallelTransformerLayerPipe +stage=26 layers=2 + 55: ParallelTransformerLayerPipe + 56: ParallelTransformerLayerPipe +stage=27 layers=2 + 57: ParallelTransformerLayerPipe + 58: ParallelTransformerLayerPipe +stage=28 layers=2 + 59: ParallelTransformerLayerPipe + 60: ParallelTransformerLayerPipe +stage=29 layers=2 + 61: ParallelTransformerLayerPipe + 62: ParallelTransformerLayerPipe +stage=30 layers=2 + 63: ParallelTransformerLayerPipe + 64: ParallelTransformerLayerPipe +stage=31 layers=6 + 65: ParallelTransformerLayerPipe + 66: ParallelTransformerLayerPipe + 67: + 68: MixedFusedLayerNorm + 69: EmbeddingPipe + 70: float16_to_fp32 + loss: CrossEntropy + > number of parameters on (tensor, pipeline) model parallel rank (3, 6): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (3, 19): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (1, 29): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 17): 807539800 > number of parameters on (tensor, pipeline) model parallel rank (1, 17): 807539800 > number of parameters on (tensor, pipeline) model parallel rank (3, 17): 807539800 + + + > number of parameters on (tensor, pipeline) model parallel rank (0, 17): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (1, 2): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 19): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (3, 21): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 10): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 22): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (0, 10): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (3, 18): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 21): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (0, 22): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (3, 20): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 8): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (0, 25): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 25): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 24): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (3, 24): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (0, 24): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (1, 24): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (1, 14): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (3, 14): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 14): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (0, 14): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (1, 21): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (0, 1): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (0, 21): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (0, 16): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (0, 3): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 3): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (1, 3): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 16): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (3, 3): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (1, 5): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (3, 5): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (0, 5): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (1, 20): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (0, 20): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 20): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (0, 11): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 29): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (0, 29): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (3, 29): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (1, 11): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (0, 23): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (1, 16): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (3, 16): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (3, 10): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (3, 15): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 15): 807539800 > number of parameters on (tensor, pipeline) model parallel rank (1, 15): 807539800 + + > number of parameters on (tensor, pipeline) model parallel rank (0, 15): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 26): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (1, 9): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (0, 26): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (0, 9): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (3, 26): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (1, 26): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (0, 7): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (1, 22): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (1, 12): 807539800 > number of parameters on (tensor, pipeline) model parallel rank (0, 12): 807539800 + + > number of parameters on (tensor, pipeline) model parallel rank (3, 12): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 12): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (3, 13): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 27): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (3, 11): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (3, 27): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 13): 807539800 > number of parameters on (tensor, pipeline) model parallel rank (1, 13): 807539800 + + > number of parameters on (tensor, pipeline) model parallel rank (0, 13): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (1, 27): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (0, 27): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 9): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (1, 28): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (3, 28): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (3, 23): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (0, 28): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (1, 30): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (1, 23): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (3, 30): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 11): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (3, 9): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 28): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 30): 807539800 > number of parameters on (tensor, pipeline) model parallel rank (0, 30): 807539800 + + > number of parameters on (tensor, pipeline) model parallel rank (3, 2): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 2): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (3, 22): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 23): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 7): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (0, 2): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (1, 7): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (1, 25): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (1, 19): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (3, 25): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (1, 4): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (0, 19): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (3, 4): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (0, 4): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (0, 8): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (3, 8): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 4): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (1, 8): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 5): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (0, 18): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 18): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (1, 18): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (3, 7): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (0, 6): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (3, 1): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (1, 1): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (1, 6): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (1, 10): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 6): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (2, 1): 807539800 + > number of parameters on (tensor, pipeline) model parallel rank (0, 31): 978315000 + > number of parameters on (tensor, pipeline) model parallel rank (1, 31): 978315000 + > number of parameters on (tensor, pipeline) model parallel rank (2, 31): 978315000 + > number of parameters on (tensor, pipeline) model parallel rank (2, 0): 978291800 + > number of parameters on (tensor, pipeline) model parallel rank (1, 0): 978291800 + > number of parameters on (tensor, pipeline) model parallel rank (3, 0): 978291800 + > number of parameters on (tensor, pipeline) model parallel rank (3, 31): 978315000 +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + + + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + + + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + + + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + + + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + + + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + + + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + + + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + + + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + + + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + + + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + + + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + + + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + + + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + + + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + + + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + + + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + + + +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... + + + +[2021-10-22 18:29:04,502] [INFO] [utils.py:806:see_memory_usage] After Building Model +[2021-10-22 18:29:04,503] [INFO] [utils.py:807:see_memory_usage] MA 1.88 GB Max_MA 1.88 GB CA 1.91 GB Max_CA 2 GB +[2021-10-22 18:29:04,503] [INFO] [utils.py:815:see_memory_usage] CPU Virtual Memory: used = 40.2 GB, percent = 21.5% + > number of parameters on (tensor, pipeline) model parallel rank (0, 0): 978291800 +setting training iterations to 292968 +> learning rate decay style: cosine +DeepSpeed is enabled. +[2021-10-22 18:29:04,504] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed info: version=0.5.5+29bee73, git-hash=29bee73, git-branch=master +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root... +[2021-10-22 18:29:04,541] [INFO] [engine.py:207:__init__] DeepSpeed Flops Profiler Enabled: False +[2021-10-22 18:29:04,541] [INFO] [engine.py:862:_configure_optimizer] Removing param_group that has no 'params' in the client Optimizer +[2021-10-22 18:29:04,541] [INFO] [engine.py:868:_configure_optimizer] Using client Optimizer as basic optimizer +[2021-10-22 18:29:04,542] [INFO] [engine.py:884:_configure_optimizer] DeepSpeed Basic Optimizer = FusedAdam +[2021-10-22 18:29:04,542] [INFO] [utils.py:43:is_zero_supported_optimizer] Checking ZeRO support for optimizer=FusedAdam type= +[2021-10-22 18:29:04,542] [INFO] [logging.py:68:log_dist] [Rank 0] Creating fp16 ZeRO stage 1 optimizer +[2021-10-22 18:29:04,542] [INFO] [stage2.py:111:__init__] Reduce bucket size 500000000 +[2021-10-22 18:29:04,542] [INFO] [stage2.py:112:__init__] Allgather bucket size 500000000 +[2021-10-22 18:29:04,542] [INFO] [stage2.py:113:__init__] CPU Offload: False +[2021-10-22 18:29:04,542] [INFO] [stage2.py:114:__init__] Round robin gradient partitioning: False +Using /gpfsdswork/projects/rech/eha/ura81os/.cache/torch_extensions as PyTorch extensions root...