diff --git "a/146m174b400m/3326770.err" "b/146m174b400m/3326770.err"
new file mode 100644--- /dev/null
+++ "b/146m174b400m/3326770.err"
@@ -0,0 +1,2749 @@
+0: Lmod has detected the following error: These module(s) or extension(s) exist
+0: but cannot be loaded as requested: "CrayEnv"
+0:    Try: "module spider CrayEnv" to see how to load the module(s).
+0: 
+0: 
+0: 
+3: Lmod has detected the following error: These module(s) or extension(s) exist
+3: but cannot be loaded as requested: "CrayEnv"
+3:    Try: "module spider CrayEnv" to see how to load the module(s).
+3: 
+3: 
+3: 
+0: Lmod has detected the following error: The following module(s) are unknown:
+0: "suse-repo-deps/sam-default"
+0: 
+0: Please check the spelling or version number. Also try "module spider ..."
+0: It is also possible your cache file is out-of-date; it may help to try:
+0:   $ module --ignore-cache load "suse-repo-deps/sam-default"
+0: 
+0: Also make sure that all modulefiles written in TCL start with the string
+0: #%Module
+0: 
+0: 
+0: 
+6: Lmod has detected the following error: These module(s) or extension(s) exist
+6: but cannot be loaded as requested: "CrayEnv"
+6:    Try: "module spider CrayEnv" to see how to load the module(s).
+6: 
+6: 
+6: 
+0: Lmod has detected the following error: The following module(s) are unknown:
+0: "rocm/sam-5.2.3"
+0: 
+0: Please check the spelling or version number. Also try "module spider ..."
+0: It is also possible your cache file is out-of-date; it may help to try:
+0:   $ module --ignore-cache load "rocm/sam-5.2.3"
+0: 
+0: Also make sure that all modulefiles written in TCL start with the string
+0: #%Module
+0: 
+0: 
+0: 
+7: Lmod has detected the following error: The following module(s) are unknown:
+7: "suse-repo-deps/sam-default"
+7: 
+7: Please check the spelling or version number. Also try "module spider ..."
+7: It is also possible your cache file is out-of-date; it may help to try:
+7:   $ module --ignore-cache load "suse-repo-deps/sam-default"
+7: 
+7: Also make sure that all modulefiles written in TCL start with the string
+7: #%Module
+7: 
+7: 
+7: 
+0: Lmod has detected the following error: The following module(s) are unknown:
+0: "rccl/sam-develop"
+0: 
+0: Please check the spelling or version number. Also try "module spider ..."
+0: It is also possible your cache file is out-of-date; it may help to try:
+0:   $ module --ignore-cache load "rccl/sam-develop"
+0: 
+0: Also make sure that all modulefiles written in TCL start with the string
+0: #%Module
+0: 
+0: 
+0: 
+2: Lmod has detected the following error: The following module(s) are unknown:
+2: "suse-repo-deps/sam-default"
+2: 
+2: Please check the spelling or version number. Also try "module spider ..."
+2: It is also possible your cache file is out-of-date; it may help to try:
+2:   $ module --ignore-cache load "suse-repo-deps/sam-default"
+2: 
+2: Also make sure that all modulefiles written in TCL start with the string
+2: #%Module
+2: 
+2: 
+2: 
+1: Lmod has detected the following error: The following module(s) are unknown:
+1: "suse-repo-deps/sam-default"
+1: 
+1: Please check the spelling or version number. Also try "module spider ..."
+1: It is also possible your cache file is out-of-date; it may help to try:
+1:   $ module --ignore-cache load "suse-repo-deps/sam-default"
+1: 
+1: Also make sure that all modulefiles written in TCL start with the string
+1: #%Module
+1: 
+1: 
+1: 
+3: Lmod has detected the following error: The following module(s) are unknown:
+3: "suse-repo-deps/sam-default"
+3: 
+3: Please check the spelling or version number. Also try "module spider ..."
+3: It is also possible your cache file is out-of-date; it may help to try:
+3:   $ module --ignore-cache load "suse-repo-deps/sam-default"
+3: 
+3: Also make sure that all modulefiles written in TCL start with the string
+3: #%Module
+3: 
+3: 
+3: 
+0: Lmod has detected the following error: The following module(s) are unknown:
+0: "aws-ofi-rccl/sam-default"
+0: 
+0: Please check the spelling or version number. Also try "module spider ..."
+0: It is also possible your cache file is out-of-date; it may help to try:
+0:   $ module --ignore-cache load "aws-ofi-rccl/sam-default"
+0: 
+0: Also make sure that all modulefiles written in TCL start with the string
+0: #%Module
+0: 
+0: 
+0: 
+7: Lmod has detected the following error: The following module(s) are unknown:
+7: "rocm/sam-5.2.3"
+7: 
+7: Please check the spelling or version number. Also try "module spider ..."
+7: It is also possible your cache file is out-of-date; it may help to try:
+7:   $ module --ignore-cache load "rocm/sam-5.2.3"
+7: 
+7: Also make sure that all modulefiles written in TCL start with the string
+7: #%Module
+7: 
+7: 
+7: 
+2: Lmod has detected the following error: The following module(s) are unknown:
+2: "rocm/sam-5.2.3"
+2: 
+2: Please check the spelling or version number. Also try "module spider ..."
+2: It is also possible your cache file is out-of-date; it may help to try:
+2:   $ module --ignore-cache load "rocm/sam-5.2.3"
+2: 
+2: Also make sure that all modulefiles written in TCL start with the string
+2: #%Module
+2: 
+2: 
+2: 
+5: Lmod has detected the following error: The following module(s) are unknown:
+5: "suse-repo-deps/sam-default"
+5: 
+5: Please check the spelling or version number. Also try "module spider ..."
+5: It is also possible your cache file is out-of-date; it may help to try:
+5:   $ module --ignore-cache load "suse-repo-deps/sam-default"
+5: 
+5: Also make sure that all modulefiles written in TCL start with the string
+5: #%Module
+5: 
+5: 
+5: 
+4: Lmod has detected the following error: The following module(s) are unknown:
+4: "suse-repo-deps/sam-default"
+4: 
+4: Please check the spelling or version number. Also try "module spider ..."
+4: It is also possible your cache file is out-of-date; it may help to try:
+4:   $ module --ignore-cache load "suse-repo-deps/sam-default"
+4: 
+4: Also make sure that all modulefiles written in TCL start with the string
+4: #%Module
+4: 
+4: 
+4: 
+3: Lmod has detected the following error: The following module(s) are unknown:
+3: "rocm/sam-5.2.3"
+3: 
+3: Please check the spelling or version number. Also try "module spider ..."
+3: It is also possible your cache file is out-of-date; it may help to try:
+3:   $ module --ignore-cache load "rocm/sam-5.2.3"
+3: 
+3: Also make sure that all modulefiles written in TCL start with the string
+3: #%Module
+3: 
+3: 
+3: 
+1: Lmod has detected the following error: The following module(s) are unknown:
+1: "rocm/sam-5.2.3"
+1: 
+1: Please check the spelling or version number. Also try "module spider ..."
+1: It is also possible your cache file is out-of-date; it may help to try:
+1:   $ module --ignore-cache load "rocm/sam-5.2.3"
+1: 
+1: Also make sure that all modulefiles written in TCL start with the string
+1: #%Module
+1: 
+1: 
+1: 
+6: Lmod has detected the following error: The following module(s) are unknown:
+6: "suse-repo-deps/sam-default"
+6: 
+6: Please check the spelling or version number. Also try "module spider ..."
+6: It is also possible your cache file is out-of-date; it may help to try:
+6:   $ module --ignore-cache load "suse-repo-deps/sam-default"
+6: 
+6: Also make sure that all modulefiles written in TCL start with the string
+6: #%Module
+6: 
+6: 
+6: 
+7: Lmod has detected the following error: The following module(s) are unknown:
+7: "rccl/sam-develop"
+7: 
+7: Please check the spelling or version number. Also try "module spider ..."
+7: It is also possible your cache file is out-of-date; it may help to try:
+7:   $ module --ignore-cache load "rccl/sam-develop"
+7: 
+7: Also make sure that all modulefiles written in TCL start with the string
+7: #%Module
+7: 
+7: 
+7: 
+2: Lmod has detected the following error: The following module(s) are unknown:
+2: "rccl/sam-develop"
+2: 
+2: Please check the spelling or version number. Also try "module spider ..."
+2: It is also possible your cache file is out-of-date; it may help to try:
+2:   $ module --ignore-cache load "rccl/sam-develop"
+2: 
+2: Also make sure that all modulefiles written in TCL start with the string
+2: #%Module
+2: 
+2: 
+2: 
+5: Lmod has detected the following error: The following module(s) are unknown:
+5: "rocm/sam-5.2.3"
+5: 
+5: Please check the spelling or version number. Also try "module spider ..."
+5: It is also possible your cache file is out-of-date; it may help to try:
+5:   $ module --ignore-cache load "rocm/sam-5.2.3"
+5: 
+5: Also make sure that all modulefiles written in TCL start with the string
+5: #%Module
+5: 
+5: 
+5: 
+4: Lmod has detected the following error: The following module(s) are unknown:
+4: "rocm/sam-5.2.3"
+4: 
+4: Please check the spelling or version number. Also try "module spider ..."
+4: It is also possible your cache file is out-of-date; it may help to try:
+4:   $ module --ignore-cache load "rocm/sam-5.2.3"
+4: 
+4: Also make sure that all modulefiles written in TCL start with the string
+4: #%Module
+4: 
+4: 
+4: 
+3: Lmod has detected the following error: The following module(s) are unknown:
+3: "rccl/sam-develop"
+3: 
+3: Please check the spelling or version number. Also try "module spider ..."
+3: It is also possible your cache file is out-of-date; it may help to try:
+3:   $ module --ignore-cache load "rccl/sam-develop"
+3: 
+3: Also make sure that all modulefiles written in TCL start with the string
+3: #%Module
+3: 
+3: 
+3: 
+6: Lmod has detected the following error: The following module(s) are unknown:
+6: "rocm/sam-5.2.3"
+6: 
+6: Please check the spelling or version number. Also try "module spider ..."
+6: It is also possible your cache file is out-of-date; it may help to try:
+6:   $ module --ignore-cache load "rocm/sam-5.2.3"
+6: 
+6: Also make sure that all modulefiles written in TCL start with the string
+6: #%Module
+6: 
+6: 
+6: 
+1: Lmod has detected the following error: The following module(s) are unknown:
+1: "rccl/sam-develop"
+1: 
+1: Please check the spelling or version number. Also try "module spider ..."
+1: It is also possible your cache file is out-of-date; it may help to try:
+1:   $ module --ignore-cache load "rccl/sam-develop"
+1: 
+1: Also make sure that all modulefiles written in TCL start with the string
+1: #%Module
+1: 
+1: 
+1: 
+7: Lmod has detected the following error: The following module(s) are unknown:
+7: "aws-ofi-rccl/sam-default"
+7: 
+7: Please check the spelling or version number. Also try "module spider ..."
+7: It is also possible your cache file is out-of-date; it may help to try:
+7:   $ module --ignore-cache load "aws-ofi-rccl/sam-default"
+7: 
+7: Also make sure that all modulefiles written in TCL start with the string
+7: #%Module
+7: 
+7: 
+7: 
+3: Lmod has detected the following error: The following module(s) are unknown:
+3: "aws-ofi-rccl/sam-default"
+3: 
+3: Please check the spelling or version number. Also try "module spider ..."
+3: It is also possible your cache file is out-of-date; it may help to try:
+3:   $ module --ignore-cache load "aws-ofi-rccl/sam-default"
+3: 
+3: Also make sure that all modulefiles written in TCL start with the string
+3: #%Module
+3: 
+3: 
+3: 
+2: Lmod has detected the following error: The following module(s) are unknown:
+2: "aws-ofi-rccl/sam-default"
+2: 
+2: Please check the spelling or version number. Also try "module spider ..."
+2: It is also possible your cache file is out-of-date; it may help to try:
+2:   $ module --ignore-cache load "aws-ofi-rccl/sam-default"
+2: 
+2: Also make sure that all modulefiles written in TCL start with the string
+2: #%Module
+2: 
+2: 
+2: 
+5: Lmod has detected the following error: The following module(s) are unknown:
+5: "rccl/sam-develop"
+5: 
+5: Please check the spelling or version number. Also try "module spider ..."
+5: It is also possible your cache file is out-of-date; it may help to try:
+5:   $ module --ignore-cache load "rccl/sam-develop"
+5: 
+5: Also make sure that all modulefiles written in TCL start with the string
+5: #%Module
+5: 
+5: 
+5: 
+4: Lmod has detected the following error: The following module(s) are unknown:
+4: "rccl/sam-develop"
+4: 
+4: Please check the spelling or version number. Also try "module spider ..."
+4: It is also possible your cache file is out-of-date; it may help to try:
+4:   $ module --ignore-cache load "rccl/sam-develop"
+4: 
+4: Also make sure that all modulefiles written in TCL start with the string
+4: #%Module
+4: 
+4: 
+4: 
+6: Lmod has detected the following error: The following module(s) are unknown:
+6: "rccl/sam-develop"
+6: 
+6: Please check the spelling or version number. Also try "module spider ..."
+6: It is also possible your cache file is out-of-date; it may help to try:
+6:   $ module --ignore-cache load "rccl/sam-develop"
+6: 
+6: Also make sure that all modulefiles written in TCL start with the string
+6: #%Module
+6: 
+6: 
+6: 
+1: Lmod has detected the following error: The following module(s) are unknown:
+1: "aws-ofi-rccl/sam-default"
+1: 
+1: Please check the spelling or version number. Also try "module spider ..."
+1: It is also possible your cache file is out-of-date; it may help to try:
+1:   $ module --ignore-cache load "aws-ofi-rccl/sam-default"
+1: 
+1: Also make sure that all modulefiles written in TCL start with the string
+1: #%Module
+1: 
+1: 
+1: 
+6: Lmod has detected the following error: The following module(s) are unknown:
+6: "aws-ofi-rccl/sam-default"
+6: 
+6: Please check the spelling or version number. Also try "module spider ..."
+6: It is also possible your cache file is out-of-date; it may help to try:
+6:   $ module --ignore-cache load "aws-ofi-rccl/sam-default"
+6: 
+6: Also make sure that all modulefiles written in TCL start with the string
+6: #%Module
+6: 
+6: 
+6: 
+5: Lmod has detected the following error: The following module(s) are unknown:
+5: "aws-ofi-rccl/sam-default"
+5: 
+5: Please check the spelling or version number. Also try "module spider ..."
+5: It is also possible your cache file is out-of-date; it may help to try:
+5:   $ module --ignore-cache load "aws-ofi-rccl/sam-default"
+5: 
+5: Also make sure that all modulefiles written in TCL start with the string
+5: #%Module
+5: 
+5: 
+5: 
+4: Lmod has detected the following error: The following module(s) are unknown:
+4: "aws-ofi-rccl/sam-default"
+4: 
+4: Please check the spelling or version number. Also try "module spider ..."
+4: It is also possible your cache file is out-of-date; it may help to try:
+4:   $ module --ignore-cache load "aws-ofi-rccl/sam-default"
+4: 
+4: Also make sure that all modulefiles written in TCL start with the string
+4: #%Module
+4: 
+4: 
+4: 
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: 2023-03-17 13:19:59.455061: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+0: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+0: 2023-03-17 13:19:59.455095: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+0: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+0: 2023-03-17 13:19:59.455122: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+0: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+0: 2023-03-17 13:19:59.455119: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+0: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+0: 2023-03-17 13:19:59.455140: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+0: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+0: 2023-03-17 13:19:59.455152: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+0: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+0: 2023-03-17 13:19:59.455170: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+0: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+0: 2023-03-17 13:19:59.455208: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+0: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+5: 2023-03-17 13:19:59.455621: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+5: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+5: 2023-03-17 13:19:59.455651: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+5: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+5: 2023-03-17 13:19:59.455621: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+5: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+5: 2023-03-17 13:19:59.455674: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+5: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+5: 2023-03-17 13:19:59.455720: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+5: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+5: 2023-03-17 13:19:59.455742: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+5: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+5: 2023-03-17 13:19:59.455793: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+5: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+5: 2023-03-17 13:19:59.455792: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+5: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+4: 2023-03-17 13:19:59.456022: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+4: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+4: 2023-03-17 13:19:59.456062: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+4: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+4: 2023-03-17 13:19:59.456087: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+4: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+4: 2023-03-17 13:19:59.456138: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+4: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+4: 2023-03-17 13:19:59.456163: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+4: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+4: 2023-03-17 13:19:59.456179: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+4: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+4: 2023-03-17 13:19:59.456181: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+4: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+4: 2023-03-17 13:19:59.456238: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+4: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+6: 2023-03-17 13:19:59.456099: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+6: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+6: 2023-03-17 13:19:59.456112: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+6: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+6: 2023-03-17 13:19:59.456114: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+6: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+7: 2023-03-17 13:19:59.456817: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+7: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+7: 2023-03-17 13:19:59.456835: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+7: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+7: 2023-03-17 13:19:59.456838: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+6: 2023-03-17 13:19:59.456146: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+6: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+7: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+7: 2023-03-17 13:19:59.456852: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+7: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+6: 2023-03-17 13:19:59.456177: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+6: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+7: 2023-03-17 13:19:59.456887: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+7: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+7: 2023-03-17 13:19:59.456883: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+7: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+7: 2023-03-17 13:19:59.456897: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+6: 2023-03-17 13:19:59.456241: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+6: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+6: 2023-03-17 13:19:59.456234: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+6: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+6: 2023-03-17 13:19:59.456258: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+7: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+7: 2023-03-17 13:19:59.456907: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+7: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+6: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+1: 2023-03-17 13:19:59.456976: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+1: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+1: 2023-03-17 13:19:59.456995: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+1: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+1: 2023-03-17 13:19:59.457012: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+1: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+1: 2023-03-17 13:19:59.457041: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+1: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+1: 2023-03-17 13:19:59.457038: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+1: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+1: 2023-03-17 13:19:59.457072: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+1: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+1: 2023-03-17 13:19:59.457051: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+1: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+1: 2023-03-17 13:19:59.457053: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+1: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+2: 2023-03-17 13:19:59.458343: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+2: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+2: 2023-03-17 13:19:59.458334: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+2: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+2: 2023-03-17 13:19:59.458357: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+2: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+2: 2023-03-17 13:19:59.458358: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+2: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+2: 2023-03-17 13:19:59.458342: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+2: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+2: 2023-03-17 13:19:59.458375: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+2: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+2: 2023-03-17 13:19:59.458389: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+2: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+2: 2023-03-17 13:19:59.458363: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+2: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+3: 2023-03-17 13:19:59.461757: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+3: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+3: 2023-03-17 13:19:59.461760: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+3: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+3: 2023-03-17 13:19:59.461775: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+3: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+3: 2023-03-17 13:19:59.461772: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+3: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+3: 2023-03-17 13:19:59.461790: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+3: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+3: 2023-03-17 13:19:59.461802: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+3: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+3: 2023-03-17 13:19:59.461809: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+3: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+3: 2023-03-17 13:19:59.461814: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
+3: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+7: 2023-03-17 13:20:14.995629: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+7: 2023-03-17 13:20:14.995657: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+7: 2023-03-17 13:20:14.995683: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+7: 2023-03-17 13:20:14.995684: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+7: 2023-03-17 13:20:14.995715: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+7: 2023-03-17 13:20:14.995675: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+7: 2023-03-17 13:20:14.995716: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+7: 2023-03-17 13:20:14.995728: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+7: 2023-03-17 13:20:14.996314: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+7: 2023-03-17 13:20:14.996333: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+7: 2023-03-17 13:20:14.996343: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+7: 2023-03-17 13:20:14.996362: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+7: 2023-03-17 13:20:14.996366: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+7: 2023-03-17 13:20:14.996363: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+7: 2023-03-17 13:20:14.996377: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+7: 2023-03-17 13:20:14.996388: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+4: 2023-03-17 13:20:14.996400: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+4: 2023-03-17 13:20:14.996430: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+4: 2023-03-17 13:20:14.996442: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+4: 2023-03-17 13:20:14.996728: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+4: 2023-03-17 13:20:14.996459: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+4: 2023-03-17 13:20:14.996753: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+4: 2023-03-17 13:20:14.996468: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+4: 2023-03-17 13:20:14.996762: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+4: 2023-03-17 13:20:14.996485: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+4: 2023-03-17 13:20:14.996488: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+4: 2023-03-17 13:20:14.996776: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+4: 2023-03-17 13:20:14.996783: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+4: 2023-03-17 13:20:14.996794: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+4: 2023-03-17 13:20:14.996495: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+4: 2023-03-17 13:20:14.996808: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+4: 2023-03-17 13:20:14.996815: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+5: 2023-03-17 13:20:14.996708: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+5: 2023-03-17 13:20:14.996738: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+5: 2023-03-17 13:20:14.996797: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+5: 2023-03-17 13:20:14.996834: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+5: 2023-03-17 13:20:14.996844: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+5: 2023-03-17 13:20:14.996856: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+5: 2023-03-17 13:20:14.996773: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+5: 2023-03-17 13:20:14.996876: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+5: 2023-03-17 13:20:14.997497: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+5: 2023-03-17 13:20:14.997536: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+5: 2023-03-17 13:20:14.997546: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+5: 2023-03-17 13:20:14.997564: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+3: 2023-03-17 13:20:14.996798: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+5: 2023-03-17 13:20:14.997576: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+5: 2023-03-17 13:20:14.997587: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+5: 2023-03-17 13:20:14.997591: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+5: 2023-03-17 13:20:14.997596: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+3: 2023-03-17 13:20:14.996830: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+3: 2023-03-17 13:20:14.996849: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+3: 2023-03-17 13:20:14.996884: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+3: 2023-03-17 13:20:14.996871: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+3: 2023-03-17 13:20:14.996880: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+3: 2023-03-17 13:20:14.996886: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+3: 2023-03-17 13:20:14.996903: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+1: 2023-03-17 13:20:14.996916: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+1: 2023-03-17 13:20:14.996945: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+1: 2023-03-17 13:20:14.996982: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+1: 2023-03-17 13:20:14.997400: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+1: 2023-03-17 13:20:14.997408: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+1: 2023-03-17 13:20:14.997037: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+1: 2023-03-17 13:20:14.997012: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+1: 2023-03-17 13:20:14.997034: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+1: 2023-03-17 13:20:14.997423: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+3: 2023-03-17 13:20:14.997511: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+1: 2023-03-17 13:20:14.997049: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+1: 2023-03-17 13:20:14.997044: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+1: 2023-03-17 13:20:14.997434: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+1: 2023-03-17 13:20:14.997436: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+1: 2023-03-17 13:20:14.997442: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+1: 2023-03-17 13:20:14.997444: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+1: 2023-03-17 13:20:14.997447: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+3: 2023-03-17 13:20:14.997531: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+3: 2023-03-17 13:20:14.997540: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+3: 2023-03-17 13:20:14.997564: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+3: 2023-03-17 13:20:14.997563: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+3: 2023-03-17 13:20:14.997568: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+3: 2023-03-17 13:20:14.997595: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+3: 2023-03-17 13:20:14.997602: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+0: 2023-03-17 13:20:14.998468: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+0: 2023-03-17 13:20:14.998500: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+0: 2023-03-17 13:20:14.998512: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+0: 2023-03-17 13:20:14.998510: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+0: 2023-03-17 13:20:14.998516: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+0: 2023-03-17 13:20:14.998525: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+0: 2023-03-17 13:20:14.998675: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+0: 2023-03-17 13:20:14.998529: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+0: 2023-03-17 13:20:14.999296: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+0: 2023-03-17 13:20:14.999326: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+0: 2023-03-17 13:20:14.999327: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+2: 2023-03-17 13:20:14.998596: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+2: 2023-03-17 13:20:14.998632: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+2: 2023-03-17 13:20:14.998995: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+2: 2023-03-17 13:20:14.998657: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+0: 2023-03-17 13:20:14.999363: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+0: 2023-03-17 13:20:14.999372: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+0: 2023-03-17 13:20:14.999374: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+0: 2023-03-17 13:20:14.999373: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+2: 2023-03-17 13:20:14.998676: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+2: 2023-03-17 13:20:14.999018: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+0: 2023-03-17 13:20:14.999384: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+2: 2023-03-17 13:20:14.998697: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+2: 2023-03-17 13:20:14.998705: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+2: 2023-03-17 13:20:14.999042: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+2: 2023-03-17 13:20:14.998717: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+2: 2023-03-17 13:20:14.999052: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+2: 2023-03-17 13:20:14.998726: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+2: 2023-03-17 13:20:14.999060: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+2: 2023-03-17 13:20:14.999066: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+2: 2023-03-17 13:20:14.999080: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+2: 2023-03-17 13:20:14.999091: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+6: 2023-03-17 13:20:14.998316: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+6: 2023-03-17 13:20:14.998344: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+6: 2023-03-17 13:20:14.998768: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+6: 2023-03-17 13:20:14.998403: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+6: 2023-03-17 13:20:14.998784: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+6: 2023-03-17 13:20:14.998434: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+6: 2023-03-17 13:20:14.998444: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+6: 2023-03-17 13:20:14.998361: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+6: 2023-03-17 13:20:14.998808: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+6: 2023-03-17 13:20:14.998447: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+6: 2023-03-17 13:20:14.998812: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+6: 2023-03-17 13:20:14.998463: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+6: 2023-03-17 13:20:14.998823: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+6: 2023-03-17 13:20:14.998830: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+6: 2023-03-17 13:20:14.998831: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+6: 2023-03-17 13:20:14.998842: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+2: 2023-03-17 13:21:03.391855: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+2: 2023-03-17 13:21:03.391889: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+2: 2023-03-17 13:21:03.391910: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+2: 2023-03-17 13:21:03.391928: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+2: 2023-03-17 13:21:03.391944: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+2: 2023-03-17 13:21:03.391949: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+2: 2023-03-17 13:21:03.391964: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+2: 2023-03-17 13:21:03.391967: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+5: 2023-03-17 13:21:03.392979: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+5: 2023-03-17 13:21:03.392998: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+5: 2023-03-17 13:21:03.393669: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+5: 2023-03-17 13:21:03.393676: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+5: 2023-03-17 13:21:03.393680: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+5: 2023-03-17 13:21:03.393707: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+5: 2023-03-17 13:21:03.393710: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+5: 2023-03-17 13:21:03.393718: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+6: 2023-03-17 13:21:03.394160: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+6: 2023-03-17 13:21:03.394192: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+6: 2023-03-17 13:21:03.394203: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+6: 2023-03-17 13:21:03.394234: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+6: 2023-03-17 13:21:03.394238: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+6: 2023-03-17 13:21:03.394247: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+6: 2023-03-17 13:21:03.394267: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+6: 2023-03-17 13:21:03.394464: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+1: 2023-03-17 13:21:03.417352: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+1: 2023-03-17 13:21:03.417379: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+1: 2023-03-17 13:21:03.417400: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+1: 2023-03-17 13:21:03.417409: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+1: 2023-03-17 13:21:03.417424: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+1: 2023-03-17 13:21:03.417421: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+1: 2023-03-17 13:21:03.417435: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+1: 2023-03-17 13:21:03.417448: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+3: 2023-03-17 13:21:03.417971: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+3: 2023-03-17 13:21:03.418011: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+3: 2023-03-17 13:21:03.418002: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+3: 2023-03-17 13:21:03.418034: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+3: 2023-03-17 13:21:03.418046: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+3: 2023-03-17 13:21:03.418068: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+3: 2023-03-17 13:21:03.418070: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+3: 2023-03-17 13:21:03.418246: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+0: 2023-03-17 13:21:03.419901: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+0: 2023-03-17 13:21:03.419936: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+0: 2023-03-17 13:21:03.419950: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+0: 2023-03-17 13:21:03.419962: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+0: 2023-03-17 13:21:03.419972: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+0: 2023-03-17 13:21:03.419981: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+0: 2023-03-17 13:21:03.419989: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+0: 2023-03-17 13:21:03.420000: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+1: 2023-03-17 13:21:03.517638: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+1: 2023-03-17 13:21:03.517641: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+1: 2023-03-17 13:21:03.517643: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+1: 2023-03-17 13:21:03.517645: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+1: 2023-03-17 13:21:03.517649: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+1: 2023-03-17 13:21:03.517645: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+1: 2023-03-17 13:21:03.517645: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+1: 2023-03-17 13:21:03.517651: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+1: 2023-03-17 13:21:03.517669: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+1: 2023-03-17 13:21:03.517674: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+1: 2023-03-17 13:21:03.517676: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+1: 2023-03-17 13:21:03.517674: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+1: 2023-03-17 13:21:03.517681: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+1: 2023-03-17 13:21:03.517683: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+1: 2023-03-17 13:21:03.517682: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+1: 2023-03-17 13:21:03.517684: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+3: 2023-03-17 13:21:03.518133: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+3: 2023-03-17 13:21:03.518136: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+3: 2023-03-17 13:21:03.518157: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+3: 2023-03-17 13:21:03.518157: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+3: 2023-03-17 13:21:03.518153: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+3: 2023-03-17 13:21:03.518154: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+3: 2023-03-17 13:21:03.518157: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+3: 2023-03-17 13:21:03.518160: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+3: 2023-03-17 13:21:03.518162: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+3: 2023-03-17 13:21:03.518162: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+3: 2023-03-17 13:21:03.518190: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+3: 2023-03-17 13:21:03.518191: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+3: 2023-03-17 13:21:03.518192: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+3: 2023-03-17 13:21:03.518193: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+3: 2023-03-17 13:21:03.518195: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+3: 2023-03-17 13:21:03.518194: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+2: 2023-03-17 13:21:03.518964: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+2: 2023-03-17 13:21:03.518965: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+2: 2023-03-17 13:21:03.518966: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+2: 2023-03-17 13:21:03.518967: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+2: 2023-03-17 13:21:03.518968: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+2: 2023-03-17 13:21:03.518972: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+2: 2023-03-17 13:21:03.518975: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+2: 2023-03-17 13:21:03.518983: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+2: 2023-03-17 13:21:03.519009: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+2: 2023-03-17 13:21:03.519011: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+2: 2023-03-17 13:21:03.519012: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+2: 2023-03-17 13:21:03.519013: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+2: 2023-03-17 13:21:03.519015: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+2: 2023-03-17 13:21:03.519016: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+2: 2023-03-17 13:21:03.519016: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+2: 2023-03-17 13:21:03.519019: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+6: 2023-03-17 13:21:03.518774: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+6: 2023-03-17 13:21:03.518775: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+5: 2023-03-17 13:21:03.519302: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+5: 2023-03-17 13:21:03.519306: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+0: 2023-03-17 13:21:03.519788: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+0: 2023-03-17 13:21:03.519797: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+0: 2023-03-17 13:21:03.519800: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+0: 2023-03-17 13:21:03.519801: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+0: 2023-03-17 13:21:03.519813: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+6: 2023-03-17 13:21:03.518773: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+6: 2023-03-17 13:21:03.518776: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+5: 2023-03-17 13:21:03.519308: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+5: 2023-03-17 13:21:03.519312: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+0: 2023-03-17 13:21:03.519804: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+6: 2023-03-17 13:21:03.518780: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+6: 2023-03-17 13:21:03.518780: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+5: 2023-03-17 13:21:03.519316: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+5: 2023-03-17 13:21:03.519315: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+0: 2023-03-17 13:21:03.519822: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+0: 2023-03-17 13:21:03.519825: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+0: 2023-03-17 13:21:03.519826: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+6: 2023-03-17 13:21:03.518786: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+6: 2023-03-17 13:21:03.518782: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+6: 2023-03-17 13:21:03.518799: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+5: 2023-03-17 13:21:03.519321: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+5: 2023-03-17 13:21:03.519325: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+5: 2023-03-17 13:21:03.519347: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+0: 2023-03-17 13:21:03.519828: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+0: 2023-03-17 13:21:03.519826: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+0: 2023-03-17 13:21:03.519827: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+6: 2023-03-17 13:21:03.518798: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+6: 2023-03-17 13:21:03.518799: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+6: 2023-03-17 13:21:03.518798: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+5: 2023-03-17 13:21:03.519348: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+5: 2023-03-17 13:21:03.519349: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+5: 2023-03-17 13:21:03.519353: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+0: 2023-03-17 13:21:03.519829: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+6: 2023-03-17 13:21:03.518808: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+6: 2023-03-17 13:21:03.518809: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+6: 2023-03-17 13:21:03.518810: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+5: 2023-03-17 13:21:03.519355: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+5: 2023-03-17 13:21:03.519356: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+5: 2023-03-17 13:21:03.519357: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+0: 2023-03-17 13:21:03.519850: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+0: 2023-03-17 13:21:03.519853: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+0: 2023-03-17 13:21:03.519856: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+6: 2023-03-17 13:21:03.518810: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+5: 2023-03-17 13:21:03.519359: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+4: 2023-03-17 13:21:04.513675: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+4: 2023-03-17 13:21:04.513689: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+4: 2023-03-17 13:21:04.513684: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+4: 2023-03-17 13:21:04.513686: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+4: 2023-03-17 13:21:04.513695: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+4: 2023-03-17 13:21:04.513691: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+4: 2023-03-17 13:21:04.513690: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+4: 2023-03-17 13:21:04.513710: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+7: 2023-03-17 13:21:04.513778: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+7: 2023-03-17 13:21:04.513785: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+7: 2023-03-17 13:21:04.513793: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+7: 2023-03-17 13:21:04.513789: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+7: 2023-03-17 13:21:04.513793: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+7: 2023-03-17 13:21:04.513793: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+4: 2023-03-17 13:21:04.514516: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+4: 2023-03-17 13:21:04.514518: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+7: 2023-03-17 13:21:04.513788: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+7: 2023-03-17 13:21:04.513798: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+4: 2023-03-17 13:21:04.514522: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+4: 2023-03-17 13:21:04.514523: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+4: 2023-03-17 13:21:04.514523: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+4: 2023-03-17 13:21:04.514525: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+4: 2023-03-17 13:21:04.514527: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+4: 2023-03-17 13:21:04.514529: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+4: 2023-03-17 13:21:04.514528: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+4: 2023-03-17 13:21:04.514533: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+4: 2023-03-17 13:21:04.514538: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+4: 2023-03-17 13:21:04.514537: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+4: 2023-03-17 13:21:04.514539: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+4: 2023-03-17 13:21:04.514539: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+4: 2023-03-17 13:21:04.514582: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+4: 2023-03-17 13:21:04.514593: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+7: 2023-03-17 13:21:04.514615: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+7: 2023-03-17 13:21:04.514618: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+7: 2023-03-17 13:21:04.514619: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+7: 2023-03-17 13:21:04.514625: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+7: 2023-03-17 13:21:04.514623: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+7: 2023-03-17 13:21:04.514628: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+7: 2023-03-17 13:21:04.514633: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+7: 2023-03-17 13:21:04.514634: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+7: 2023-03-17 13:21:04.514635: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+7: 2023-03-17 13:21:04.514637: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+7: 2023-03-17 13:21:04.514637: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+7: 2023-03-17 13:21:04.514639: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+7: 2023-03-17 13:21:04.514635: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+7: 2023-03-17 13:21:04.514649: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+7: 2023-03-17 13:21:04.514696: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
+7: 2023-03-17 13:21:04.514709: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+0: Loading extension module scaled_upper_triang_masked_softmax_cuda...
+0: Loading extension module scaled_masked_softmax_cuda...
+0: Loading extension module fused_mix_prec_layer_norm_cuda...
+0: Successfully preprocessed all matching files.
+0: Successfully preprocessed all matching files.
+7: Successfully preprocessed all matching files.
+0: Traceback (most recent call last):
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1900, in _run_ninja_build
+0: Traceback (most recent call last):
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1900, in _run_ninja_build
+0:     subprocess.run(
+0:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/subprocess.py", line 528, in run
+0:     raise CalledProcessError(retcode, process.args,
+0: subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
+0: 
+0: The above exception was the direct cause of the following exception:
+0: 
+0: Traceback (most recent call last):
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 235, in <module>
+0:     main()
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
+0:     subprocess.run(
+0:     return f(*args, **kwargs)  File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/subprocess.py", line 528, in run
+0: 
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 231, in main
+0:     pretrain(train_valid_test_datasets_provider, model_provider, forward_step,
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/training.py", line 99, in pretrain
+0:     raise CalledProcessError(retcode, process.args,
+0: subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
+0: 
+0: The above exception was the direct cause of the following exception:
+0: 
+0: Traceback (most recent call last):
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1508, in _jit_compile
+0:     initialize_megatron(extra_args_provider=extra_args_provider,
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 164, in initialize_megatron
+0:     _compile_dependencies()
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 222, in _compile_dependencies
+0:     fused_kernels.load(args)
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/__init__.py", line 95, in load
+0:     _write_ninja_file_and_build_library(
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1623, in _write_ninja_file_and_build_library
+0:     scaled_masked_softmax_cuda = _cpp_extention_load_helper(
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/__init__.py", line 56, in _cpp_extention_load_helper
+0:     return cpp_extension.load(
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1284, in load
+0:     _run_ninja_build(
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1916, in _run_ninja_build
+0:     return _jit_compile(
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1508, in _jit_compile
+0:     raise RuntimeError(message) from e
+0: RuntimeError: Error building extension 'scaled_masked_softmax_cuda': [1/2] /opt/rocm-5.1.0/bin/hipcc  -DWITH_HIP -DTORCH_EXTENSION_NAME=scaled_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/TH -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THC -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-be
+0: ttercom/venv/lib/python3.9/site-packages/torch/include/THH -isystem /opt/rocm-5.1.0/include -isystem /opt/rocm-5.1.0/miopen/include -isystem /opt/rocm-5.1.0/hip/include -isystem /opt/cray/pe/python/3.9.12.1/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -fPIC -D__HIP_PLATFORM_HCC__=1 -DUSE_ROCM=1 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 --amdgpu-target=gfx900 --amdgpu-target=gfx906 --amdgpu-target=gfx908 --amdgpu-target=gfx90a --amdgpu-target=gfx1030 -fno-gpu-rdc -c /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip -o scaled_masked_softmax_hip.cuda.o 
+0: FAILED: scaled_masked_softmax_hip.cuda.o 
+0: /opt/rocm-5.1.0/bin/hipcc  -DWITH_HIP -DTORCH_EXTENSION_NAME=scaled_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/TH -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THC -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THH -isystem /opt/ro
+0: cm-5.1.0/include -isystem /opt/rocm-5.1.0/miopen/include -isystem /opt/rocm-5.1.0/hip/include -isystem /opt/cray/pe/python/3.9.12.1/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -fPIC -D__HIP_PLATFORM_HCC__=1 -DUSE_ROCM=1 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 --amdgpu-target=gfx900 --amdgpu-target=gfx906 --amdgpu-target=gfx908 --amdgpu-target=gfx90a --amdgpu-target=gfx1030 -fno-gpu-rdc -c /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip -o scaled_masked_softmax_hip.cuda.o 
+0: In file included from /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip:25:
+0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/ATen/hip/HIPContext.h:7:10: fatal error: 'hipsparse/hipsparse.h' file not found
+0: #include <hipsparse/hipsparse.h>
+0:          ^~~~~~~~~~~~~~~~~~~~~~~
+0: 1 error generated when compiling for gfx1030.
+0: ninja: build stopped: subcommand failed.
+0: 
+0:     
+0: During handling of the above exception, another exception occurred:
+0: 
+0: _write_ninja_file_and_build_library(Traceback (most recent call last):
+0: 
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1623, in _write_ninja_file_and_build_library
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 235, in <module>
+0:     main()
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
+0:     return f(*args, **kwargs)
+0:       File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 231, in main
+0: _run_ninja_build(
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1916, in _run_ninja_build
+0:     pretrain(train_valid_test_datasets_provider, model_provider, forward_step,
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/training.py", line 99, in pretrain
+0:     initialize_megatron(extra_args_provider=extra_args_provider,
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 164, in initialize_megatron
+0:     _compile_dependencies()
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 222, in _compile_dependencies
+0:     raise RuntimeError(message) from e
+0: RuntimeError: Error building extension 'scaled_masked_softmax_cuda': [1/2] /opt/rocm-5.1.0/bin/hipcc  -DWITH_HIP -DTORCH_EXTENSION_NAME=scaled_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/TH -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THC -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-be
+0: ttercom/venv/lib/python3.9/site-packages/torch/include/THH -isystem /opt/rocm-5.1.0/include -isystem /opt/rocm-5.1.0/miopen/include -isystem /opt/rocm-5.1.0/hip/include -isystem /opt/cray/pe/python/3.9.12.1/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -fPIC -D__HIP_PLATFORM_HCC__=1 -DUSE_ROCM=1 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 --amdgpu-target=gfx900 --amdgpu-target=gfx906 --amdgpu-target=gfx908 --amdgpu-target=gfx90a --amdgpu-target=gfx1030 -fno-gpu-rdc -c /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip -o scaled_masked_softmax_hip.cuda.o 
+0: FAILED: scaled_masked_softmax_hip.cuda.o 
+0: /opt/rocm-5.1.0/bin/hipcc  -DWITH_HIP -DTORCH_EXTENSION_NAME=scaled_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/TH -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THC -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THH -isystem /opt/ro
+0: cm-5.1.0/include -isystem /opt/rocm-5.1.0/miopen/include -isystem /opt/rocm-5.1.0/hip/include -isystem /opt/cray/pe/python/3.9.12.1/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -fPIC -D__HIP_PLATFORM_HCC__=1 -DUSE_ROCM=1 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 --amdgpu-target=gfx900 --amdgpu-target=gfx906 --amdgpu-target=gfx908 --amdgpu-target=gfx90a --amdgpu-target=gfx1030 -fno-gpu-rdc -c /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip -o scaled_masked_softmax_hip.cuda.o 
+0: In file included from /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip:25:
+0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/ATen/hip/HIPContext.h:7:10: fatal error: 'hipsparse/hipsparse.h' file not found
+0: #include <hipsparse/hipsparse.h>
+0:          ^~~~~~~~~~~~~~~~~~~~~~~
+0: 1 error generated when compiling for gfx1030.
+0: ninja: build stopped: subcommand failed.
+0: 
+0:     fused_kernels.load(args)
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/__init__.py", line 95, in load
+0:     scaled_masked_softmax_cuda = _cpp_extention_load_helper(
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/__init__.py", line 56, in _cpp_extention_load_helper
+0:     return cpp_extension.load(
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1284, in load
+0:     return _jit_compile(
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1520, in _jit_compile
+0:     baton.release()
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/file_baton.py", line 49, in release
+0:     os.remove(self.lock_file_path)
+0: FileNotFoundError: [Errno 2] No such file or directory: '/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/build/lock'
+7: Traceback (most recent call last):
+7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1900, in _run_ninja_build
+7:     subprocess.run(
+7:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/subprocess.py", line 528, in run
+7:     raise CalledProcessError(retcode, process.args,
+7: subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
+7: 
+7: The above exception was the direct cause of the following exception:
+7: 
+7: Traceback (most recent call last):
+7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1508, in _jit_compile
+7:     _write_ninja_file_and_build_library(
+7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1623, in _write_ninja_file_and_build_library
+7:     _run_ninja_build(
+7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1916, in _run_ninja_build
+7:     raise RuntimeError(message) from e
+7: RuntimeError: Error building extension 'scaled_masked_softmax_cuda': [1/2] /opt/rocm-5.1.0/bin/hipcc  -DWITH_HIP -DTORCH_EXTENSION_NAME=scaled_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/TH -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THC -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-be
+7: ttercom/venv/lib/python3.9/site-packages/torch/include/THH -isystem /opt/rocm-5.1.0/include -isystem /opt/rocm-5.1.0/miopen/include -isystem /opt/rocm-5.1.0/hip/include -isystem /opt/cray/pe/python/3.9.12.1/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -fPIC -D__HIP_PLATFORM_HCC__=1 -DUSE_ROCM=1 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 --amdgpu-target=gfx900 --amdgpu-target=gfx906 --amdgpu-target=gfx908 --amdgpu-target=gfx90a --amdgpu-target=gfx1030 -fno-gpu-rdc -c /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip -o scaled_masked_softmax_hip.cuda.o 
+7: FAILED: scaled_masked_softmax_hip.cuda.o 
+7: /opt/rocm-5.1.0/bin/hipcc  -DWITH_HIP -DTORCH_EXTENSION_NAME=scaled_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/TH -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THC -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THH -isystem /opt/ro
+7: cm-5.1.0/include -isystem /opt/rocm-5.1.0/miopen/include -isystem /opt/rocm-5.1.0/hip/include -isystem /opt/cray/pe/python/3.9.12.1/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -fPIC -D__HIP_PLATFORM_HCC__=1 -DUSE_ROCM=1 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 --amdgpu-target=gfx900 --amdgpu-target=gfx906 --amdgpu-target=gfx908 --amdgpu-target=gfx90a --amdgpu-target=gfx1030 -fno-gpu-rdc -c /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip -o scaled_masked_softmax_hip.cuda.o 
+7: In file included from /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip:25:
+7: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/ATen/hip/HIPContext.h:7:10: fatal error: 'hipsparse/hipsparse.h' file not found
+7: #include <hipsparse/hipsparse.h>
+7:          ^~~~~~~~~~~~~~~~~~~~~~~
+7: 1 error generated when compiling for gfx1030.
+7: ninja: build stopped: subcommand failed.
+7: 
+7: 
+7: During handling of the above exception, another exception occurred:
+7: 
+7: Traceback (most recent call last):
+7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 235, in <module>
+7:     main()
+7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
+7:     return f(*args, **kwargs)
+7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 231, in main
+7:     pretrain(train_valid_test_datasets_provider, model_provider, forward_step,
+7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/training.py", line 99, in pretrain
+7:     initialize_megatron(extra_args_provider=extra_args_provider,
+7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 164, in initialize_megatron
+7:     _compile_dependencies()
+7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 222, in _compile_dependencies
+7:     fused_kernels.load(args)
+7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/__init__.py", line 95, in load
+7:     scaled_masked_softmax_cuda = _cpp_extention_load_helper(
+7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/__init__.py", line 56, in _cpp_extention_load_helper
+7:     return cpp_extension.load(
+7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1284, in load
+7:     return _jit_compile(
+7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1520, in _jit_compile
+7:     baton.release()
+7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/file_baton.py", line 49, in release
+7:     os.remove(self.lock_file_path)
+7: FileNotFoundError: [Errno 2] No such file or directory: '/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/build/lock'
+0: Successfully preprocessed all matching files.
+7: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 90799 closing signal SIGTERM
+7: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 90800 closing signal SIGTERM
+7: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 90802 closing signal SIGTERM
+7: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 90803 closing signal SIGTERM
+7: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 90804 closing signal SIGTERM
+7: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 90805 closing signal SIGTERM
+7: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 90806 closing signal SIGTERM
+0: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 106009 closing signal SIGTERM
+0: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 106011 closing signal SIGTERM
+0: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 106013 closing signal SIGTERM
+0: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 106014 closing signal SIGTERM
+0: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 106015 closing signal SIGTERM
+0: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 106016 closing signal SIGTERM
+0: ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 1 (pid: 106010) of binary: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/bin/python
+7: ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 2 (pid: 90801) of binary: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/bin/python
+6: [E ProcessGroupNCCL.cpp:456] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
+6: [E ProcessGroupNCCL.cpp:461] To avoid data inconsistency, we are taking the entire process down.
+6: terminate called after throwing an instance of 'std::runtime_error'
+6:   what():  NCCL error: unhandled system error, NCCL version 2.11.4
+6: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. It can be also caused by unexpected exit of a remote peer.
+6: Fatal Python error: Aborted
+6: 
+6: Thread 0x000014c813ac7b80 (most recent call first):
+6:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 3152 in barrier
+6:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 228 in _compile_dependencies
+6:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 164 in initialize_megatron
+6:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/training.py", line 99 in pretrain
+6:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 231 in main
+6:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346 in wrapper
+6:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 235 in <module>
+4: [E ProcessGroupNCCL.cpp:456] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
+4: [E ProcessGroupNCCL.cpp:461] To avoid data inconsistency, we are taking the entire process down.
+4: terminate called after throwing an instance of 'std::runtime_error'
+4:   what():  NCCL error: unhandled system error, NCCL version 2.11.4
+4: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. It can be also caused by unexpected exit of a remote peer.
+4: Fatal Python error: Aborted
+4: 
+4: Thread 0x000014f1f5ae0b80 (most recent call first):
+4:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 3152 in barrier
+4:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 228 in _compile_dependencies
+4:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 164 in initialize_megatron
+4:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/training.py", line 99 in pretrain
+4:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 231 in main
+4:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346 in wrapper
+4:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 235 in <module>
+4: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 102637 closing signal SIGTERM
+4: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 102638 closing signal SIGTERM
+4: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 102639 closing signal SIGTERM
+4: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 102640 closing signal SIGTERM
+4: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 102641 closing signal SIGTERM
+4: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 102642 closing signal SIGTERM
+4: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 102643 closing signal SIGTERM
+6: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 7574 closing signal SIGTERM
+6: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 7575 closing signal SIGTERM
+6: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 7576 closing signal SIGTERM
+6: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 7577 closing signal SIGTERM
+6: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 7578 closing signal SIGTERM
+6: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 7579 closing signal SIGTERM
+6: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 7580 closing signal SIGTERM
+4: ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 0 (pid: 102636) of binary: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/bin/python
+6: ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 0 (pid: 7573) of binary: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/bin/python
+5: [E ProcessGroupNCCL.cpp:456] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
+5: [E ProcessGroupNCCL.cpp:461] To avoid data inconsistency, we are taking the entire process down.
+5: terminate called after throwing an instance of 'std::runtime_error'
+5:   what():  NCCL error: unhandled system error, NCCL version 2.11.4
+5: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. It can be also caused by unexpected exit of a remote peer.
+5: Fatal Python error: Aborted
+5: 
+5: Thread 0x000014eb231bab80 (most recent call first):
+5:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 3152 in barrier
+5:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 228 in _compile_dependencies
+5:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 164 in initialize_megatron
+5:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/training.py", line 99 in pretrain
+5:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 231 in main
+5:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346 in wrapper
+5:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 235 in <module>
+2: [E ProcessGroupNCCL.cpp:456] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
+2: [E ProcessGroupNCCL.cpp:461] To avoid data inconsistency, we are taking the entire process down.
+2: terminate called after throwing an instance of 'std::runtime_error'
+2:   what():  NCCL error: unhandled system error, NCCL version 2.11.4
+2: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. It can be also caused by unexpected exit of a remote peer.
+2: Fatal Python error: Aborted
+2: 
+2: Thread 0x00001534ede95b80 (most recent call first):
+2:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 3152 in barrier
+2:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 228 in _compile_dependencies
+2:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 164 in initialize_megatron
+2:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/training.py", line 99 in pretrain
+2:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 231 in main
+2:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346 in wrapper
+2:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 235 in <module>
+3: [E ProcessGroupNCCL.cpp:456] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
+3: [E ProcessGroupNCCL.cpp:461] To avoid data inconsistency, we are taking the entire process down.
+3: terminate called after throwing an instance of 'std::runtime_error'
+3:   what():  NCCL error: unhandled system error, NCCL version 2.11.4
+3: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. It can be also caused by unexpected exit of a remote peer.
+3: Fatal Python error: Aborted
+3: 
+3: Thread 0x00001470142fbb80 (most recent call first):
+3:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 3152 in barrier
+3:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 228 in _compile_dependencies
+3:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 164 in initialize_megatron
+3:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/training.py", line 99 in pretrain
+3:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 231 in main
+3:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346 in wrapper
+3:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 235 in <module>
+3: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 106549 closing signal SIGTERM
+3: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 106550 closing signal SIGTERM
+3: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 106551 closing signal SIGTERM
+3: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 106552 closing signal SIGTERM
+3: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 106553 closing signal SIGTERM
+3: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 106554 closing signal SIGTERM
+3: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 106555 closing signal SIGTERM
+2: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 23541 closing signal SIGTERM
+2: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 23542 closing signal SIGTERM
+2: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 23543 closing signal SIGTERM
+2: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 23544 closing signal SIGTERM
+2: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 23545 closing signal SIGTERM
+2: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 23546 closing signal SIGTERM
+2: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 23547 closing signal SIGTERM
+5: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 96928 closing signal SIGTERM
+5: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 96929 closing signal SIGTERM
+5: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 96930 closing signal SIGTERM
+5: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 96931 closing signal SIGTERM
+5: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 96932 closing signal SIGTERM
+5: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 96933 closing signal SIGTERM
+5: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 96934 closing signal SIGTERM
+3: ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 0 (pid: 106548) of binary: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/bin/python
+2: ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 0 (pid: 23540) of binary: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/bin/python
+5: ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 0 (pid: 96927) of binary: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/bin/python
+1: [E ProcessGroupNCCL.cpp:456] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
+1: [E ProcessGroupNCCL.cpp:461] To avoid data inconsistency, we are taking the entire process down.
+1: terminate called after throwing an instance of 'std::runtime_error'
+1:   what():  NCCL error: unhandled system error, NCCL version 2.11.4
+1: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. It can be also caused by unexpected exit of a remote peer.
+1: Fatal Python error: Aborted
+1: 
+1: Thread 0x000014a0ec61eb80 (most recent call first):
+1:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 3152 in barrier
+1:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 228 in _compile_dependencies
+1:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 164 in initialize_megatron
+1:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/training.py", line 99 in pretrain
+1:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 231 in main
+1:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346 in wrapper
+1:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 235 in <module>
+1: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 110370 closing signal SIGTERM
+1: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 110371 closing signal SIGTERM
+1: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 110372 closing signal SIGTERM
+1: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 110375 closing signal SIGTERM
+1: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 110376 closing signal SIGTERM
+1: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 110377 closing signal SIGTERM
+1: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 110378 closing signal SIGTERM
+1: ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 0 (pid: 110369) of binary: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/bin/python
+4: Traceback (most recent call last):
+4:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 197, in _run_module_as_main
+4:     return _run_code(code, main_globals, None,
+4:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 87, in _run_code
+4:     exec(code, run_globals)
+4:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 766, in <module>
+7: ERROR:torch.distributed.elastic.multiprocessing.errors.error_handler:no error file defined for parent, to copy child error file (/tmp/torchelastic_nqvzuk6t/none_soqb0ebb/attempt_0/2/error.json)
+6: Traceback (most recent call last):
+6:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 197, in _run_module_as_main
+6:     return _run_code(code, main_globals, None,
+6:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 87, in _run_code
+7: Traceback (most recent call last):
+7:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 197, in _run_module_as_main
+6:     exec(code, run_globals)
+6:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 766, in <module>
+7:     return _run_code(code, main_globals, None,
+7:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 87, in _run_code
+7:     exec(code, run_globals)
+7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 766, in <module>
+4:     main()
+4:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
+4:     return f(*args, **kwargs)
+4:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 762, in main
+3: Traceback (most recent call last):
+3:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 197, in _run_module_as_main
+5: Traceback (most recent call last):
+5:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 197, in _run_module_as_main
+3:     return _run_code(code, main_globals, None,
+3:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 87, in _run_code
+5:     return _run_code(code, main_globals, None,
+5:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 87, in _run_code
+3:     exec(code, run_globals)
+3:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 766, in <module>
+5:     exec(code, run_globals)
+5:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 766, in <module>
+6:     main()
+6:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
+4:     run(args)
+4:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 753, in run
+7:     main()
+7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
+6:     return f(*args, **kwargs)
+6:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 762, in main
+7:     return f(*args, **kwargs)
+7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 762, in main
+4:     elastic_launch(
+4:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
+6:     run(args)
+6:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 753, in run
+5:     main()
+5:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
+4:     return launch_agent(self._config, self._entrypoint, list(args))
+4:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent
+3:     main()
+3:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
+7:     run(args)
+7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 753, in run
+5:     return f(*args, **kwargs)
+5:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 762, in main
+3:     return f(*args, **kwargs)
+3:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 762, in main
+6:     elastic_launch(
+6:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
+4:     raise ChildFailedError(
+7:     elastic_launch(
+7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
+4: torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
+4: =======================================================
+4: Megatron-DeepSpeed/pretrain_gpt.py FAILED
+4: -------------------------------------------------------
+4: Failures:
+4:   <NO_OTHER_FAILURES>
+4: -------------------------------------------------------
+4: Root Cause (first observed failure):
+4: [0]:
+4:   time      : 2023-03-17_13:50:34
+4:   host      : nid005624
+4:   rank      : 32 (local_rank: 0)
+4:   exitcode  : -6 (pid: 102636)
+4:   error_file: <N/A>
+4:   traceback : Signal 6 (SIGABRT) received by PID 102636
+4: =======================================================
+6:     return launch_agent(self._config, self._entrypoint, list(args))
+6:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent
+5:     run(args)
+5:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 753, in run
+3:     run(args)
+3:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 753, in run
+7:     return launch_agent(self._config, self._entrypoint, list(args))
+7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent
+6:     raise ChildFailedError(
+5:     elastic_launch(
+5:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
+2: Traceback (most recent call last):
+2:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 197, in _run_module_as_main
+7:     raise ChildFailedError(
+6: torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
+6: =====================================================
+6: Megatron-DeepSpeed/pretrain_gpt.py FAILED
+6: -----------------------------------------------------
+6: Failures:
+6:   <NO_OTHER_FAILURES>
+6: -----------------------------------------------------
+6: Root Cause (first observed failure):
+6: [0]:
+6:   time      : 2023-03-17_13:50:34
+6:   host      : nid005626
+6:   rank      : 48 (local_rank: 0)
+6:   exitcode  : -6 (pid: 7573)
+6:   error_file: <N/A>
+6:   traceback : Signal 6 (SIGABRT) received by PID 7573
+6: =====================================================
+3:     elastic_launch(
+3:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
+0: ERROR:torch.distributed.elastic.multiprocessing.errors.error_handler:no error file defined for parent, to copy child error file (/tmp/torchelastic__af85h8z/none_zcro5z6y/attempt_0/1/error.json)
+7: torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
+7: ============================================================
+7: Megatron-DeepSpeed/pretrain_gpt.py FAILED
+7: ------------------------------------------------------------
+7: Failures:
+7:   <NO_OTHER_FAILURES>
+7: ------------------------------------------------------------
+7: Root Cause (first observed failure):
+7: [0]:
+7:   time      : 2023-03-17_13:50:24
+7:   host      : nid005627
+7:   rank      : 58 (local_rank: 2)
+7:   exitcode  : 1 (pid: 90801)
+7:   error_file: /tmp/torchelastic_nqvzuk6t/none_soqb0ebb/attempt_0/2/error.json
+7:   traceback : Traceback (most recent call last):
+7:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1900, in _run_ninja_build
+7:       subprocess.run(
+7:     File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/subprocess.py", line 528, in run
+7:       raise CalledProcessError(retcode, process.args,
+2:     return _run_code(code, main_globals, None,
+2:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 87, in _run_code
+0: Traceback (most recent call last):
+0:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 197, in _run_module_as_main
+7:   subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
+7:   
+7:   The above exception was the direct cause of the following exception:
+7:   
+7:   Traceback (most recent call last):
+7:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1508, in _jit_compile
+7:       _write_ninja_file_and_build_library(
+7:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1623, in _write_ninja_file_and_build_library
+7:       _run_ninja_build(
+7:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1916, in _run_ninja_build
+7:       raise RuntimeError(message) from e
+2:     exec(code, run_globals)
+2:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 766, in <module>
+7:   RuntimeError: Error building extension 'scaled_masked_softmax_cuda': [1/2] /opt/rocm-5.1.0/bin/hipcc  -DWITH_HIP -DTORCH_EXTENSION_NAME=scaled_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/TH -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THC -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-
+0:     return _run_code(code, main_globals, None,
+0:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 87, in _run_code
+7: bettercom/venv/lib/python3.9/site-packages/torch/include/THH -isystem /opt/rocm-5.1.0/include -isystem /opt/rocm-5.1.0/miopen/include -isystem /opt/rocm-5.1.0/hip/include -isystem /opt/cray/pe/python/3.9.12.1/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -fPIC -D__HIP_PLATFORM_HCC__=1 -DUSE_ROCM=1 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 --amdgpu-target=gfx900 --amdgpu-target=gfx906 --amdgpu-target=gfx908 --amdgpu-target=gfx90a --amdgpu-target=gfx1030 -fno-gpu-rdc -c /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip -o scaled_masked_softmax_hip.cuda.o 
+7:   FAILED: scaled_masked_softmax_hip.cuda.o 
+5:     return launch_agent(self._config, self._entrypoint, list(args))
+5:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent
+0:     exec(code, run_globals)
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 766, in <module>
+7:   /opt/rocm-5.1.0/bin/hipcc  -DWITH_HIP -DTORCH_EXTENSION_NAME=scaled_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/TH -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THC -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THH -isystem /opt/
+7: rocm-5.1.0/include -isystem /opt/rocm-5.1.0/miopen/include -isystem /opt/rocm-5.1.0/hip/include -isystem /opt/cray/pe/python/3.9.12.1/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -fPIC -D__HIP_PLATFORM_HCC__=1 -DUSE_ROCM=1 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 --amdgpu-target=gfx900 --amdgpu-target=gfx906 --amdgpu-target=gfx908 --amdgpu-target=gfx90a --amdgpu-target=gfx1030 -fno-gpu-rdc -c /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip -o scaled_masked_softmax_hip.cuda.o 
+7:   In file included from /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip:25:
+7:   /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/ATen/hip/HIPContext.h:7:10: fatal error: 'hipsparse/hipsparse.h' file not found
+7:   #include <hipsparse/hipsparse.h>
+7:            ^~~~~~~~~~~~~~~~~~~~~~~
+7:   1 error generated when compiling for gfx1030.
+7:   ninja: build stopped: subcommand failed.
+7:   
+7:   
+7:   During handling of the above exception, another exception occurred:
+7:   
+7:   Traceback (most recent call last):
+7:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
+7:       return f(*args, **kwargs)
+7:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 231, in main
+7:       pretrain(train_valid_test_datasets_provider, model_provider, forward_step,
+7:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/training.py", line 99, in pretrain
+7:       initialize_megatron(extra_args_provider=extra_args_provider,
+7:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 164, in initialize_megatron
+7:       _compile_dependencies()
+7:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 222, in _compile_dependencies
+7:       fused_kernels.load(args)
+7:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/__init__.py", line 95, in load
+7:       scaled_masked_softmax_cuda = _cpp_extention_load_helper(
+7:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/__init__.py", line 56, in _cpp_extention_load_helper
+7:       return cpp_extension.load(
+3:     return launch_agent(self._config, self._entrypoint, list(args))
+3:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent
+7:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1284, in load
+7:       return _jit_compile(
+7:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1520, in _jit_compile
+7:       baton.release()
+7:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/file_baton.py", line 49, in release
+7:       os.remove(self.lock_file_path)
+7:   FileNotFoundError: [Errno 2] No such file or directory: '/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/build/lock'
+7:   
+7: ============================================================
+1: Traceback (most recent call last):
+1:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 197, in _run_module_as_main
+5:     raise ChildFailedError(
+1:     return _run_code(code, main_globals, None,
+1:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 87, in _run_code
+5: torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
+5: ======================================================
+5: Megatron-DeepSpeed/pretrain_gpt.py FAILED
+5: ------------------------------------------------------
+5: Failures:
+5:   <NO_OTHER_FAILURES>
+5: ------------------------------------------------------
+5: Root Cause (first observed failure):
+5: [0]:
+5:   time      : 2023-03-17_13:50:39
+5:   host      : nid005625
+5:   rank      : 40 (local_rank: 0)
+5:   exitcode  : -6 (pid: 96927)
+5:   error_file: <N/A>
+5:   traceback : Signal 6 (SIGABRT) received by PID 96927
+5: ======================================================
+3:     raise ChildFailedError(
+1:     exec(code, run_globals)
+1:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 766, in <module>
+3: torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
+3: =======================================================
+3: Megatron-DeepSpeed/pretrain_gpt.py FAILED
+3: -------------------------------------------------------
+3: Failures:
+3:   <NO_OTHER_FAILURES>
+3: -------------------------------------------------------
+3: Root Cause (first observed failure):
+3: [0]:
+3:   time      : 2023-03-17_13:50:39
+3:   host      : nid005623
+3:   rank      : 24 (local_rank: 0)
+3:   exitcode  : -6 (pid: 106548)
+3:   error_file: <N/A>
+3:   traceback : Signal 6 (SIGABRT) received by PID 106548
+3: =======================================================
+0:     main()
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
+2:     main()
+2:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
+0:     return f(*args, **kwargs)
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 762, in main
+2:     return f(*args, **kwargs)
+2:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 762, in main
+0:     run(args)
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 753, in run
+2:     run(args)
+2:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 753, in run
+0:     elastic_launch(
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
+2:     elastic_launch(
+2:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
+0:     return launch_agent(self._config, self._entrypoint, list(args))
+0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent
+1:     main()
+1:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
+1:     return f(*args, **kwargs)
+1:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 762, in main
+0:     raise ChildFailedError(
+2:     return launch_agent(self._config, self._entrypoint, list(args))
+2:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent
+0: torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
+0: ============================================================
+0: Megatron-DeepSpeed/pretrain_gpt.py FAILED
+0: ------------------------------------------------------------
+0: Failures:
+0: [1]:
+0:   time      : 2023-03-17_13:50:24
+0:   host      : nid005620
+0:   rank      : 3 (local_rank: 3)
+0:   exitcode  : 1 (pid: 106012)
+0:   error_file: /tmp/torchelastic__af85h8z/none_zcro5z6y/attempt_0/3/error.json
+0:   traceback : Traceback (most recent call last):
+0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1900, in _run_ninja_build
+0:       subprocess.run(
+0:     File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/subprocess.py", line 528, in run
+0:       raise CalledProcessError(retcode, process.args,
+0:   subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
+0:   
+0:   The above exception was the direct cause of the following exception:
+0:   
+0:   Traceback (most recent call last):
+0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1508, in _jit_compile
+0:       _write_ninja_file_and_build_library(
+0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1623, in _write_ninja_file_and_build_library
+0:       _run_ninja_build(
+0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1916, in _run_ninja_build
+0:       raise RuntimeError(message) from e
+0:   RuntimeError: Error building extension 'scaled_masked_softmax_cuda': [1/2] /opt/rocm-5.1.0/bin/hipcc  -DWITH_HIP -DTORCH_EXTENSION_NAME=scaled_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/TH -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THC -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-
+0: bettercom/venv/lib/python3.9/site-packages/torch/include/THH -isystem /opt/rocm-5.1.0/include -isystem /opt/rocm-5.1.0/miopen/include -isystem /opt/rocm-5.1.0/hip/include -isystem /opt/cray/pe/python/3.9.12.1/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -fPIC -D__HIP_PLATFORM_HCC__=1 -DUSE_ROCM=1 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 --amdgpu-target=gfx900 --amdgpu-target=gfx906 --amdgpu-target=gfx908 --amdgpu-target=gfx90a --amdgpu-target=gfx1030 -fno-gpu-rdc -c /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip -o scaled_masked_softmax_hip.cuda.o 
+0:   FAILED: scaled_masked_softmax_hip.cuda.o 
+2:     raise ChildFailedError(
+2: torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
+2: ======================================================
+2: Megatron-DeepSpeed/pretrain_gpt.py FAILED
+2: ------------------------------------------------------
+2: Failures:
+2:   <NO_OTHER_FAILURES>
+2: ------------------------------------------------------
+2: Root Cause (first observed failure):
+2: [0]:
+2:   time      : 2023-03-17_13:50:39
+2:   host      : nid005622
+2:   rank      : 16 (local_rank: 0)
+2:   exitcode  : -6 (pid: 23540)
+2:   error_file: <N/A>
+2:   traceback : Signal 6 (SIGABRT) received by PID 23540
+2: ======================================================
+0:   /opt/rocm-5.1.0/bin/hipcc  -DWITH_HIP -DTORCH_EXTENSION_NAME=scaled_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/TH -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THC -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THH -isystem /opt/
+0: rocm-5.1.0/include -isystem /opt/rocm-5.1.0/miopen/include -isystem /opt/rocm-5.1.0/hip/include -isystem /opt/cray/pe/python/3.9.12.1/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -fPIC -D__HIP_PLATFORM_HCC__=1 -DUSE_ROCM=1 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 --amdgpu-target=gfx900 --amdgpu-target=gfx906 --amdgpu-target=gfx908 --amdgpu-target=gfx90a --amdgpu-target=gfx1030 -fno-gpu-rdc -c /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip -o scaled_masked_softmax_hip.cuda.o 
+0:   In file included from /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip:25:
+0:   /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/ATen/hip/HIPContext.h:7:10: fatal error: 'hipsparse/hipsparse.h' file not found
+0:   #include <hipsparse/hipsparse.h>
+0:            ^~~~~~~~~~~~~~~~~~~~~~~
+0:   1 error generated when compiling for gfx1030.
+0:   ninja: build stopped: subcommand failed.
+0:   
+0:   
+0:   During handling of the above exception, another exception occurred:
+0:   
+0:   Traceback (most recent call last):
+0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
+0:       return f(*args, **kwargs)
+0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 231, in main
+0:       pretrain(train_valid_test_datasets_provider, model_provider, forward_step,
+1:     run(args)
+0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/training.py", line 99, in pretrain
+0:       initialize_megatron(extra_args_provider=extra_args_provider,
+0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 164, in initialize_megatron
+0:       _compile_dependencies()
+0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 222, in _compile_dependencies
+0:       fused_kernels.load(args)
+0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/__init__.py", line 95, in load
+0:       scaled_masked_softmax_cuda = _cpp_extention_load_helper(
+0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/__init__.py", line 56, in _cpp_extention_load_helper
+0:       return cpp_extension.load(
+0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1284, in load
+0:       return _jit_compile(
+0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1520, in _jit_compile
+0:       baton.release()
+0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/file_baton.py", line 49, in release
+0:       os.remove(self.lock_file_path)
+0:   FileNotFoundError: [Errno 2] No such file or directory: '/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/build/lock'
+0:   
+0: ------------------------------------------------------------
+0: Root Cause (first observed failure):
+0: [0]:
+0:   time      : 2023-03-17_13:50:24
+0:   host      : nid005620
+0:   rank      : 1 (local_rank: 1)
+0:   exitcode  : 1 (pid: 106010)
+0:   error_file: /tmp/torchelastic__af85h8z/none_zcro5z6y/attempt_0/1/error.json
+0:   traceback : Traceback (most recent call last):
+1:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 753, in run
+0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1900, in _run_ninja_build
+0:       subprocess.run(
+0:     File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/subprocess.py", line 528, in run
+0:       raise CalledProcessError(retcode, process.args,
+0:   subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
+0:   
+0:   The above exception was the direct cause of the following exception:
+0:   
+0:   Traceback (most recent call last):
+0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
+0:       return f(*args, **kwargs)
+0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 231, in main
+0:       pretrain(train_valid_test_datasets_provider, model_provider, forward_step,
+0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/training.py", line 99, in pretrain
+0:       initialize_megatron(extra_args_provider=extra_args_provider,
+0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 164, in initialize_megatron
+0:       _compile_dependencies()
+0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 222, in _compile_dependencies
+0:       fused_kernels.load(args)
+0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/__init__.py", line 95, in load
+0:       scaled_masked_softmax_cuda = _cpp_extention_load_helper(
+0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/__init__.py", line 56, in _cpp_extention_load_helper
+0:       return cpp_extension.load(
+1:     elastic_launch(
+0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1284, in load
+0:       return _jit_compile(
+0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1508, in _jit_compile
+0:       _write_ninja_file_and_build_library(
+0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1623, in _write_ninja_file_and_build_library
+0:       _run_ninja_build(
+0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1916, in _run_ninja_build
+0:       raise RuntimeError(message) from e
+0:   RuntimeError: Error building extension 'scaled_masked_softmax_cuda': [1/2] /opt/rocm-5.1.0/bin/hipcc  -DWITH_HIP -DTORCH_EXTENSION_NAME=scaled_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/TH -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THC -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-
+1:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
+0: bettercom/venv/lib/python3.9/site-packages/torch/include/THH -isystem /opt/rocm-5.1.0/include -isystem /opt/rocm-5.1.0/miopen/include -isystem /opt/rocm-5.1.0/hip/include -isystem /opt/cray/pe/python/3.9.12.1/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -fPIC -D__HIP_PLATFORM_HCC__=1 -DUSE_ROCM=1 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 --amdgpu-target=gfx900 --amdgpu-target=gfx906 --amdgpu-target=gfx908 --amdgpu-target=gfx90a --amdgpu-target=gfx1030 -fno-gpu-rdc -c /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip -o scaled_masked_softmax_hip.cuda.o 
+0:   FAILED: scaled_masked_softmax_hip.cuda.o 
+0:   /opt/rocm-5.1.0/bin/hipcc  -DWITH_HIP -DTORCH_EXTENSION_NAME=scaled_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/TH -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THC -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THH -isystem /opt/
+0: rocm-5.1.0/include -isystem /opt/rocm-5.1.0/miopen/include -isystem /opt/rocm-5.1.0/hip/include -isystem /opt/cray/pe/python/3.9.12.1/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -fPIC -D__HIP_PLATFORM_HCC__=1 -DUSE_ROCM=1 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 --amdgpu-target=gfx900 --amdgpu-target=gfx906 --amdgpu-target=gfx908 --amdgpu-target=gfx90a --amdgpu-target=gfx1030 -fno-gpu-rdc -c /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip -o scaled_masked_softmax_hip.cuda.o 
+0:   In file included from /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip:25:
+0:   /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/ATen/hip/HIPContext.h:7:10: fatal error: 'hipsparse/hipsparse.h' file not found
+0:   #include <hipsparse/hipsparse.h>
+0:            ^~~~~~~~~~~~~~~~~~~~~~~
+0:   1 error generated when compiling for gfx1030.
+0:   ninja: build stopped: subcommand failed.
+0:   
+0:   
+0: ============================================================
+1:     return launch_agent(self._config, self._entrypoint, list(args))
+1:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent
+1:     raise ChildFailedError(
+1: torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
+1: =======================================================
+1: Megatron-DeepSpeed/pretrain_gpt.py FAILED
+1: -------------------------------------------------------
+1: Failures:
+1:   <NO_OTHER_FAILURES>
+1: -------------------------------------------------------
+1: Root Cause (first observed failure):
+1: [0]:
+1:   time      : 2023-03-17_13:50:44
+1:   host      : nid005621
+1:   rank      : 8 (local_rank: 0)
+1:   exitcode  : -6 (pid: 110369)
+1:   error_file: <N/A>
+1:   traceback : Signal 6 (SIGABRT) received by PID 110369
+1: =======================================================
+srun: error: nid005620: task 0: Exited with exit code 1
+srun: launch/slurm: _step_signal: Terminating StepId=3326770.0
+srun: error: nid005625: task 5: Exited with exit code 1
+srun: error: nid005627: task 7: Exited with exit code 1
+srun: error: nid005623: task 3: Exited with exit code 1
+srun: error: nid005626: task 6: Exited with exit code 1
+srun: error: nid005624: task 4: Exited with exit code 1
+srun: error: nid005622: task 2: Exited with exit code 1
+srun: error: nid005621: task 1: Exited with exit code 1