Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics [2024-05-19 14:19:05,764] [ INFO] distributed_strategy.py:214 - distributed strategy initialized ======================= Modified FLAGS detected ======================= FLAGS(name='FLAGS_selected_gpus', current_value='1', default_value='') ======================================================================= I0519 14:19:05.765990 199 tcp_utils.cc:130] Successfully connected to 172.19.2.2:47457 I0519 14:19:05.766265 199 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 [2024-05-19 14:19:05,766] [ INFO] topology.py:358 - Total 2 pipe comm group(s) create successfully! W0519 14:19:05.767158 199 gpu_resources.cc:119] Please NOTE: device: 1, GPU Compute Capability: 7.5, Driver API Version: 12.2, Runtime API Version: 11.8 W0519 14:19:05.784675 199 gpu_resources.cc:164] device: 1, cuDNN Version: 8.9. I0519 14:19:05.968580 199 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 [2024-05-19 14:19:05,968] [ INFO] topology.py:358 - Total 1 data comm group(s) create successfully! [2024-05-19 14:19:05,968] [ INFO] topology.py:358 - Total 2 model comm group(s) create successfully! [2024-05-19 14:19:05,968] [ INFO] topology.py:358 - Total 2 sharding comm group(s) create successfully! I0519 14:19:05.969017 199 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 [2024-05-19 14:19:05,969] [ INFO] topology.py:288 - HybridParallelInfo: rank_id: 1, mp_degree: 1, sharding_degree: 1, pp_degree: 1, dp_degree: 2, sep_degree: 1, mp_group: [1], sharding_group: [1], pp_group: [1], dp_group: [0, 1], sep:group: None, check/clip group: [1] Config annotation datasets/VisDrone/datasets/VisDrone/annotations_VisDrone_train.json is not a file, dataset config is not valid Traceback (most recent call last): File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 183, in main() File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 179, in main run(FLAGS, cfg) File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 126, in run trainer = Trainer(cfg, mode='train') File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/trainer.py", line 77, in __init__ self.loader = create('{}Reader'.format(capital_mode))( File "/kaggle/working/ObjectDetection/DETR/ppdet/data/reader.py", line 167, in __call__ self.dataset.check_or_download_dataset() File "/kaggle/working/ObjectDetection/DETR/ppdet/data/source/dataset.py", line 105, in check_or_download_dataset self.dataset_dir = get_dataset_path(self.dataset_dir, self.anno_path, File "/kaggle/working/ObjectDetection/DETR/ppdet/utils/download.py", line 190, in get_dataset_path raise ValueError( ValueError: Dataset /kaggle/working/ObjectDetection/DETR/datasets/VisDrone is not valid for reason above, please check again. I0519 14:19:06.217675 199 process_group_nccl.cc:132] ProcessGroupNCCL destruct I0519 14:19:06.217739 199 process_group_nccl.cc:132] ProcessGroupNCCL destruct I0519 14:19:06.217757 199 process_group_nccl.cc:132] ProcessGroupNCCL destruct Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics [2024-05-19 14:21:15,561] [ INFO] distributed_strategy.py:214 - distributed strategy initialized ======================= Modified FLAGS detected ======================= FLAGS(name='FLAGS_selected_gpus', current_value='1', default_value='') ======================================================================= I0519 14:21:15.562182 275 tcp_utils.cc:130] Successfully connected to 172.19.2.2:58840 I0519 14:21:15.601843 275 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 [2024-05-19 14:21:15,602] [ INFO] topology.py:358 - Total 2 pipe comm group(s) create successfully! W0519 14:21:15.602743 275 gpu_resources.cc:119] Please NOTE: device: 1, GPU Compute Capability: 7.5, Driver API Version: 12.2, Runtime API Version: 11.8 W0519 14:21:15.604059 275 gpu_resources.cc:164] device: 1, cuDNN Version: 8.9. I0519 14:21:15.770931 275 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 [2024-05-19 14:21:15,771] [ INFO] topology.py:358 - Total 1 data comm group(s) create successfully! [2024-05-19 14:21:15,771] [ INFO] topology.py:358 - Total 2 model comm group(s) create successfully! [2024-05-19 14:21:15,771] [ INFO] topology.py:358 - Total 2 sharding comm group(s) create successfully! I0519 14:21:15.771346 275 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 [2024-05-19 14:21:15,771] [ INFO] topology.py:288 - HybridParallelInfo: rank_id: 1, mp_degree: 1, sharding_degree: 1, pp_degree: 1, dp_degree: 2, sep_degree: 1, mp_group: [1], sharding_group: [1], pp_group: [1], dp_group: [0, 1], sep:group: None, check/clip group: [1] Config annotation datasets/VisDrone/datasets/VisDrone/annotations_VisDrone_train.json is not a file, dataset config is not valid Traceback (most recent call last): File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 183, in main() File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 179, in main run(FLAGS, cfg) File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 126, in run trainer = Trainer(cfg, mode='train') File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/trainer.py", line 77, in __init__ self.loader = create('{}Reader'.format(capital_mode))( File "/kaggle/working/ObjectDetection/DETR/ppdet/data/reader.py", line 167, in __call__ self.dataset.check_or_download_dataset() File "/kaggle/working/ObjectDetection/DETR/ppdet/data/source/dataset.py", line 105, in check_or_download_dataset self.dataset_dir = get_dataset_path(self.dataset_dir, self.anno_path, File "/kaggle/working/ObjectDetection/DETR/ppdet/utils/download.py", line 190, in get_dataset_path raise ValueError( ValueError: Dataset /kaggle/working/ObjectDetection/DETR/datasets/VisDrone is not valid for reason above, please check again. I0519 14:21:16.015441 275 process_group_nccl.cc:132] ProcessGroupNCCL destruct I0519 14:21:16.015496 275 process_group_nccl.cc:132] ProcessGroupNCCL destruct I0519 14:21:16.015507 275 process_group_nccl.cc:132] ProcessGroupNCCL destruct Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics [2024-05-19 14:25:56,459] [ INFO] distributed_strategy.py:214 - distributed strategy initialized ======================= Modified FLAGS detected ======================= FLAGS(name='FLAGS_selected_gpus', current_value='1', default_value='') ======================================================================= I0519 14:25:56.461030 341 tcp_utils.cc:107] Retry to connect to 172.19.2.2:58530 while the server is not yet listening. I0519 14:25:59.461315 341 tcp_utils.cc:130] Successfully connected to 172.19.2.2:58530 I0519 14:25:59.489733 341 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 [2024-05-19 14:25:59,490] [ INFO] topology.py:358 - Total 2 pipe comm group(s) create successfully! W0519 14:25:59.490772 341 gpu_resources.cc:119] Please NOTE: device: 1, GPU Compute Capability: 7.5, Driver API Version: 12.2, Runtime API Version: 11.8 W0519 14:25:59.492075 341 gpu_resources.cc:164] device: 1, cuDNN Version: 8.9. I0519 14:25:59.615023 341 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 [2024-05-19 14:25:59,615] [ INFO] topology.py:358 - Total 1 data comm group(s) create successfully! [2024-05-19 14:25:59,615] [ INFO] topology.py:358 - Total 2 model comm group(s) create successfully! [2024-05-19 14:25:59,615] [ INFO] topology.py:358 - Total 2 sharding comm group(s) create successfully! I0519 14:25:59.615588 341 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 [2024-05-19 14:25:59,615] [ INFO] topology.py:288 - HybridParallelInfo: rank_id: 1, mp_degree: 1, sharding_degree: 1, pp_degree: 1, dp_degree: 2, sep_degree: 1, mp_group: [1], sharding_group: [1], pp_group: [1], dp_group: [0, 1], sep:group: None, check/clip group: [1] loading annotations into memory... Done (t=2.02s) creating index... index created! Traceback (most recent call last): File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 183, in main() File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 179, in main run(FLAGS, cfg) File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 126, in run trainer = Trainer(cfg, mode='train') File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/trainer.py", line 77, in __init__ self.loader = create('{}Reader'.format(capital_mode))( File "/kaggle/working/ObjectDetection/DETR/ppdet/data/reader.py", line 168, in __call__ self.dataset.parse_dataset() File "/kaggle/working/ObjectDetection/DETR/ppdet/data/source/coco.py", line 186, in parse_dataset gt_class[i][0] = self.catid2clsid[catid] KeyError: 0 I0519 14:26:02.515522 341 process_group_nccl.cc:132] ProcessGroupNCCL destruct I0519 14:26:02.515580 341 process_group_nccl.cc:132] ProcessGroupNCCL destruct I0519 14:26:02.515591 341 process_group_nccl.cc:132] ProcessGroupNCCL destruct Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics [2024-05-19 14:31:28,629] [ INFO] distributed_strategy.py:214 - distributed strategy initialized ======================= Modified FLAGS detected ======================= FLAGS(name='FLAGS_selected_gpus', current_value='1', default_value='') ======================================================================= I0519 14:31:28.630641 422 tcp_utils.cc:107] Retry to connect to 172.19.2.2:52124 while the server is not yet listening. I0519 14:31:31.630841 422 tcp_utils.cc:130] Successfully connected to 172.19.2.2:52124 I0519 14:31:31.659746 422 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 [2024-05-19 14:31:31,660] [ INFO] topology.py:358 - Total 2 pipe comm group(s) create successfully! W0519 14:31:31.660879 422 gpu_resources.cc:119] Please NOTE: device: 1, GPU Compute Capability: 7.5, Driver API Version: 12.2, Runtime API Version: 11.8 W0519 14:31:31.662217 422 gpu_resources.cc:164] device: 1, cuDNN Version: 8.9. I0519 14:31:31.785684 422 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 [2024-05-19 14:31:31,785] [ INFO] topology.py:358 - Total 1 data comm group(s) create successfully! [2024-05-19 14:31:31,785] [ INFO] topology.py:358 - Total 2 model comm group(s) create successfully! [2024-05-19 14:31:31,786] [ INFO] topology.py:358 - Total 2 sharding comm group(s) create successfully! I0519 14:31:31.786106 422 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 [2024-05-19 14:31:31,786] [ INFO] topology.py:288 - HybridParallelInfo: rank_id: 1, mp_degree: 1, sharding_degree: 1, pp_degree: 1, dp_degree: 2, sep_degree: 1, mp_group: [1], sharding_group: [1], pp_group: [1], dp_group: [0, 1], sep:group: None, check/clip group: [1] loading annotations into memory... Done (t=1.99s) creating index... index created! Found an invalid bbox in annotations: im_id: 201, area: 0.0 x1: 611, y1: 158, x2: 615, y2: 158. W0519 14:32:10.965493 422 reducer.cc:721] All parameters are involved in the backward pass. It is recommended to set find_unused_parameters to False to improve performance. However, if unused parameters appear in subsequent iterative training, then an error will occur. Please make it clear that in the subsequent training, there will be no parameters that are not used in the backward pass, and then set find_unused_parameters Traceback (most recent call last): File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 183, in main() File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 179, in main run(FLAGS, cfg) File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 135, in run trainer.train(FLAGS.eval) File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/trainer.py", line 377, in train outputs = model(data) File "/opt/conda/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1429, in __call__ return self.forward(*inputs, **kwargs) File "/opt/conda/lib/python3.10/site-packages/paddle/distributed/parallel.py", line 528, in forward outputs = self._layers(*inputs, **kwargs) File "/opt/conda/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1429, in __call__ return self.forward(*inputs, **kwargs) File "/kaggle/working/ObjectDetection/DETR/ppdet/modeling/architectures/meta_arch.py", line 60, in forward out = self.get_loss() File "/kaggle/working/ObjectDetection/DETR/ppdet/modeling/architectures/detr.py", line 113, in get_loss return self._forward() File "/kaggle/working/ObjectDetection/DETR/ppdet/modeling/architectures/detr.py", line 87, in _forward out_transformer = self.transformer(body_feats, pad_mask, self.inputs) File "/opt/conda/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1429, in __call__ return self.forward(*inputs, **kwargs) File "/kaggle/working/ObjectDetection/DETR/ppdet/modeling/transformers/rtdetr_transformer.py", line 434, in forward out_bboxes, out_logits = self.decoder( File "/opt/conda/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1429, in __call__ return self.forward(*inputs, **kwargs) File "/kaggle/working/ObjectDetection/DETR/ppdet/modeling/transformers/rtdetr_transformer.py", line 206, in forward output = layer(output, ref_points_input, memory, File "/opt/conda/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1429, in __call__ return self.forward(*inputs, **kwargs) File "/kaggle/working/ObjectDetection/DETR/ppdet/modeling/transformers/rtdetr_transformer.py", line 160, in forward tgt2 = self.self_attn(q, k, value=tgt, attn_mask=attn_mask) File "/opt/conda/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1429, in __call__ return self.forward(*inputs, **kwargs) File "/kaggle/working/ObjectDetection/DETR/ppdet/modeling/layers.py", line 1287, in forward product = product * scaling MemoryError: -------------------------------------- C++ Traceback (most recent call last): -------------------------------------- 0 paddle::pybind::CallScalarFuction(paddle::Tensor const&, double, std::string) 1 scale_ad_func(paddle::Tensor const&, paddle::experimental::ScalarBase, float, bool) 2 paddle::experimental::scale(paddle::Tensor const&, paddle::experimental::ScalarBase const&, float, bool) 3 void phi::ScaleKernel(phi::GPUContext const&, phi::DenseTensor const&, paddle::experimental::ScalarBase const&, float, bool, phi::DenseTensor*) 4 float* phi::DeviceContext::Alloc(phi::TensorBase*, unsigned long, bool) const 5 phi::DeviceContext::Impl::Alloc(phi::TensorBase*, phi::Place const&, phi::DataType, unsigned long, bool, bool) const 6 phi::DenseTensor::AllocateFrom(phi::Allocator*, phi::DataType, unsigned long, bool) 7 paddle::memory::allocation::Allocator::Allocate(unsigned long) 8 paddle::memory::allocation::StatAllocator::AllocateImpl(unsigned long) 9 paddle::memory::allocation::Allocator::Allocate(unsigned long) 10 paddle::memory::allocation::Allocator::Allocate(unsigned long) 11 paddle::memory::allocation::Allocator::Allocate(unsigned long) 12 paddle::memory::allocation::Allocator::Allocate(unsigned long) 13 paddle::memory::allocation::CUDAAllocator::AllocateImpl(unsigned long) 14 std::string phi::enforce::GetCompleteTraceBackString(std::string&&, char const*, int) 15 phi::enforce::GetCurrentTraceBackString[abi:cxx11](bool) ---------------------- Error Message Summary: ---------------------- ResourceExhaustedError: Out of memory error on GPU 1. Cannot allocate 71.296875MB memory on GPU 1, 14.733398GB memory has been allocated and available memory is only 15.062500MB. Please check whether there is any other process using GPU 1. 1. If yes, please stop them, or start PaddlePaddle on another GPU. 2. If no, please decrease the batch size of your model. (at /paddle/paddle/fluid/memory/allocation/cuda_allocator.cc:86) I0519 14:32:58.850878 422 process_group_nccl.cc:132] ProcessGroupNCCL destruct I0519 14:32:58.851009 422 process_group_nccl.cc:132] ProcessGroupNCCL destruct I0519 14:32:58.851034 422 process_group_nccl.cc:132] ProcessGroupNCCL destruct Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics usage: train.py [-h] [-c CONFIG] [-o [OPT ...]] [--eval] [-r RESUME] [--slim_config SLIM_CONFIG] [--enable_ce ENABLE_CE] [--amp] [--fleet] [--use_vdl USE_VDL] [--vdl_log_dir VDL_LOG_DIR] [--use_wandb USE_WANDB] [--save_prediction_only] [--profiler_options PROFILER_OPTIONS] [--save_proposals] [--proposals_path PROPOSALS_PATH] [--to_static] train.py: error: argument --use_wandb: expected one argument Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics [2024-05-19 14:36:22,326] [ INFO] distributed_strategy.py:214 - distributed strategy initialized ======================= Modified FLAGS detected ======================= FLAGS(name='FLAGS_selected_gpus', current_value='1', default_value='') ======================================================================= I0519 14:36:22.328052 629 tcp_utils.cc:107] Retry to connect to 172.19.2.2:37848 while the server is not yet listening. I0519 14:36:25.328375 629 tcp_utils.cc:130] Successfully connected to 172.19.2.2:37848 I0519 14:36:25.358937 629 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 [2024-05-19 14:36:25,359] [ INFO] topology.py:358 - Total 2 pipe comm group(s) create successfully! W0519 14:36:25.360067 629 gpu_resources.cc:119] Please NOTE: device: 1, GPU Compute Capability: 7.5, Driver API Version: 12.2, Runtime API Version: 11.8 W0519 14:36:25.361441 629 gpu_resources.cc:164] device: 1, cuDNN Version: 8.9. I0519 14:36:25.489009 629 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 [2024-05-19 14:36:25,489] [ INFO] topology.py:358 - Total 1 data comm group(s) create successfully! [2024-05-19 14:36:25,489] [ INFO] topology.py:358 - Total 2 model comm group(s) create successfully! [2024-05-19 14:36:25,489] [ INFO] topology.py:358 - Total 2 sharding comm group(s) create successfully! I0519 14:36:25.489432 629 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 [2024-05-19 14:36:25,489] [ INFO] topology.py:288 - HybridParallelInfo: rank_id: 1, mp_degree: 1, sharding_degree: 1, pp_degree: 1, dp_degree: 2, sep_degree: 1, mp_group: [1], sharding_group: [1], pp_group: [1], dp_group: [0, 1], sep:group: None, check/clip group: [1] loading annotations into memory... Done (t=1.98s) creating index... index created! Found an invalid bbox in annotations: im_id: 201, area: 0.0 x1: 611, y1: 158, x2: 615, y2: 158. W0519 14:36:40.838984 629 reducer.cc:721] All parameters are involved in the backward pass. It is recommended to set find_unused_parameters to False to improve performance. However, if unused parameters appear in subsequent iterative training, then an error will occur. Please make it clear that in the subsequent training, there will be no parameters that are not used in the backward pass, and then set find_unused_parameters Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics [2024-05-19 14:39:04,754] [ INFO] distributed_strategy.py:214 - distributed strategy initialized ======================= Modified FLAGS detected ======================= FLAGS(name='FLAGS_selected_gpus', current_value='1', default_value='') ======================================================================= I0519 14:39:04.756042 798 tcp_utils.cc:130] Successfully connected to 172.19.2.2:58013 I0519 14:39:04.769721 798 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 [2024-05-19 14:39:04,770] [ INFO] topology.py:358 - Total 2 pipe comm group(s) create successfully! W0519 14:39:04.770710 798 gpu_resources.cc:119] Please NOTE: device: 1, GPU Compute Capability: 7.5, Driver API Version: 12.2, Runtime API Version: 11.8 W0519 14:39:04.772061 798 gpu_resources.cc:164] device: 1, cuDNN Version: 8.9. I0519 14:39:04.955181 798 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 [2024-05-19 14:39:04,955] [ INFO] topology.py:358 - Total 1 data comm group(s) create successfully! [2024-05-19 14:39:04,955] [ INFO] topology.py:358 - Total 2 model comm group(s) create successfully! [2024-05-19 14:39:04,955] [ INFO] topology.py:358 - Total 2 sharding comm group(s) create successfully! I0519 14:39:04.955638 798 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 [2024-05-19 14:39:04,955] [ INFO] topology.py:288 - HybridParallelInfo: rank_id: 1, mp_degree: 1, sharding_degree: 1, pp_degree: 1, dp_degree: 2, sep_degree: 1, mp_group: [1], sharding_group: [1], pp_group: [1], dp_group: [0, 1], sep:group: None, check/clip group: [1] loading annotations into memory... Done (t=2.35s) creating index... index created! Found an invalid bbox in annotations: im_id: 201, area: 0.0 x1: 611, y1: 158, x2: 615, y2: 158. wandb not found, please install wandb. Use: `pip install wandb`. Traceback (most recent call last): File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 183, in main() File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 179, in main run(FLAGS, cfg) File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 126, in run trainer = Trainer(cfg, mode='train') File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/trainer.py", line 150, in __init__ self._init_callbacks() File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/trainer.py", line 162, in _init_callbacks self._callbacks.append(WandbCallback(self)) File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/callbacks.py", line 323, in __init__ raise e File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/callbacks.py", line 318, in __init__ import wandb File "/opt/conda/lib/python3.10/site-packages/wandb/__init__.py", line 27, in from wandb import sdk as wandb_sdk File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/__init__.py", line 25, in from .artifacts.artifact import Artifact File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/artifacts/artifact.py", line 46, in from wandb.apis.normalize import normalize_exceptions File "/opt/conda/lib/python3.10/site-packages/wandb/apis/__init__.py", line 43, in from .internal import Api as InternalApi # noqa File "/opt/conda/lib/python3.10/site-packages/wandb/apis/internal.py", line 3, in from wandb.sdk.internal.internal_api import Api as InternalApi File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/internal/internal_api.py", line 48, in from ..lib import retry File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/lib/retry.py", line 17, in from .mailbox import ContextCancelledError File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/lib/mailbox.py", line 102, in class _MailboxSlot: File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/lib/mailbox.py", line 103, in _MailboxSlot _result: Optional[pb.Result] AttributeError: module 'wandb.proto.wandb_internal_pb2' has no attribute 'Result' I0519 14:39:20.125581 798 process_group_nccl.cc:132] ProcessGroupNCCL destruct I0519 14:39:20.126137 798 process_group_nccl.cc:132] ProcessGroupNCCL destruct I0519 14:39:20.126168 798 process_group_nccl.cc:132] ProcessGroupNCCL destruct Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics [2024-05-19 14:47:12,782] [ INFO] distributed_strategy.py:214 - distributed strategy initialized ======================= Modified FLAGS detected ======================= FLAGS(name='FLAGS_selected_gpus', current_value='1', default_value='') ======================================================================= I0519 14:47:12.783265 179 tcp_utils.cc:107] Retry to connect to 172.19.2.2:41929 while the server is not yet listening. I0519 14:47:15.783504 179 tcp_utils.cc:130] Successfully connected to 172.19.2.2:41929 I0519 14:47:15.811667 179 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 [2024-05-19 14:47:15,812] [ INFO] topology.py:358 - Total 2 pipe comm group(s) create successfully! W0519 14:47:15.812635 179 gpu_resources.cc:119] Please NOTE: device: 1, GPU Compute Capability: 7.5, Driver API Version: 12.2, Runtime API Version: 11.8 W0519 14:47:15.826958 179 gpu_resources.cc:164] device: 1, cuDNN Version: 8.9. I0519 14:47:15.988464 179 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 [2024-05-19 14:47:15,988] [ INFO] topology.py:358 - Total 1 data comm group(s) create successfully! [2024-05-19 14:47:15,988] [ INFO] topology.py:358 - Total 2 model comm group(s) create successfully! [2024-05-19 14:47:15,988] [ INFO] topology.py:358 - Total 2 sharding comm group(s) create successfully! I0519 14:47:15.988860 179 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 [2024-05-19 14:47:15,988] [ INFO] topology.py:288 - HybridParallelInfo: rank_id: 1, mp_degree: 1, sharding_degree: 1, pp_degree: 1, dp_degree: 2, sep_degree: 1, mp_group: [1], sharding_group: [1], pp_group: [1], dp_group: [0, 1], sep:group: None, check/clip group: [1] loading annotations into memory... Done (t=1.97s) creating index... index created! Found an invalid bbox in annotations: im_id: 201, area: 0.0 x1: 611, y1: 158, x2: 615, y2: 158. W0519 14:48:20.116335 179 reducer.cc:721] All parameters are involved in the backward pass. It is recommended to set find_unused_parameters to False to improve performance. However, if unused parameters appear in subsequent iterative training, then an error will occur. Please make it clear that in the subsequent training, there will be no parameters that are not used in the backward pass, and then set find_unused_parameters Traceback (most recent call last): File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 183, in main() File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 179, in main run(FLAGS, cfg) File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 135, in run trainer.train(FLAGS.eval) File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/trainer.py", line 377, in train outputs = model(data) File "/root/.local/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1429, in __call__ return self.forward(*inputs, **kwargs) File "/root/.local/lib/python3.10/site-packages/paddle/distributed/parallel.py", line 528, in forward outputs = self._layers(*inputs, **kwargs) File "/root/.local/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1429, in __call__ return self.forward(*inputs, **kwargs) File "/kaggle/working/ObjectDetection/DETR/ppdet/modeling/architectures/meta_arch.py", line 60, in forward out = self.get_loss() File "/kaggle/working/ObjectDetection/DETR/ppdet/modeling/architectures/detr.py", line 113, in get_loss return self._forward() File "/kaggle/working/ObjectDetection/DETR/ppdet/modeling/architectures/detr.py", line 87, in _forward out_transformer = self.transformer(body_feats, pad_mask, self.inputs) File "/root/.local/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1429, in __call__ return self.forward(*inputs, **kwargs) File "/kaggle/working/ObjectDetection/DETR/ppdet/modeling/transformers/rtdetr_transformer.py", line 434, in forward out_bboxes, out_logits = self.decoder( File "/root/.local/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1429, in __call__ return self.forward(*inputs, **kwargs) File "/kaggle/working/ObjectDetection/DETR/ppdet/modeling/transformers/rtdetr_transformer.py", line 206, in forward output = layer(output, ref_points_input, memory, File "/root/.local/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1429, in __call__ return self.forward(*inputs, **kwargs) File "/kaggle/working/ObjectDetection/DETR/ppdet/modeling/transformers/rtdetr_transformer.py", line 160, in forward tgt2 = self.self_attn(q, k, value=tgt, attn_mask=attn_mask) File "/root/.local/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1429, in __call__ return self.forward(*inputs, **kwargs) File "/kaggle/working/ObjectDetection/DETR/ppdet/modeling/layers.py", line 1287, in forward product = product * scaling MemoryError: -------------------------------------- C++ Traceback (most recent call last): -------------------------------------- 0 paddle::pybind::CallScalarFuction(paddle::Tensor const&, double, std::string) 1 scale_ad_func(paddle::Tensor const&, paddle::experimental::ScalarBase, float, bool) 2 paddle::experimental::scale(paddle::Tensor const&, paddle::experimental::ScalarBase const&, float, bool) 3 void phi::ScaleKernel(phi::GPUContext const&, phi::DenseTensor const&, paddle::experimental::ScalarBase const&, float, bool, phi::DenseTensor*) 4 float* phi::DeviceContext::Alloc(phi::TensorBase*, unsigned long, bool) const 5 phi::DeviceContext::Impl::Alloc(phi::TensorBase*, phi::Place const&, phi::DataType, unsigned long, bool, bool) const 6 phi::DenseTensor::AllocateFrom(phi::Allocator*, phi::DataType, unsigned long, bool) 7 paddle::memory::allocation::Allocator::Allocate(unsigned long) 8 paddle::memory::allocation::StatAllocator::AllocateImpl(unsigned long) 9 paddle::memory::allocation::Allocator::Allocate(unsigned long) 10 paddle::memory::allocation::Allocator::Allocate(unsigned long) 11 paddle::memory::allocation::Allocator::Allocate(unsigned long) 12 paddle::memory::allocation::Allocator::Allocate(unsigned long) 13 paddle::memory::allocation::CUDAAllocator::AllocateImpl(unsigned long) 14 std::string phi::enforce::GetCompleteTraceBackString(std::string&&, char const*, int) 15 phi::enforce::GetCurrentTraceBackString[abi:cxx11](bool) ---------------------- Error Message Summary: ---------------------- ResourceExhaustedError: Out of memory error on GPU 1. Cannot allocate 540.382812MB memory on GPU 1, 14.481445GB memory has been allocated and available memory is only 273.062500MB. Please check whether there is any other process using GPU 1. 1. If yes, please stop them, or start PaddlePaddle on another GPU. 2. If no, please decrease the batch size of your model. (at /paddle/paddle/fluid/memory/allocation/cuda_allocator.cc:86) I0519 23:16:03.615808 179 process_group_nccl.cc:132] ProcessGroupNCCL destruct I0519 23:16:03.616861 179 process_group_nccl.cc:132] ProcessGroupNCCL destruct I0519 23:16:03.616915 179 process_group_nccl.cc:132] ProcessGroupNCCL destruct Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics [2024-05-20 01:06:18,200] [ INFO] distributed_strategy.py:214 - distributed strategy initialized ======================= Modified FLAGS detected ======================= FLAGS(name='FLAGS_selected_gpus', current_value='1', default_value='') ======================================================================= I0520 01:06:18.201324 214 tcp_utils.cc:107] Retry to connect to 172.19.2.2:45378 while the server is not yet listening. I0520 01:06:21.201637 214 tcp_utils.cc:130] Successfully connected to 172.19.2.2:45378 I0520 01:06:21.230141 214 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 [2024-05-20 01:06:21,230] [ INFO] topology.py:358 - Total 2 pipe comm group(s) create successfully! W0520 01:06:21.231158 214 gpu_resources.cc:119] Please NOTE: device: 1, GPU Compute Capability: 7.5, Driver API Version: 12.2, Runtime API Version: 11.8 W0520 01:06:21.242787 214 gpu_resources.cc:164] device: 1, cuDNN Version: 8.9. I0520 01:06:21.446182 214 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 [2024-05-20 01:06:21,446] [ INFO] topology.py:358 - Total 1 data comm group(s) create successfully! [2024-05-20 01:06:21,446] [ INFO] topology.py:358 - Total 2 model comm group(s) create successfully! [2024-05-20 01:06:21,446] [ INFO] topology.py:358 - Total 2 sharding comm group(s) create successfully! I0520 01:06:21.446622 214 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 [2024-05-20 01:06:21,446] [ INFO] topology.py:288 - HybridParallelInfo: rank_id: 1, mp_degree: 1, sharding_degree: 1, pp_degree: 1, dp_degree: 2, sep_degree: 1, mp_group: [1], sharding_group: [1], pp_group: [1], dp_group: [0, 1], sep:group: None, check/clip group: [1] loading annotations into memory... Done (t=2.07s) creating index... index created! Found an invalid bbox in annotations: im_id: 201, area: 0.0 x1: 611, y1: 158, x2: 615, y2: 158. Traceback (most recent call last): File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 183, in main() File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 179, in main run(FLAGS, cfg) File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 130, in run trainer.resume_weights(FLAGS.resume) File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/trainer.py", line 259, in resume_weights self.start_epoch = load_weight(self.model, weights, self.optimizer, File "/kaggle/working/ObjectDetection/DETR/ppdet/utils/checkpoint.py", line 55, in load_weight raise ValueError("Model pretrain path {} does not " ValueError: Model pretrain path /kaggle/working/ObjectDetection/DETR/output/rtdetr_hgnetv2_x_6x_coco/latest.pdparams does not exists. I0520 01:06:37.745325 214 process_group_nccl.cc:132] ProcessGroupNCCL destruct I0520 01:06:37.745388 214 process_group_nccl.cc:132] ProcessGroupNCCL destruct I0520 01:06:37.745405 214 process_group_nccl.cc:132] ProcessGroupNCCL destruct Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics [2024-05-20 01:07:01,880] [ INFO] distributed_strategy.py:214 - distributed strategy initialized ======================= Modified FLAGS detected ======================= FLAGS(name='FLAGS_selected_gpus', current_value='1', default_value='') ======================================================================= I0520 01:07:01.881565 364 tcp_utils.cc:130] Successfully connected to 172.19.2.2:36395 I0520 01:07:01.921115 364 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 [2024-05-20 01:07:01,921] [ INFO] topology.py:358 - Total 2 pipe comm group(s) create successfully! W0520 01:07:01.922013 364 gpu_resources.cc:119] Please NOTE: device: 1, GPU Compute Capability: 7.5, Driver API Version: 12.2, Runtime API Version: 11.8 W0520 01:07:01.923400 364 gpu_resources.cc:164] device: 1, cuDNN Version: 8.9. I0520 01:07:02.108011 364 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 [2024-05-20 01:07:02,108] [ INFO] topology.py:358 - Total 1 data comm group(s) create successfully! [2024-05-20 01:07:02,108] [ INFO] topology.py:358 - Total 2 model comm group(s) create successfully! [2024-05-20 01:07:02,108] [ INFO] topology.py:358 - Total 2 sharding comm group(s) create successfully! I0520 01:07:02.108417 364 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 [2024-05-20 01:07:02,108] [ INFO] topology.py:288 - HybridParallelInfo: rank_id: 1, mp_degree: 1, sharding_degree: 1, pp_degree: 1, dp_degree: 2, sep_degree: 1, mp_group: [1], sharding_group: [1], pp_group: [1], dp_group: [0, 1], sep:group: None, check/clip group: [1] loading annotations into memory... Done (t=1.99s) creating index... index created! Found an invalid bbox in annotations: im_id: 201, area: 0.0 x1: 611, y1: 158, x2: 615, y2: 158. W0520 01:07:37.872128 364 reducer.cc:721] All parameters are involved in the backward pass. It is recommended to set find_unused_parameters to False to improve performance. However, if unused parameters appear in subsequent iterative training, then an error will occur. Please make it clear that in the subsequent training, there will be no parameters that are not used in the backward pass, and then set find_unused_parameters -------------------------------------- C++ Traceback (most recent call last): -------------------------------------- 0 paddle::pybind::eager_api_sync_batch_norm_(_object*, _object*, _object*) 1 sync_batch_norm__ad_func(paddle::Tensor const&, paddle::Tensor&, paddle::Tensor&, paddle::Tensor const&, paddle::Tensor const&, bool, float, float, std::string, bool, bool) 2 paddle::experimental::sync_batch_norm_(paddle::Tensor const&, paddle::Tensor&, paddle::Tensor&, paddle::Tensor const&, paddle::Tensor const&, bool, float, float, std::string const&, bool, bool) 3 void phi::SyncBatchNormKernel(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, bool, float, float, std::string const&, bool, bool, phi::DenseTensor*, phi::DenseTensor*, phi::DenseTensor*, phi::DenseTensor*, phi::DenseTensor*, phi::DenseTensor*) ---------------------- Error Message Summary: ---------------------- FatalError: `Termination signal` is detected by the operating system. [TimeInfo: *** Aborted at 1716178280 (unix time) try "date -d @1716178280" if you are using GNU date ***] [SignalInfo: *** SIGTERM (@0x15c) received by PID 364 (TID 0x7d69c38aa740) from PID 348 ***]