|
Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics |
|
Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics |
|
[2024-05-19 14:19:05,764] [ INFO] distributed_strategy.py:214 - distributed strategy initialized |
|
======================= Modified FLAGS detected ======================= |
|
FLAGS(name='FLAGS_selected_gpus', current_value='1', default_value='') |
|
======================================================================= |
|
I0519 14:19:05.765990 199 tcp_utils.cc:130] Successfully connected to 172.19.2.2:47457 |
|
I0519 14:19:05.766265 199 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 |
|
[2024-05-19 14:19:05,766] [ INFO] topology.py:358 - Total 2 pipe comm group(s) create successfully! |
|
W0519 14:19:05.767158 199 gpu_resources.cc:119] Please NOTE: device: 1, GPU Compute Capability: 7.5, Driver API Version: 12.2, Runtime API Version: 11.8 |
|
W0519 14:19:05.784675 199 gpu_resources.cc:164] device: 1, cuDNN Version: 8.9. |
|
I0519 14:19:05.968580 199 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 |
|
[2024-05-19 14:19:05,968] [ INFO] topology.py:358 - Total 1 data comm group(s) create successfully! |
|
[2024-05-19 14:19:05,968] [ INFO] topology.py:358 - Total 2 model comm group(s) create successfully! |
|
[2024-05-19 14:19:05,968] [ INFO] topology.py:358 - Total 2 sharding comm group(s) create successfully! |
|
I0519 14:19:05.969017 199 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 |
|
[2024-05-19 14:19:05,969] [ INFO] topology.py:288 - HybridParallelInfo: rank_id: 1, mp_degree: 1, sharding_degree: 1, pp_degree: 1, dp_degree: 2, sep_degree: 1, mp_group: [1], sharding_group: [1], pp_group: [1], dp_group: [0, 1], sep:group: None, check/clip group: [1] |
|
Config annotation datasets/VisDrone/datasets/VisDrone/annotations_VisDrone_train.json is not a file, dataset config is not valid |
|
Traceback (most recent call last): |
|
File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 183, in <module> |
|
main() |
|
File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 179, in main |
|
run(FLAGS, cfg) |
|
File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 126, in run |
|
trainer = Trainer(cfg, mode='train') |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/trainer.py", line 77, in __init__ |
|
self.loader = create('{}Reader'.format(capital_mode))( |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/data/reader.py", line 167, in __call__ |
|
self.dataset.check_or_download_dataset() |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/data/source/dataset.py", line 105, in check_or_download_dataset |
|
self.dataset_dir = get_dataset_path(self.dataset_dir, self.anno_path, |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/utils/download.py", line 190, in get_dataset_path |
|
raise ValueError( |
|
ValueError: Dataset /kaggle/working/ObjectDetection/DETR/datasets/VisDrone is not valid for reason above, please check again. |
|
I0519 14:19:06.217675 199 process_group_nccl.cc:132] ProcessGroupNCCL destruct |
|
I0519 14:19:06.217739 199 process_group_nccl.cc:132] ProcessGroupNCCL destruct |
|
I0519 14:19:06.217757 199 process_group_nccl.cc:132] ProcessGroupNCCL destruct |
|
Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics |
|
Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics |
|
[2024-05-19 14:21:15,561] [ INFO] distributed_strategy.py:214 - distributed strategy initialized |
|
======================= Modified FLAGS detected ======================= |
|
FLAGS(name='FLAGS_selected_gpus', current_value='1', default_value='') |
|
======================================================================= |
|
I0519 14:21:15.562182 275 tcp_utils.cc:130] Successfully connected to 172.19.2.2:58840 |
|
I0519 14:21:15.601843 275 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 |
|
[2024-05-19 14:21:15,602] [ INFO] topology.py:358 - Total 2 pipe comm group(s) create successfully! |
|
W0519 14:21:15.602743 275 gpu_resources.cc:119] Please NOTE: device: 1, GPU Compute Capability: 7.5, Driver API Version: 12.2, Runtime API Version: 11.8 |
|
W0519 14:21:15.604059 275 gpu_resources.cc:164] device: 1, cuDNN Version: 8.9. |
|
I0519 14:21:15.770931 275 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 |
|
[2024-05-19 14:21:15,771] [ INFO] topology.py:358 - Total 1 data comm group(s) create successfully! |
|
[2024-05-19 14:21:15,771] [ INFO] topology.py:358 - Total 2 model comm group(s) create successfully! |
|
[2024-05-19 14:21:15,771] [ INFO] topology.py:358 - Total 2 sharding comm group(s) create successfully! |
|
I0519 14:21:15.771346 275 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 |
|
[2024-05-19 14:21:15,771] [ INFO] topology.py:288 - HybridParallelInfo: rank_id: 1, mp_degree: 1, sharding_degree: 1, pp_degree: 1, dp_degree: 2, sep_degree: 1, mp_group: [1], sharding_group: [1], pp_group: [1], dp_group: [0, 1], sep:group: None, check/clip group: [1] |
|
Config annotation datasets/VisDrone/datasets/VisDrone/annotations_VisDrone_train.json is not a file, dataset config is not valid |
|
Traceback (most recent call last): |
|
File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 183, in <module> |
|
main() |
|
File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 179, in main |
|
run(FLAGS, cfg) |
|
File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 126, in run |
|
trainer = Trainer(cfg, mode='train') |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/trainer.py", line 77, in __init__ |
|
self.loader = create('{}Reader'.format(capital_mode))( |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/data/reader.py", line 167, in __call__ |
|
self.dataset.check_or_download_dataset() |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/data/source/dataset.py", line 105, in check_or_download_dataset |
|
self.dataset_dir = get_dataset_path(self.dataset_dir, self.anno_path, |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/utils/download.py", line 190, in get_dataset_path |
|
raise ValueError( |
|
ValueError: Dataset /kaggle/working/ObjectDetection/DETR/datasets/VisDrone is not valid for reason above, please check again. |
|
I0519 14:21:16.015441 275 process_group_nccl.cc:132] ProcessGroupNCCL destruct |
|
I0519 14:21:16.015496 275 process_group_nccl.cc:132] ProcessGroupNCCL destruct |
|
I0519 14:21:16.015507 275 process_group_nccl.cc:132] ProcessGroupNCCL destruct |
|
Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics |
|
Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics |
|
[2024-05-19 14:25:56,459] [ INFO] distributed_strategy.py:214 - distributed strategy initialized |
|
======================= Modified FLAGS detected ======================= |
|
FLAGS(name='FLAGS_selected_gpus', current_value='1', default_value='') |
|
======================================================================= |
|
I0519 14:25:56.461030 341 tcp_utils.cc:107] Retry to connect to 172.19.2.2:58530 while the server is not yet listening. |
|
I0519 14:25:59.461315 341 tcp_utils.cc:130] Successfully connected to 172.19.2.2:58530 |
|
I0519 14:25:59.489733 341 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 |
|
[2024-05-19 14:25:59,490] [ INFO] topology.py:358 - Total 2 pipe comm group(s) create successfully! |
|
W0519 14:25:59.490772 341 gpu_resources.cc:119] Please NOTE: device: 1, GPU Compute Capability: 7.5, Driver API Version: 12.2, Runtime API Version: 11.8 |
|
W0519 14:25:59.492075 341 gpu_resources.cc:164] device: 1, cuDNN Version: 8.9. |
|
I0519 14:25:59.615023 341 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 |
|
[2024-05-19 14:25:59,615] [ INFO] topology.py:358 - Total 1 data comm group(s) create successfully! |
|
[2024-05-19 14:25:59,615] [ INFO] topology.py:358 - Total 2 model comm group(s) create successfully! |
|
[2024-05-19 14:25:59,615] [ INFO] topology.py:358 - Total 2 sharding comm group(s) create successfully! |
|
I0519 14:25:59.615588 341 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 |
|
[2024-05-19 14:25:59,615] [ INFO] topology.py:288 - HybridParallelInfo: rank_id: 1, mp_degree: 1, sharding_degree: 1, pp_degree: 1, dp_degree: 2, sep_degree: 1, mp_group: [1], sharding_group: [1], pp_group: [1], dp_group: [0, 1], sep:group: None, check/clip group: [1] |
|
loading annotations into memory... |
|
Done (t=2.02s) |
|
creating index... |
|
index created! |
|
Traceback (most recent call last): |
|
File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 183, in <module> |
|
main() |
|
File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 179, in main |
|
run(FLAGS, cfg) |
|
File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 126, in run |
|
trainer = Trainer(cfg, mode='train') |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/trainer.py", line 77, in __init__ |
|
self.loader = create('{}Reader'.format(capital_mode))( |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/data/reader.py", line 168, in __call__ |
|
self.dataset.parse_dataset() |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/data/source/coco.py", line 186, in parse_dataset |
|
gt_class[i][0] = self.catid2clsid[catid] |
|
KeyError: 0 |
|
I0519 14:26:02.515522 341 process_group_nccl.cc:132] ProcessGroupNCCL destruct |
|
I0519 14:26:02.515580 341 process_group_nccl.cc:132] ProcessGroupNCCL destruct |
|
I0519 14:26:02.515591 341 process_group_nccl.cc:132] ProcessGroupNCCL destruct |
|
Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics |
|
Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics |
|
[2024-05-19 14:31:28,629] [ INFO] distributed_strategy.py:214 - distributed strategy initialized |
|
======================= Modified FLAGS detected ======================= |
|
FLAGS(name='FLAGS_selected_gpus', current_value='1', default_value='') |
|
======================================================================= |
|
I0519 14:31:28.630641 422 tcp_utils.cc:107] Retry to connect to 172.19.2.2:52124 while the server is not yet listening. |
|
I0519 14:31:31.630841 422 tcp_utils.cc:130] Successfully connected to 172.19.2.2:52124 |
|
I0519 14:31:31.659746 422 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 |
|
[2024-05-19 14:31:31,660] [ INFO] topology.py:358 - Total 2 pipe comm group(s) create successfully! |
|
W0519 14:31:31.660879 422 gpu_resources.cc:119] Please NOTE: device: 1, GPU Compute Capability: 7.5, Driver API Version: 12.2, Runtime API Version: 11.8 |
|
W0519 14:31:31.662217 422 gpu_resources.cc:164] device: 1, cuDNN Version: 8.9. |
|
I0519 14:31:31.785684 422 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 |
|
[2024-05-19 14:31:31,785] [ INFO] topology.py:358 - Total 1 data comm group(s) create successfully! |
|
[2024-05-19 14:31:31,785] [ INFO] topology.py:358 - Total 2 model comm group(s) create successfully! |
|
[2024-05-19 14:31:31,786] [ INFO] topology.py:358 - Total 2 sharding comm group(s) create successfully! |
|
I0519 14:31:31.786106 422 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 |
|
[2024-05-19 14:31:31,786] [ INFO] topology.py:288 - HybridParallelInfo: rank_id: 1, mp_degree: 1, sharding_degree: 1, pp_degree: 1, dp_degree: 2, sep_degree: 1, mp_group: [1], sharding_group: [1], pp_group: [1], dp_group: [0, 1], sep:group: None, check/clip group: [1] |
|
loading annotations into memory... |
|
Done (t=1.99s) |
|
creating index... |
|
index created! |
|
Found an invalid bbox in annotations: im_id: 201, area: 0.0 x1: 611, y1: 158, x2: 615, y2: 158. |
|
W0519 14:32:10.965493 422 reducer.cc:721] All parameters are involved in the backward pass. It is recommended to set find_unused_parameters to False to improve performance. However, if unused parameters appear in subsequent iterative training, then an error will occur. Please make it clear that in the subsequent training, there will be no parameters that are not used in the backward pass, and then set find_unused_parameters |
|
Traceback (most recent call last): |
|
File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 183, in <module> |
|
main() |
|
File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 179, in main |
|
run(FLAGS, cfg) |
|
File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 135, in run |
|
trainer.train(FLAGS.eval) |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/trainer.py", line 377, in train |
|
outputs = model(data) |
|
File "/opt/conda/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1429, in __call__ |
|
return self.forward(*inputs, **kwargs) |
|
File "/opt/conda/lib/python3.10/site-packages/paddle/distributed/parallel.py", line 528, in forward |
|
outputs = self._layers(*inputs, **kwargs) |
|
File "/opt/conda/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1429, in __call__ |
|
return self.forward(*inputs, **kwargs) |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/modeling/architectures/meta_arch.py", line 60, in forward |
|
out = self.get_loss() |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/modeling/architectures/detr.py", line 113, in get_loss |
|
return self._forward() |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/modeling/architectures/detr.py", line 87, in _forward |
|
out_transformer = self.transformer(body_feats, pad_mask, self.inputs) |
|
File "/opt/conda/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1429, in __call__ |
|
return self.forward(*inputs, **kwargs) |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/modeling/transformers/rtdetr_transformer.py", line 434, in forward |
|
out_bboxes, out_logits = self.decoder( |
|
File "/opt/conda/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1429, in __call__ |
|
return self.forward(*inputs, **kwargs) |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/modeling/transformers/rtdetr_transformer.py", line 206, in forward |
|
output = layer(output, ref_points_input, memory, |
|
File "/opt/conda/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1429, in __call__ |
|
return self.forward(*inputs, **kwargs) |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/modeling/transformers/rtdetr_transformer.py", line 160, in forward |
|
tgt2 = self.self_attn(q, k, value=tgt, attn_mask=attn_mask) |
|
File "/opt/conda/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1429, in __call__ |
|
return self.forward(*inputs, **kwargs) |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/modeling/layers.py", line 1287, in forward |
|
product = product * scaling |
|
MemoryError: |
|
|
|
|
|
C++ Traceback (most recent call last): |
|
|
|
0 paddle::pybind::CallScalarFuction(paddle::Tensor const&, double, std::string) |
|
1 scale_ad_func(paddle::Tensor const&, paddle::experimental::ScalarBase<paddle::Tensor>, float, bool) |
|
2 paddle::experimental::scale(paddle::Tensor const&, paddle::experimental::ScalarBase<paddle::Tensor> const&, float, bool) |
|
3 void phi::ScaleKernel<float, phi::GPUContext>(phi::GPUContext const&, phi::DenseTensor const&, paddle::experimental::ScalarBase<phi::DenseTensor> const&, float, bool, phi::DenseTensor*) |
|
4 float* phi::DeviceContext::Alloc<float>(phi::TensorBase*, unsigned long, bool) const |
|
5 phi::DeviceContext::Impl::Alloc(phi::TensorBase*, phi::Place const&, phi::DataType, unsigned long, bool, bool) const |
|
6 phi::DenseTensor::AllocateFrom(phi::Allocator*, phi::DataType, unsigned long, bool) |
|
7 paddle::memory::allocation::Allocator::Allocate(unsigned long) |
|
8 paddle::memory::allocation::StatAllocator::AllocateImpl(unsigned long) |
|
9 paddle::memory::allocation::Allocator::Allocate(unsigned long) |
|
10 paddle::memory::allocation::Allocator::Allocate(unsigned long) |
|
11 paddle::memory::allocation::Allocator::Allocate(unsigned long) |
|
12 paddle::memory::allocation::Allocator::Allocate(unsigned long) |
|
13 paddle::memory::allocation::CUDAAllocator::AllocateImpl(unsigned long) |
|
14 std::string phi::enforce::GetCompleteTraceBackString<std::string >(std::string&&, char const*, int) |
|
15 phi::enforce::GetCurrentTraceBackString[abi:cxx11](bool) |
|
|
|
|
|
Error Message Summary: |
|
|
|
ResourceExhaustedError: |
|
|
|
Out of memory error on GPU 1. Cannot allocate 71.296875MB memory on GPU 1, 14.733398GB memory has been allocated and available memory is only 15.062500MB. |
|
|
|
Please check whether there is any other process using GPU 1. |
|
1. If yes, please stop them, or start PaddlePaddle on another GPU. |
|
2. If no, please decrease the batch size of your model. |
|
(at /paddle/paddle/fluid/memory/allocation/cuda_allocator.cc:86) |
|
|
|
I0519 14:32:58.850878 422 process_group_nccl.cc:132] ProcessGroupNCCL destruct |
|
I0519 14:32:58.851009 422 process_group_nccl.cc:132] ProcessGroupNCCL destruct |
|
I0519 14:32:58.851034 422 process_group_nccl.cc:132] ProcessGroupNCCL destruct |
|
Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics |
|
Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics |
|
usage: train.py [-h] [-c CONFIG] [-o [OPT ...]] [ |
|
[ |
|
[ |
|
[ |
|
[ |
|
[ |
|
train.py: error: argument |
|
Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics |
|
Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics |
|
[2024-05-19 14:36:22,326] [ INFO] distributed_strategy.py:214 - distributed strategy initialized |
|
======================= Modified FLAGS detected ======================= |
|
FLAGS(name='FLAGS_selected_gpus', current_value='1', default_value='') |
|
======================================================================= |
|
I0519 14:36:22.328052 629 tcp_utils.cc:107] Retry to connect to 172.19.2.2:37848 while the server is not yet listening. |
|
I0519 14:36:25.328375 629 tcp_utils.cc:130] Successfully connected to 172.19.2.2:37848 |
|
I0519 14:36:25.358937 629 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 |
|
[2024-05-19 14:36:25,359] [ INFO] topology.py:358 - Total 2 pipe comm group(s) create successfully! |
|
W0519 14:36:25.360067 629 gpu_resources.cc:119] Please NOTE: device: 1, GPU Compute Capability: 7.5, Driver API Version: 12.2, Runtime API Version: 11.8 |
|
W0519 14:36:25.361441 629 gpu_resources.cc:164] device: 1, cuDNN Version: 8.9. |
|
I0519 14:36:25.489009 629 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 |
|
[2024-05-19 14:36:25,489] [ INFO] topology.py:358 - Total 1 data comm group(s) create successfully! |
|
[2024-05-19 14:36:25,489] [ INFO] topology.py:358 - Total 2 model comm group(s) create successfully! |
|
[2024-05-19 14:36:25,489] [ INFO] topology.py:358 - Total 2 sharding comm group(s) create successfully! |
|
I0519 14:36:25.489432 629 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 |
|
[2024-05-19 14:36:25,489] [ INFO] topology.py:288 - HybridParallelInfo: rank_id: 1, mp_degree: 1, sharding_degree: 1, pp_degree: 1, dp_degree: 2, sep_degree: 1, mp_group: [1], sharding_group: [1], pp_group: [1], dp_group: [0, 1], sep:group: None, check/clip group: [1] |
|
loading annotations into memory... |
|
Done (t=1.98s) |
|
creating index... |
|
index created! |
|
Found an invalid bbox in annotations: im_id: 201, area: 0.0 x1: 611, y1: 158, x2: 615, y2: 158. |
|
W0519 14:36:40.838984 629 reducer.cc:721] All parameters are involved in the backward pass. It is recommended to set find_unused_parameters to False to improve performance. However, if unused parameters appear in subsequent iterative training, then an error will occur. Please make it clear that in the subsequent training, there will be no parameters that are not used in the backward pass, and then set find_unused_parameters |
|
Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics |
|
Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics |
|
[2024-05-19 14:39:04,754] [ INFO] distributed_strategy.py:214 - distributed strategy initialized |
|
======================= Modified FLAGS detected ======================= |
|
FLAGS(name='FLAGS_selected_gpus', current_value='1', default_value='') |
|
======================================================================= |
|
I0519 14:39:04.756042 798 tcp_utils.cc:130] Successfully connected to 172.19.2.2:58013 |
|
I0519 14:39:04.769721 798 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 |
|
[2024-05-19 14:39:04,770] [ INFO] topology.py:358 - Total 2 pipe comm group(s) create successfully! |
|
W0519 14:39:04.770710 798 gpu_resources.cc:119] Please NOTE: device: 1, GPU Compute Capability: 7.5, Driver API Version: 12.2, Runtime API Version: 11.8 |
|
W0519 14:39:04.772061 798 gpu_resources.cc:164] device: 1, cuDNN Version: 8.9. |
|
I0519 14:39:04.955181 798 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 |
|
[2024-05-19 14:39:04,955] [ INFO] topology.py:358 - Total 1 data comm group(s) create successfully! |
|
[2024-05-19 14:39:04,955] [ INFO] topology.py:358 - Total 2 model comm group(s) create successfully! |
|
[2024-05-19 14:39:04,955] [ INFO] topology.py:358 - Total 2 sharding comm group(s) create successfully! |
|
I0519 14:39:04.955638 798 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 |
|
[2024-05-19 14:39:04,955] [ INFO] topology.py:288 - HybridParallelInfo: rank_id: 1, mp_degree: 1, sharding_degree: 1, pp_degree: 1, dp_degree: 2, sep_degree: 1, mp_group: [1], sharding_group: [1], pp_group: [1], dp_group: [0, 1], sep:group: None, check/clip group: [1] |
|
loading annotations into memory... |
|
Done (t=2.35s) |
|
creating index... |
|
index created! |
|
Found an invalid bbox in annotations: im_id: 201, area: 0.0 x1: 611, y1: 158, x2: 615, y2: 158. |
|
wandb not found, please install wandb. Use: `pip install wandb`. |
|
Traceback (most recent call last): |
|
File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 183, in <module> |
|
main() |
|
File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 179, in main |
|
run(FLAGS, cfg) |
|
File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 126, in run |
|
trainer = Trainer(cfg, mode='train') |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/trainer.py", line 150, in __init__ |
|
self._init_callbacks() |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/trainer.py", line 162, in _init_callbacks |
|
self._callbacks.append(WandbCallback(self)) |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/callbacks.py", line 323, in __init__ |
|
raise e |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/callbacks.py", line 318, in __init__ |
|
import wandb |
|
File "/opt/conda/lib/python3.10/site-packages/wandb/__init__.py", line 27, in <module> |
|
from wandb import sdk as wandb_sdk |
|
File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/__init__.py", line 25, in <module> |
|
from .artifacts.artifact import Artifact |
|
File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/artifacts/artifact.py", line 46, in <module> |
|
from wandb.apis.normalize import normalize_exceptions |
|
File "/opt/conda/lib/python3.10/site-packages/wandb/apis/__init__.py", line 43, in <module> |
|
from .internal import Api as InternalApi # noqa |
|
File "/opt/conda/lib/python3.10/site-packages/wandb/apis/internal.py", line 3, in <module> |
|
from wandb.sdk.internal.internal_api import Api as InternalApi |
|
File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/internal/internal_api.py", line 48, in <module> |
|
from ..lib import retry |
|
File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/lib/retry.py", line 17, in <module> |
|
from .mailbox import ContextCancelledError |
|
File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/lib/mailbox.py", line 102, in <module> |
|
class _MailboxSlot: |
|
File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/lib/mailbox.py", line 103, in _MailboxSlot |
|
_result: Optional[pb.Result] |
|
AttributeError: module 'wandb.proto.wandb_internal_pb2' has no attribute 'Result' |
|
I0519 14:39:20.125581 798 process_group_nccl.cc:132] ProcessGroupNCCL destruct |
|
I0519 14:39:20.126137 798 process_group_nccl.cc:132] ProcessGroupNCCL destruct |
|
I0519 14:39:20.126168 798 process_group_nccl.cc:132] ProcessGroupNCCL destruct |
|
Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics |
|
Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics |
|
[2024-05-19 14:47:12,782] [ INFO] distributed_strategy.py:214 - distributed strategy initialized |
|
======================= Modified FLAGS detected ======================= |
|
FLAGS(name='FLAGS_selected_gpus', current_value='1', default_value='') |
|
======================================================================= |
|
I0519 14:47:12.783265 179 tcp_utils.cc:107] Retry to connect to 172.19.2.2:41929 while the server is not yet listening. |
|
I0519 14:47:15.783504 179 tcp_utils.cc:130] Successfully connected to 172.19.2.2:41929 |
|
I0519 14:47:15.811667 179 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 |
|
[2024-05-19 14:47:15,812] [ INFO] topology.py:358 - Total 2 pipe comm group(s) create successfully! |
|
W0519 14:47:15.812635 179 gpu_resources.cc:119] Please NOTE: device: 1, GPU Compute Capability: 7.5, Driver API Version: 12.2, Runtime API Version: 11.8 |
|
W0519 14:47:15.826958 179 gpu_resources.cc:164] device: 1, cuDNN Version: 8.9. |
|
I0519 14:47:15.988464 179 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 |
|
[2024-05-19 14:47:15,988] [ INFO] topology.py:358 - Total 1 data comm group(s) create successfully! |
|
[2024-05-19 14:47:15,988] [ INFO] topology.py:358 - Total 2 model comm group(s) create successfully! |
|
[2024-05-19 14:47:15,988] [ INFO] topology.py:358 - Total 2 sharding comm group(s) create successfully! |
|
I0519 14:47:15.988860 179 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 |
|
[2024-05-19 14:47:15,988] [ INFO] topology.py:288 - HybridParallelInfo: rank_id: 1, mp_degree: 1, sharding_degree: 1, pp_degree: 1, dp_degree: 2, sep_degree: 1, mp_group: [1], sharding_group: [1], pp_group: [1], dp_group: [0, 1], sep:group: None, check/clip group: [1] |
|
loading annotations into memory... |
|
Done (t=1.97s) |
|
creating index... |
|
index created! |
|
Found an invalid bbox in annotations: im_id: 201, area: 0.0 x1: 611, y1: 158, x2: 615, y2: 158. |
|
W0519 14:48:20.116335 179 reducer.cc:721] All parameters are involved in the backward pass. It is recommended to set find_unused_parameters to False to improve performance. However, if unused parameters appear in subsequent iterative training, then an error will occur. Please make it clear that in the subsequent training, there will be no parameters that are not used in the backward pass, and then set find_unused_parameters |
|
Traceback (most recent call last): |
|
File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 183, in <module> |
|
main() |
|
File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 179, in main |
|
run(FLAGS, cfg) |
|
File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 135, in run |
|
trainer.train(FLAGS.eval) |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/trainer.py", line 377, in train |
|
outputs = model(data) |
|
File "/root/.local/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1429, in __call__ |
|
return self.forward(*inputs, **kwargs) |
|
File "/root/.local/lib/python3.10/site-packages/paddle/distributed/parallel.py", line 528, in forward |
|
outputs = self._layers(*inputs, **kwargs) |
|
File "/root/.local/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1429, in __call__ |
|
return self.forward(*inputs, **kwargs) |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/modeling/architectures/meta_arch.py", line 60, in forward |
|
out = self.get_loss() |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/modeling/architectures/detr.py", line 113, in get_loss |
|
return self._forward() |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/modeling/architectures/detr.py", line 87, in _forward |
|
out_transformer = self.transformer(body_feats, pad_mask, self.inputs) |
|
File "/root/.local/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1429, in __call__ |
|
return self.forward(*inputs, **kwargs) |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/modeling/transformers/rtdetr_transformer.py", line 434, in forward |
|
out_bboxes, out_logits = self.decoder( |
|
File "/root/.local/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1429, in __call__ |
|
return self.forward(*inputs, **kwargs) |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/modeling/transformers/rtdetr_transformer.py", line 206, in forward |
|
output = layer(output, ref_points_input, memory, |
|
File "/root/.local/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1429, in __call__ |
|
return self.forward(*inputs, **kwargs) |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/modeling/transformers/rtdetr_transformer.py", line 160, in forward |
|
tgt2 = self.self_attn(q, k, value=tgt, attn_mask=attn_mask) |
|
File "/root/.local/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1429, in __call__ |
|
return self.forward(*inputs, **kwargs) |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/modeling/layers.py", line 1287, in forward |
|
product = product * scaling |
|
MemoryError: |
|
|
|
|
|
C++ Traceback (most recent call last): |
|
|
|
0 paddle::pybind::CallScalarFuction(paddle::Tensor const&, double, std::string) |
|
1 scale_ad_func(paddle::Tensor const&, paddle::experimental::ScalarBase<paddle::Tensor>, float, bool) |
|
2 paddle::experimental::scale(paddle::Tensor const&, paddle::experimental::ScalarBase<paddle::Tensor> const&, float, bool) |
|
3 void phi::ScaleKernel<float, phi::GPUContext>(phi::GPUContext const&, phi::DenseTensor const&, paddle::experimental::ScalarBase<phi::DenseTensor> const&, float, bool, phi::DenseTensor*) |
|
4 float* phi::DeviceContext::Alloc<float>(phi::TensorBase*, unsigned long, bool) const |
|
5 phi::DeviceContext::Impl::Alloc(phi::TensorBase*, phi::Place const&, phi::DataType, unsigned long, bool, bool) const |
|
6 phi::DenseTensor::AllocateFrom(phi::Allocator*, phi::DataType, unsigned long, bool) |
|
7 paddle::memory::allocation::Allocator::Allocate(unsigned long) |
|
8 paddle::memory::allocation::StatAllocator::AllocateImpl(unsigned long) |
|
9 paddle::memory::allocation::Allocator::Allocate(unsigned long) |
|
10 paddle::memory::allocation::Allocator::Allocate(unsigned long) |
|
11 paddle::memory::allocation::Allocator::Allocate(unsigned long) |
|
12 paddle::memory::allocation::Allocator::Allocate(unsigned long) |
|
13 paddle::memory::allocation::CUDAAllocator::AllocateImpl(unsigned long) |
|
14 std::string phi::enforce::GetCompleteTraceBackString<std::string >(std::string&&, char const*, int) |
|
15 phi::enforce::GetCurrentTraceBackString[abi:cxx11](bool) |
|
|
|
|
|
Error Message Summary: |
|
|
|
ResourceExhaustedError: |
|
|
|
Out of memory error on GPU 1. Cannot allocate 540.382812MB memory on GPU 1, 14.481445GB memory has been allocated and available memory is only 273.062500MB. |
|
|
|
Please check whether there is any other process using GPU 1. |
|
1. If yes, please stop them, or start PaddlePaddle on another GPU. |
|
2. If no, please decrease the batch size of your model. |
|
(at /paddle/paddle/fluid/memory/allocation/cuda_allocator.cc:86) |
|
|
|
I0519 23:16:03.615808 179 process_group_nccl.cc:132] ProcessGroupNCCL destruct |
|
I0519 23:16:03.616861 179 process_group_nccl.cc:132] ProcessGroupNCCL destruct |
|
I0519 23:16:03.616915 179 process_group_nccl.cc:132] ProcessGroupNCCL destruct |
|
Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics |
|
Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics |
|
[2024-05-20 01:06:18,200] [ INFO] distributed_strategy.py:214 - distributed strategy initialized |
|
======================= Modified FLAGS detected ======================= |
|
FLAGS(name='FLAGS_selected_gpus', current_value='1', default_value='') |
|
======================================================================= |
|
I0520 01:06:18.201324 214 tcp_utils.cc:107] Retry to connect to 172.19.2.2:45378 while the server is not yet listening. |
|
I0520 01:06:21.201637 214 tcp_utils.cc:130] Successfully connected to 172.19.2.2:45378 |
|
I0520 01:06:21.230141 214 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 |
|
[2024-05-20 01:06:21,230] [ INFO] topology.py:358 - Total 2 pipe comm group(s) create successfully! |
|
W0520 01:06:21.231158 214 gpu_resources.cc:119] Please NOTE: device: 1, GPU Compute Capability: 7.5, Driver API Version: 12.2, Runtime API Version: 11.8 |
|
W0520 01:06:21.242787 214 gpu_resources.cc:164] device: 1, cuDNN Version: 8.9. |
|
I0520 01:06:21.446182 214 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 |
|
[2024-05-20 01:06:21,446] [ INFO] topology.py:358 - Total 1 data comm group(s) create successfully! |
|
[2024-05-20 01:06:21,446] [ INFO] topology.py:358 - Total 2 model comm group(s) create successfully! |
|
[2024-05-20 01:06:21,446] [ INFO] topology.py:358 - Total 2 sharding comm group(s) create successfully! |
|
I0520 01:06:21.446622 214 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 |
|
[2024-05-20 01:06:21,446] [ INFO] topology.py:288 - HybridParallelInfo: rank_id: 1, mp_degree: 1, sharding_degree: 1, pp_degree: 1, dp_degree: 2, sep_degree: 1, mp_group: [1], sharding_group: [1], pp_group: [1], dp_group: [0, 1], sep:group: None, check/clip group: [1] |
|
loading annotations into memory... |
|
Done (t=2.07s) |
|
creating index... |
|
index created! |
|
Found an invalid bbox in annotations: im_id: 201, area: 0.0 x1: 611, y1: 158, x2: 615, y2: 158. |
|
Traceback (most recent call last): |
|
File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 183, in <module> |
|
main() |
|
File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 179, in main |
|
run(FLAGS, cfg) |
|
File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 130, in run |
|
trainer.resume_weights(FLAGS.resume) |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/trainer.py", line 259, in resume_weights |
|
self.start_epoch = load_weight(self.model, weights, self.optimizer, |
|
File "/kaggle/working/ObjectDetection/DETR/ppdet/utils/checkpoint.py", line 55, in load_weight |
|
raise ValueError("Model pretrain path {} does not " |
|
ValueError: Model pretrain path /kaggle/working/ObjectDetection/DETR/output/rtdetr_hgnetv2_x_6x_coco/latest.pdparams does not exists. |
|
I0520 01:06:37.745325 214 process_group_nccl.cc:132] ProcessGroupNCCL destruct |
|
I0520 01:06:37.745388 214 process_group_nccl.cc:132] ProcessGroupNCCL destruct |
|
I0520 01:06:37.745405 214 process_group_nccl.cc:132] ProcessGroupNCCL destruct |
|
Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics |
|
Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics |
|
[2024-05-20 01:07:01,880] [ INFO] distributed_strategy.py:214 - distributed strategy initialized |
|
======================= Modified FLAGS detected ======================= |
|
FLAGS(name='FLAGS_selected_gpus', current_value='1', default_value='') |
|
======================================================================= |
|
I0520 01:07:01.881565 364 tcp_utils.cc:130] Successfully connected to 172.19.2.2:36395 |
|
I0520 01:07:01.921115 364 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 |
|
[2024-05-20 01:07:01,921] [ INFO] topology.py:358 - Total 2 pipe comm group(s) create successfully! |
|
W0520 01:07:01.922013 364 gpu_resources.cc:119] Please NOTE: device: 1, GPU Compute Capability: 7.5, Driver API Version: 12.2, Runtime API Version: 11.8 |
|
W0520 01:07:01.923400 364 gpu_resources.cc:164] device: 1, cuDNN Version: 8.9. |
|
I0520 01:07:02.108011 364 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 |
|
[2024-05-20 01:07:02,108] [ INFO] topology.py:358 - Total 1 data comm group(s) create successfully! |
|
[2024-05-20 01:07:02,108] [ INFO] topology.py:358 - Total 2 model comm group(s) create successfully! |
|
[2024-05-20 01:07:02,108] [ INFO] topology.py:358 - Total 2 sharding comm group(s) create successfully! |
|
I0520 01:07:02.108417 364 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000 |
|
[2024-05-20 01:07:02,108] [ INFO] topology.py:288 - HybridParallelInfo: rank_id: 1, mp_degree: 1, sharding_degree: 1, pp_degree: 1, dp_degree: 2, sep_degree: 1, mp_group: [1], sharding_group: [1], pp_group: [1], dp_group: [0, 1], sep:group: None, check/clip group: [1] |
|
loading annotations into memory... |
|
Done (t=1.99s) |
|
creating index... |
|
index created! |
|
Found an invalid bbox in annotations: im_id: 201, area: 0.0 x1: 611, y1: 158, x2: 615, y2: 158. |
|
W0520 01:07:37.872128 364 reducer.cc:721] All parameters are involved in the backward pass. It is recommended to set find_unused_parameters to False to improve performance. However, if unused parameters appear in subsequent iterative training, then an error will occur. Please make it clear that in the subsequent training, there will be no parameters that are not used in the backward pass, and then set find_unused_parameters |
|
|
|
|
|
|
|
C++ Traceback (most recent call last): |
|
|
|
0 paddle::pybind::eager_api_sync_batch_norm_(_object*, _object*, _object*) |
|
1 sync_batch_norm__ad_func(paddle::Tensor const&, paddle::Tensor&, paddle::Tensor&, paddle::Tensor const&, paddle::Tensor const&, bool, float, float, std::string, bool, bool) |
|
2 paddle::experimental::sync_batch_norm_(paddle::Tensor const&, paddle::Tensor&, paddle::Tensor&, paddle::Tensor const&, paddle::Tensor const&, bool, float, float, std::string const&, bool, bool) |
|
3 void phi::SyncBatchNormKernel<float, phi::GPUContext>(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, bool, float, float, std::string const&, bool, bool, phi::DenseTensor*, phi::DenseTensor*, phi::DenseTensor*, phi::DenseTensor*, phi::DenseTensor*, phi::DenseTensor*) |
|
|
|
|
|
Error Message Summary: |
|
|
|
FatalError: `Termination signal` is detected by the operating system. |
|
[TimeInfo: *** Aborted at 1716178280 (unix time) try "date -d @1716178280" if you are using GNU date ***] |
|
[SignalInfo: *** SIGTERM (@0x15c) received by PID 364 (TID 0x7d69c38aa740) from PID 348 ***] |
|
|
|
|
