Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics
Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics
[2024-05-19 14:19:05,764] [    INFO] distributed_strategy.py:214 - distributed strategy initialized
======================= Modified FLAGS detected =======================
FLAGS(name='FLAGS_selected_gpus', current_value='0', default_value='')
=======================================================================
I0519 14:19:05.765937   197 tcp_utils.cc:181] The server starts to listen on IP_ANY:47457
I0519 14:19:05.766125   197 tcp_utils.cc:130] Successfully connected to 172.19.2.2:47457
I0519 14:19:05.847883   197 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000
[2024-05-19 14:19:05,848] [    INFO] topology.py:358 - Total 2 pipe comm group(s) create successfully!
W0519 14:19:05.851115   197 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 7.5, Driver API Version: 12.2, Runtime API Version: 11.8
W0519 14:19:05.852351   197 gpu_resources.cc:164] device: 0, cuDNN Version: 8.9.
I0519 14:19:05.977974   197 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000
[2024-05-19 14:19:05,978] [    INFO] topology.py:358 - Total 1 data comm group(s) create successfully!
[2024-05-19 14:19:05,978] [    INFO] topology.py:358 - Total 2 model comm group(s) create successfully!
[2024-05-19 14:19:05,978] [    INFO] topology.py:358 - Total 2 sharding comm group(s) create successfully!
I0519 14:19:05.978358   197 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000
[2024-05-19 14:19:05,978] [    INFO] topology.py:288 - HybridParallelInfo: rank_id: 0, mp_degree: 1, sharding_degree: 1, pp_degree: 1, dp_degree: 2, sep_degree: 1, mp_group: [0],  sharding_group: [0], pp_group: [0], dp_group: [0, 1], sep:group: None, check/clip group: [0]
[05/19 14:19:05] ppdet.utils.download WARNING: Config annotation datasets/VisDrone/datasets/VisDrone/annotations_VisDrone_train.json is not a file, dataset config is not valid
Traceback (most recent call last):
  File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 183, in <module>
    main()
  File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 179, in main
    run(FLAGS, cfg)
  File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 126, in run
    trainer = Trainer(cfg, mode='train')
  File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/trainer.py", line 77, in __init__
    self.loader = create('{}Reader'.format(capital_mode))(
  File "/kaggle/working/ObjectDetection/DETR/ppdet/data/reader.py", line 167, in __call__
    self.dataset.check_or_download_dataset()
  File "/kaggle/working/ObjectDetection/DETR/ppdet/data/source/dataset.py", line 105, in check_or_download_dataset
    self.dataset_dir = get_dataset_path(self.dataset_dir, self.anno_path,
  File "/kaggle/working/ObjectDetection/DETR/ppdet/utils/download.py", line 190, in get_dataset_path
    raise ValueError(
ValueError: Dataset /kaggle/working/ObjectDetection/DETR/datasets/VisDrone is not valid for reason above, please check again.
I0519 14:19:06.230679   197 process_group_nccl.cc:132] ProcessGroupNCCL destruct 
I0519 14:19:06.230718   197 process_group_nccl.cc:132] ProcessGroupNCCL destruct 
I0519 14:19:06.230729   197 process_group_nccl.cc:132] ProcessGroupNCCL destruct 
I0519 14:19:06.272063   223 tcp_store.cc:289] receive shutdown event and so quit from MasterDaemon run loop
Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics
Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics
[2024-05-19 14:21:15,559] [    INFO] distributed_strategy.py:214 - distributed strategy initialized
======================= Modified FLAGS detected =======================
FLAGS(name='FLAGS_selected_gpus', current_value='0', default_value='')
=======================================================================
I0519 14:21:15.560766   273 tcp_utils.cc:181] The server starts to listen on IP_ANY:58840
I0519 14:21:15.560976   273 tcp_utils.cc:130] Successfully connected to 172.19.2.2:58840
I0519 14:21:15.642843   273 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000
[2024-05-19 14:21:15,643] [    INFO] topology.py:358 - Total 2 pipe comm group(s) create successfully!
W0519 14:21:15.644209   273 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 7.5, Driver API Version: 12.2, Runtime API Version: 11.8
W0519 14:21:15.645530   273 gpu_resources.cc:164] device: 0, cuDNN Version: 8.9.
I0519 14:21:15.788317   273 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000
[2024-05-19 14:21:15,788] [    INFO] topology.py:358 - Total 1 data comm group(s) create successfully!
[2024-05-19 14:21:15,788] [    INFO] topology.py:358 - Total 2 model comm group(s) create successfully!
[2024-05-19 14:21:15,788] [    INFO] topology.py:358 - Total 2 sharding comm group(s) create successfully!
I0519 14:21:15.788738   273 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000
[2024-05-19 14:21:15,788] [    INFO] topology.py:288 - HybridParallelInfo: rank_id: 0, mp_degree: 1, sharding_degree: 1, pp_degree: 1, dp_degree: 2, sep_degree: 1, mp_group: [0],  sharding_group: [0], pp_group: [0], dp_group: [0, 1], sep:group: None, check/clip group: [0]
[05/19 14:21:15] ppdet.utils.download WARNING: Config annotation datasets/VisDrone/datasets/VisDrone/annotations_VisDrone_train.json is not a file, dataset config is not valid
Traceback (most recent call last):
  File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 183, in <module>
    main()
  File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 179, in main
    run(FLAGS, cfg)
  File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 126, in run
    trainer = Trainer(cfg, mode='train')
  File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/trainer.py", line 77, in __init__
    self.loader = create('{}Reader'.format(capital_mode))(
  File "/kaggle/working/ObjectDetection/DETR/ppdet/data/reader.py", line 167, in __call__
    self.dataset.check_or_download_dataset()
  File "/kaggle/working/ObjectDetection/DETR/ppdet/data/source/dataset.py", line 105, in check_or_download_dataset
    self.dataset_dir = get_dataset_path(self.dataset_dir, self.anno_path,
  File "/kaggle/working/ObjectDetection/DETR/ppdet/utils/download.py", line 190, in get_dataset_path
    raise ValueError(
ValueError: Dataset /kaggle/working/ObjectDetection/DETR/datasets/VisDrone is not valid for reason above, please check again.
I0519 14:21:16.035830   273 process_group_nccl.cc:132] ProcessGroupNCCL destruct 
I0519 14:21:16.035882   273 process_group_nccl.cc:132] ProcessGroupNCCL destruct 
I0519 14:21:16.035892   273 process_group_nccl.cc:132] ProcessGroupNCCL destruct 
I0519 14:21:16.075630   299 tcp_store.cc:289] receive shutdown event and so quit from MasterDaemon run loop
Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics
Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics
[2024-05-19 14:25:56,462] [    INFO] distributed_strategy.py:214 - distributed strategy initialized
======================= Modified FLAGS detected =======================
FLAGS(name='FLAGS_selected_gpus', current_value='0', default_value='')
=======================================================================
I0519 14:25:56.463665   339 tcp_utils.cc:181] The server starts to listen on IP_ANY:58530
I0519 14:25:56.463861   339 tcp_utils.cc:130] Successfully connected to 172.19.2.2:58530
I0519 14:25:59.581831   339 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000
[2024-05-19 14:25:59,582] [    INFO] topology.py:358 - Total 2 pipe comm group(s) create successfully!
W0519 14:25:59.583158   339 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 7.5, Driver API Version: 12.2, Runtime API Version: 11.8
W0519 14:25:59.584975   339 gpu_resources.cc:164] device: 0, cuDNN Version: 8.9.
I0519 14:25:59.709707   339 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000
[2024-05-19 14:25:59,709] [    INFO] topology.py:358 - Total 1 data comm group(s) create successfully!
[2024-05-19 14:25:59,709] [    INFO] topology.py:358 - Total 2 model comm group(s) create successfully!
[2024-05-19 14:25:59,710] [    INFO] topology.py:358 - Total 2 sharding comm group(s) create successfully!
I0519 14:25:59.710146   339 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000
[2024-05-19 14:25:59,710] [    INFO] topology.py:288 - HybridParallelInfo: rank_id: 0, mp_degree: 1, sharding_degree: 1, pp_degree: 1, dp_degree: 2, sep_degree: 1, mp_group: [0],  sharding_group: [0], pp_group: [0], dp_group: [0, 1], sep:group: None, check/clip group: [0]
loading annotations into memory...
Done (t=2.01s)
creating index...
index created!
Traceback (most recent call last):
  File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 183, in <module>
    main()
  File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 179, in main
    run(FLAGS, cfg)
  File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 126, in run
    trainer = Trainer(cfg, mode='train')
  File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/trainer.py", line 77, in __init__
    self.loader = create('{}Reader'.format(capital_mode))(
  File "/kaggle/working/ObjectDetection/DETR/ppdet/data/reader.py", line 168, in __call__
    self.dataset.parse_dataset()
  File "/kaggle/working/ObjectDetection/DETR/ppdet/data/source/coco.py", line 186, in parse_dataset
    gt_class[i][0] = self.catid2clsid[catid]
KeyError: 0
I0519 14:26:02.580338   339 process_group_nccl.cc:132] ProcessGroupNCCL destruct 
I0519 14:26:02.580395   339 process_group_nccl.cc:132] ProcessGroupNCCL destruct 
I0519 14:26:02.580405   339 process_group_nccl.cc:132] ProcessGroupNCCL destruct 
I0519 14:26:02.621611   365 tcp_store.cc:289] receive shutdown event and so quit from MasterDaemon run loop
Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics
Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics
[2024-05-19 14:31:28,631] [    INFO] distributed_strategy.py:214 - distributed strategy initialized
======================= Modified FLAGS detected =======================
FLAGS(name='FLAGS_selected_gpus', current_value='0', default_value='')
=======================================================================
I0519 14:31:28.632526   420 tcp_utils.cc:181] The server starts to listen on IP_ANY:52124
I0519 14:31:28.632692   420 tcp_utils.cc:130] Successfully connected to 172.19.2.2:52124
I0519 14:31:31.751803   420 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000
[2024-05-19 14:31:31,752] [    INFO] topology.py:358 - Total 2 pipe comm group(s) create successfully!
W0519 14:31:31.752864   420 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 7.5, Driver API Version: 12.2, Runtime API Version: 11.8
W0519 14:31:31.754097   420 gpu_resources.cc:164] device: 0, cuDNN Version: 8.9.
I0519 14:31:31.888082   420 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000
[2024-05-19 14:31:31,888] [    INFO] topology.py:358 - Total 1 data comm group(s) create successfully!
[2024-05-19 14:31:31,888] [    INFO] topology.py:358 - Total 2 model comm group(s) create successfully!
[2024-05-19 14:31:31,888] [    INFO] topology.py:358 - Total 2 sharding comm group(s) create successfully!
I0519 14:31:31.888540   420 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000
[2024-05-19 14:31:31,888] [    INFO] topology.py:288 - HybridParallelInfo: rank_id: 0, mp_degree: 1, sharding_degree: 1, pp_degree: 1, dp_degree: 2, sep_degree: 1, mp_group: [0],  sharding_group: [0], pp_group: [0], dp_group: [0, 1], sep:group: None, check/clip group: [0]
loading annotations into memory...
Done (t=1.94s)
creating index...
index created!
[05/19 14:31:34] ppdet.data.source.coco WARNING: Found an invalid bbox in annotations: im_id: 201, area: 0.0 x1: 611, y1: 158, x2: 615, y2: 158.
[05/19 14:31:39] ppdet.data.source.coco INFO: Load [6471 samples valid, 0 samples invalid] in file datasets/VisDrone/annotations_VisDrone_train.json.
[05/19 14:31:45] ppdet.utils.download INFO: Downloading PPHGNetV2_X_ssld_pretrained.pdparams from https://bj.bcebos.com/v1/paddledet/models/pretrained/PPHGNetV2_X_ssld_pretrained.pdparams

  0%|          | 0/155004 [00:00<?, ?KB/s]
  0%|          | 3/155004 [00:00<2:08:19, 20.13KB/s]
  0%|          | 19/155004 [00:00<39:31, 65.34KB/s] 
  0%|          | 35/155004 [00:00<33:01, 78.20KB/s]
  0%|          | 51/155004 [00:00<30:44, 83.99KB/s]
  0%|          | 67/155004 [00:00<29:37, 87.14KB/s]
  0%|          | 83/155004 [00:01<29:00, 89.01KB/s]
  0%|          | 99/155004 [00:01<28:37, 90.18KB/s]
  0%|          | 115/155004 [00:01<28:23, 90.93KB/s]
  0%|          | 131/155004 [00:01<25:26, 101.47KB/s]
  0%|          | 163/155004 [00:01<22:46, 113.32KB/s]
  0%|          | 179/155004 [00:01<23:17, 110.75KB/s]
  0%|          | 211/155004 [00:02<20:01, 128.79KB/s]
  0%|          | 227/155004 [00:02<21:06, 122.25KB/s]
  0%|          | 259/155004 [00:02<18:15, 141.32KB/s]
  0%|          | 275/155004 [00:02<18:29, 139.42KB/s]
  0%|          | 307/155004 [00:02<18:24, 140.03KB/s]
  0%|          | 339/155004 [00:02<16:50, 153.05KB/s]
  0%|          | 371/155004 [00:03<15:49, 162.82KB/s]
  0%|          | 403/155004 [00:03<15:11, 169.52KB/s]
  0%|          | 435/155004 [00:03<14:44, 174.69KB/s]
  0%|          | 467/155004 [00:03<13:06, 196.39KB/s]
  0%|          | 499/155004 [00:03<12:18, 209.13KB/s]
  0%|          | 521/155004 [00:03<12:12, 210.95KB/s]
  0%|          | 563/155004 [00:03<11:35, 221.96KB/s]
  0%|          | 611/155004 [00:04<10:41, 240.74KB/s]
  0%|          | 659/155004 [00:04<10:04, 255.51KB/s]
  0%|          | 707/155004 [00:04<08:48, 291.81KB/s]
  0%|          | 755/155004 [00:04<08:13, 312.61KB/s]
  1%|          | 803/155004 [00:04<07:34, 339.26KB/s]
  1%|          | 851/155004 [00:04<07:46, 330.67KB/s]
  1%|          | 915/155004 [00:04<06:38, 386.58KB/s]
  1%|          | 979/155004 [00:05<05:49, 441.25KB/s]
  1%|          | 1027/155004 [00:05<05:59, 427.91KB/s]
  1%|          | 1091/155004 [00:05<05:33, 460.96KB/s]
  1%|          | 1171/155004 [00:05<05:02, 508.14KB/s]
  1%|          | 1235/155004 [00:05<04:54, 521.57KB/s]
  1%|          | 1315/155004 [00:05<04:54, 520.99KB/s]
  1%|          | 1411/155004 [00:05<04:15, 602.11KB/s]
  1%|          | 1507/155004 [00:05<03:45, 680.48KB/s]
  1%|          | 1577/155004 [00:06<03:56, 649.22KB/s]
  1%|          | 1683/155004 [00:06<03:32, 722.97KB/s]
  1%|          | 1795/155004 [00:06<03:20, 764.95KB/s]
  1%|          | 1891/155004 [00:06<03:15, 783.31KB/s]
  1%|▏         | 2019/155004 [00:06<03:10, 800.98KB/s]
  1%|▏         | 2163/155004 [00:06<02:46, 917.59KB/s]
  1%|▏         | 2291/155004 [00:06<02:33, 994.48KB/s]
  2%|▏         | 2393/155004 [00:06<02:41, 946.71KB/s]
  2%|▏         | 2547/155004 [00:07<02:24, 1052.61KB/s]
  2%|▏         | 2691/155004 [00:07<02:13, 1136.93KB/s]
  2%|▏         | 2807/155004 [00:07<02:20, 1079.72KB/s]
  2%|▏         | 2979/155004 [00:07<02:07, 1194.64KB/s]
  2%|▏         | 3155/155004 [00:07<01:54, 1321.13KB/s]
  2%|▏         | 3289/155004 [00:07<02:00, 1254.16KB/s]
  2%|▏         | 3475/155004 [00:07<01:52, 1352.62KB/s]
  2%|▏         | 3683/155004 [00:07<01:39, 1522.88KB/s]
  2%|▏         | 3838/155004 [00:07<01:44, 1444.46KB/s]
  3%|▎         | 4051/155004 [00:08<01:37, 1555.17KB/s]
  3%|▎         | 4275/155004 [00:08<01:28, 1711.98KB/s]
  3%|▎         | 4449/155004 [00:08<01:32, 1621.07KB/s]
  3%|▎         | 4691/155004 [00:08<01:25, 1754.20KB/s]
  3%|▎         | 4963/155004 [00:08<01:15, 1976.29KB/s]
  3%|▎         | 5163/155004 [00:08<01:19, 1873.03KB/s]
  4%|▎         | 5427/155004 [00:08<01:15, 1985.84KB/s]
  4%|▎         | 5715/155004 [00:08<01:08, 2190.93KB/s]
  4%|▍         | 5937/155004 [00:08<01:11, 2072.49KB/s]
  4%|▍         | 6243/155004 [00:09<01:06, 2235.81KB/s]
  4%|▍         | 6579/155004 [00:09<00:59, 2493.19KB/s]
  4%|▍         | 6831/155004 [00:09<01:02, 2357.95KB/s]
  5%|▍         | 7155/155004 [00:09<00:59, 2484.19KB/s]
  5%|▍         | 7523/155004 [00:09<00:53, 2761.95KB/s]
  5%|▌         | 7803/155004 [00:09<00:56, 2607.66KB/s]
  5%|▌         | 8179/155004 [00:09<00:52, 2794.36KB/s]
  6%|▌         | 8579/155004 [00:09<00:47, 3070.10KB/s]
  6%|▌         | 8890/155004 [00:10<00:50, 2894.39KB/s]
  6%|▌         | 9299/155004 [00:10<00:47, 3080.88KB/s]
  6%|▋         | 9763/155004 [00:10<00:42, 3445.21KB/s]
  7%|▋         | 10112/155004 [00:10<00:44, 3246.85KB/s]
  7%|▋         | 10547/155004 [00:10<00:42, 3398.81KB/s]
  7%|▋         | 11059/155004 [00:10<00:37, 3799.28KB/s]
  7%|▋         | 11444/155004 [00:10<00:40, 3581.27KB/s]
  8%|▊         | 11939/155004 [00:10<00:37, 3784.65KB/s]
  8%|▊         | 12499/155004 [00:10<00:33, 4205.35KB/s]
  8%|▊         | 12925/155004 [00:11<00:35, 3962.56KB/s]
  9%|▊         | 13459/155004 [00:11<00:34, 4159.68KB/s]
  9%|▉         | 14083/155004 [00:11<00:30, 4641.44KB/s]
  9%|▉         | 14553/155004 [00:11<00:32, 4357.31KB/s]
 10%|▉         | 15139/155004 [00:11<00:30, 4583.50KB/s]
 10%|█         | 15827/155004 [00:11<00:27, 5120.26KB/s]
 11%|█         | 16346/155004 [00:11<00:28, 4805.26KB/s]
 11%|█         | 16995/155004 [00:11<00:27, 5065.75KB/s]
 11%|█▏        | 17747/155004 [00:11<00:24, 5635.68KB/s]
 12%|█▏        | 18318/155004 [00:12<00:25, 5281.30KB/s]
 12%|█▏        | 19043/155004 [00:12<00:24, 5595.92KB/s]
 13%|█▎        | 19875/155004 [00:12<00:21, 6236.65KB/s]
 13%|█▎        | 20508/155004 [00:12<00:23, 5839.80KB/s]
 14%|█▎        | 21283/155004 [00:12<00:21, 6134.18KB/s]
 14%|█▍        | 22211/155004 [00:12<00:19, 6872.70KB/s]
 15%|█▍        | 22908/155004 [00:12<00:20, 6431.19KB/s]
 15%|█▌        | 23763/155004 [00:12<00:19, 6763.17KB/s]
 16%|█▌        | 24787/155004 [00:12<00:17, 7577.33KB/s]
 16%|█▋        | 25556/155004 [00:13<00:18, 7084.20KB/s]
 17%|█▋        | 26483/155004 [00:13<00:17, 7416.44KB/s]
 18%|█▊        | 27619/155004 [00:13<00:15, 8349.64KB/s]
 18%|█▊        | 28467/155004 [00:13<00:16, 7799.23KB/s]
 19%|█▉        | 29507/155004 [00:13<00:15, 8215.57KB/s]
 20%|█▉        | 30755/155004 [00:13<00:13, 9229.40KB/s]
 20%|██        | 31693/155004 [00:13<00:14, 8601.92KB/s]
 21%|██        | 32819/155004 [00:13<00:13, 8999.49KB/s]
 22%|██▏       | 34211/155004 [00:14<00:11, 10208.65KB/s]
 23%|██▎       | 35250/155004 [00:14<00:12, 9501.75KB/s] 
 24%|██▎       | 36467/155004 [00:14<00:11, 9883.78KB/s]
 25%|██▍       | 38003/155004 [00:14<00:10, 11231.08KB/s]
 25%|██▌       | 39147/155004 [00:14<00:11, 10449.04KB/s]
 26%|██▌       | 40499/155004 [00:14<00:10, 10910.48KB/s]
 27%|██▋       | 42195/155004 [00:14<00:09, 12399.91KB/s]
 28%|██▊       | 43459/155004 [00:14<00:09, 11519.24KB/s]
 29%|██▉       | 44947/155004 [00:14<00:09, 12041.36KB/s]
 30%|███       | 46819/155004 [00:15<00:07, 13694.15KB/s]
 31%|███       | 48215/155004 [00:15<00:08, 12687.05KB/s]
 32%|███▏      | 49843/155004 [00:15<00:07, 13261.68KB/s]
 33%|███▎      | 51907/155004 [00:15<00:06, 15096.93KB/s]
 34%|███▍      | 53448/155004 [00:15<00:07, 13946.86KB/s]
 36%|███▌      | 55235/155004 [00:15<00:06, 14613.84KB/s]
 37%|███▋      | 57523/155004 [00:15<00:05, 16651.58KB/s]
 38%|███▊      | 59223/155004 [00:15<00:06, 15382.62KB/s]
 39%|███▉      | 61187/155004 [00:15<00:05, 16115.38KB/s]
 41%|████      | 63715/155004 [00:16<00:04, 18399.41KB/s]
 42%|████▏     | 65595/155004 [00:16<00:05, 16949.47KB/s]
 44%|████▎     | 67747/155004 [00:16<00:04, 17789.64KB/s]
 46%|████▌     | 70547/155004 [00:16<00:04, 20319.03KB/s]
 47%|████▋     | 72623/155004 [00:16<00:04, 18647.23KB/s]
 48%|████▊     | 74979/155004 [00:16<00:04, 19604.10KB/s]
 50%|█████     | 78067/155004 [00:16<00:03, 22415.92KB/s]
 52%|█████▏    | 80360/155004 [00:16<00:03, 20441.90KB/s]
 54%|█████▎    | 82931/155004 [00:17<00:03, 21597.50KB/s]
 56%|█████▌    | 86136/155004 [00:17<00:02, 24436.89KB/s]
 57%|█████▋    | 88648/155004 [00:17<00:02, 22266.43KB/s]
 59%|█████▉    | 91699/155004 [00:17<00:02, 23962.75KB/s]
 61%|██████    | 94691/155004 [00:17<00:02, 25300.07KB/s]
 63%|██████▎   | 97278/155004 [00:17<00:02, 23795.24KB/s]
 65%|██████▌   | 100915/155004 [00:17<00:02, 26042.64KB/s]
 67%|██████▋   | 103651/155004 [00:17<00:01, 26108.72KB/s]
 69%|██████▊   | 106288/155004 [00:17<00:01, 24669.91KB/s]
 71%|███████   | 109763/155004 [00:18<00:01, 26030.33KB/s]
 73%|███████▎  | 113139/155004 [00:18<00:01, 27806.37KB/s]
 75%|███████▍  | 115945/155004 [00:18<00:01, 25348.40KB/s]
 77%|███████▋  | 118771/155004 [00:18<00:01, 25744.14KB/s]
 79%|███████▊  | 121955/155004 [00:18<00:01, 27120.02KB/s]
 80%|████████  | 124704/155004 [00:18<00:01, 24966.41KB/s]
 83%|████████▎ | 128195/155004 [00:18<00:00, 26964.73KB/s]
 85%|████████▍ | 131043/155004 [00:18<00:00, 27085.83KB/s]
 86%|████████▋ | 133785/155004 [00:18<00:00, 25508.48KB/s]
 89%|████████▊ | 137219/155004 [00:19<00:00, 26601.92KB/s]
 90%|█████████ | 140227/155004 [00:19<00:00, 27253.56KB/s]
 92%|█████████▏| 142974/155004 [00:19<00:00, 25218.18KB/s]
 94%|█████████▍| 146147/155004 [00:19<00:00, 26127.60KB/s]
 96%|█████████▌| 148924/155004 [00:19<00:00, 26570.06KB/s]
 98%|█████████▊| 151605/155004 [00:19<00:00, 24623.33KB/s]
100%|██████████| 155004/155004 [00:19<00:00, 7843.41KB/s] 
[05/19 14:32:07] ppdet.utils.checkpoint INFO: ['fc.bias', 'fc.weight', 'last_conv.weight'] in pretrained weight is not used in the model, and its will not be loaded
[05/19 14:32:07] ppdet.utils.checkpoint INFO: Finish loading model weights: /root/.cache/paddle/weights/PPHGNetV2_X_ssld_pretrained.pdparams
W0519 14:32:10.951179   420 reducer.cc:721] All parameters are involved in the backward pass. It is recommended to set find_unused_parameters to False to improve performance. However, if unused parameters appear in subsequent iterative training, then an error will occur. Please make it clear that in the subsequent training, there will be no parameters that are not used in the backward pass, and then set find_unused_parameters
[05/19 14:32:12] ppdet.engine INFO: Epoch: [0] [  0/539] learning_rate: 0.000000 loss_class: 0.067524 loss_bbox: 1.158027 loss_giou: 3.049131 loss_class_aux: 0.375933 loss_bbox_aux: 7.108191 loss_giou_aux: 18.298414 loss_class_dn: 1.033949 loss_bbox_dn: 0.121175 loss_giou_dn: 1.628482 loss_class_aux_dn: 5.157057 loss_bbox_aux_dn: 0.605875 loss_giou_aux_dn: 8.142411 loss: 46.746174 eta: 2 days, 7:54:20 batch_cost: 3.7340 data_cost: 0.0041 ips: 1.6069 images/s


--------------------------------------
C++ Traceback (most recent call last):
--------------------------------------
0   phi::backends::gpu::GpuMemcpySync(void*, void const*, unsigned long, cudaMemcpyKind)

----------------------
Error Message Summary:
----------------------
FatalError: `Termination signal` is detected by the operating system.
  [TimeInfo: *** Aborted at 1716129180 (unix time) try "date -d @1716129180" if you are using GNU date ***]
  [SignalInfo: *** SIGTERM (@0x196) received by PID 420 (TID 0x7cb66fc40740) from PID 406 ***]

Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics
Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics
usage: train.py [-h] [-c CONFIG] [-o [OPT ...]] [--eval] [-r RESUME]
                [--slim_config SLIM_CONFIG] [--enable_ce ENABLE_CE] [--amp]
                [--fleet] [--use_vdl USE_VDL] [--vdl_log_dir VDL_LOG_DIR]
                [--use_wandb USE_WANDB] [--save_prediction_only]
                [--profiler_options PROFILER_OPTIONS] [--save_proposals]
                [--proposals_path PROPOSALS_PATH] [--to_static]
train.py: error: argument --use_wandb: expected one argument
Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics
Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics
[2024-05-19 14:36:22,330] [    INFO] distributed_strategy.py:214 - distributed strategy initialized
======================= Modified FLAGS detected =======================
FLAGS(name='FLAGS_selected_gpus', current_value='0', default_value='')
=======================================================================
I0519 14:36:22.331928   627 tcp_utils.cc:181] The server starts to listen on IP_ANY:37848
I0519 14:36:22.332106   627 tcp_utils.cc:130] Successfully connected to 172.19.2.2:37848
I0519 14:36:25.450836   627 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000
[2024-05-19 14:36:25,451] [    INFO] topology.py:358 - Total 2 pipe comm group(s) create successfully!
W0519 14:36:25.451988   627 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 7.5, Driver API Version: 12.2, Runtime API Version: 11.8
W0519 14:36:25.453305   627 gpu_resources.cc:164] device: 0, cuDNN Version: 8.9.
I0519 14:36:25.579708   627 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000
[2024-05-19 14:36:25,579] [    INFO] topology.py:358 - Total 1 data comm group(s) create successfully!
[2024-05-19 14:36:25,579] [    INFO] topology.py:358 - Total 2 model comm group(s) create successfully!
[2024-05-19 14:36:25,580] [    INFO] topology.py:358 - Total 2 sharding comm group(s) create successfully!
I0519 14:36:25.580142   627 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000
[2024-05-19 14:36:25,580] [    INFO] topology.py:288 - HybridParallelInfo: rank_id: 0, mp_degree: 1, sharding_degree: 1, pp_degree: 1, dp_degree: 2, sep_degree: 1, mp_group: [0],  sharding_group: [0], pp_group: [0], dp_group: [0, 1], sep:group: None, check/clip group: [0]
loading annotations into memory...
Done (t=1.99s)
creating index...
index created!
[05/19 14:36:27] ppdet.data.source.coco WARNING: Found an invalid bbox in annotations: im_id: 201, area: 0.0 x1: 611, y1: 158, x2: 615, y2: 158.
[05/19 14:36:32] ppdet.data.source.coco INFO: Load [6471 samples valid, 0 samples invalid] in file datasets/VisDrone/annotations_VisDrone_train.json.
[05/19 14:36:38] ppdet.utils.checkpoint INFO: ['fc.bias', 'fc.weight', 'last_conv.weight'] in pretrained weight is not used in the model, and its will not be loaded
[05/19 14:36:38] ppdet.utils.checkpoint INFO: Finish loading model weights: /root/.cache/paddle/weights/PPHGNetV2_X_ssld_pretrained.pdparams
W0519 14:36:40.870151   627 reducer.cc:721] All parameters are involved in the backward pass. It is recommended to set find_unused_parameters to False to improve performance. However, if unused parameters appear in subsequent iterative training, then an error will occur. Please make it clear that in the subsequent training, there will be no parameters that are not used in the backward pass, and then set find_unused_parameters
[05/19 14:36:41] ppdet.engine INFO: Epoch: [0] [   0/1618] learning_rate: 0.000000 loss_class: 0.026570 loss_bbox: 0.467217 loss_giou: 1.591166 loss_class_aux: 0.163334 loss_bbox_aux: 2.759137 loss_giou_aux: 9.606020 loss_class_dn: 0.557981 loss_bbox_dn: 0.052587 loss_giou_dn: 1.066547 loss_class_aux_dn: 2.928646 loss_bbox_aux_dn: 0.262937 loss_giou_aux_dn: 5.332733 loss: 24.814877 eta: 3 days, 7:15:21 batch_cost: 1.7634 data_cost: 0.0007 ips: 1.1342 images/s


--------------------------------------
C++ Traceback (most recent call last):
--------------------------------------
0   paddle::pybind::eager_api_squeeze(_object*, _object*, _object*)
1   squeeze_ad_func(paddle::Tensor const&, paddle::experimental::IntArrayBase<paddle::Tensor>)
2   paddle::experimental::squeeze_intermediate(paddle::Tensor const&, paddle::experimental::IntArrayBase<paddle::Tensor> const&)
3   paddle::experimental::PrepareData(paddle::Tensor const&, phi::TensorArgDef const&, paddle::experimental::TransformFlag const&, bool)
4   paddle::experimental::TransformData(phi::DenseTensor const&, phi::TensorArgDef const&, paddle::experimental::TransformFlag const&, bool)
5   paddle::experimental::TransDataPlace(phi::DenseTensor const&, phi::Place)
6   void phi::Copy<phi::DeviceContext>(phi::DeviceContext const&, phi::DenseTensor const&, phi::Place, bool, phi::DenseTensor*)
7   phi::memory_utils::Copy(phi::Place const&, void*, phi::Place const&, void const*, unsigned long, void*)
8   phi::MemoryUtils::Copy(phi::Place const&, void*, phi::Place const&, void const*, unsigned long, void*)
9   void paddle::memory::Copy<phi::Place, phi::Place>(phi::Place, void*, phi::Place, void const*, unsigned long, void*)
10  void paddle::memory::Copy<phi::GPUPlace, phi::GPUPinnedPlace>(phi::GPUPlace, void*, phi::GPUPinnedPlace, void const*, unsigned long, void*)
11  phi::backends::gpu::GpuMemcpySync(void*, void const*, unsigned long, cudaMemcpyKind)

----------------------
Error Message Summary:
----------------------
FatalError: `Termination signal` is detected by the operating system.
  [TimeInfo: *** Aborted at 1716129481 (unix time) try "date -d @1716129481" if you are using GNU date ***]
  [SignalInfo: *** SIGTERM (@0x265) received by PID 627 (TID 0x7ee5ca109740) from PID 613 ***]

Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics
Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics
[2024-05-19 14:39:04,686] [    INFO] distributed_strategy.py:214 - distributed strategy initialized
======================= Modified FLAGS detected =======================
FLAGS(name='FLAGS_selected_gpus', current_value='0', default_value='')
=======================================================================
I0519 14:39:04.687780   796 tcp_utils.cc:181] The server starts to listen on IP_ANY:58013
I0519 14:39:04.687999   796 tcp_utils.cc:130] Successfully connected to 172.19.2.2:58013
I0519 14:39:04.861780   796 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000
[2024-05-19 14:39:04,862] [    INFO] topology.py:358 - Total 2 pipe comm group(s) create successfully!
W0519 14:39:04.863056   796 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 7.5, Driver API Version: 12.2, Runtime API Version: 11.8
W0519 14:39:04.864679   796 gpu_resources.cc:164] device: 0, cuDNN Version: 8.9.
I0519 14:39:05.022365   796 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000
[2024-05-19 14:39:05,022] [    INFO] topology.py:358 - Total 1 data comm group(s) create successfully!
[2024-05-19 14:39:05,022] [    INFO] topology.py:358 - Total 2 model comm group(s) create successfully!
[2024-05-19 14:39:05,022] [    INFO] topology.py:358 - Total 2 sharding comm group(s) create successfully!
I0519 14:39:05.022951   796 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000
[2024-05-19 14:39:05,023] [    INFO] topology.py:288 - HybridParallelInfo: rank_id: 0, mp_degree: 1, sharding_degree: 1, pp_degree: 1, dp_degree: 2, sep_degree: 1, mp_group: [0],  sharding_group: [0], pp_group: [0], dp_group: [0, 1], sep:group: None, check/clip group: [0]
loading annotations into memory...
Done (t=2.76s)
creating index...
index created!
[05/19 14:39:08] ppdet.data.source.coco WARNING: Found an invalid bbox in annotations: im_id: 201, area: 0.0 x1: 611, y1: 158, x2: 615, y2: 158.
[05/19 14:39:14] ppdet.data.source.coco INFO: Load [6471 samples valid, 0 samples invalid] in file datasets/VisDrone/annotations_VisDrone_train.json.
[05/19 14:39:20] ppdet.engine ERROR: wandb not found, please install wandb. Use: `pip install wandb`.
Traceback (most recent call last):
  File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 183, in <module>
    main()
  File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 179, in main
    run(FLAGS, cfg)
  File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 126, in run
    trainer = Trainer(cfg, mode='train')
  File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/trainer.py", line 150, in __init__
    self._init_callbacks()
  File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/trainer.py", line 162, in _init_callbacks
    self._callbacks.append(WandbCallback(self))
  File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/callbacks.py", line 323, in __init__
    raise e
  File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/callbacks.py", line 318, in __init__
    import wandb
  File "/opt/conda/lib/python3.10/site-packages/wandb/__init__.py", line 27, in <module>
    from wandb import sdk as wandb_sdk
  File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/__init__.py", line 25, in <module>
    from .artifacts.artifact import Artifact
  File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/artifacts/artifact.py", line 46, in <module>
    from wandb.apis.normalize import normalize_exceptions
  File "/opt/conda/lib/python3.10/site-packages/wandb/apis/__init__.py", line 43, in <module>
    from .internal import Api as InternalApi  # noqa
  File "/opt/conda/lib/python3.10/site-packages/wandb/apis/internal.py", line 3, in <module>
    from wandb.sdk.internal.internal_api import Api as InternalApi
  File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/internal/internal_api.py", line 48, in <module>
    from ..lib import retry
  File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/lib/retry.py", line 17, in <module>
    from .mailbox import ContextCancelledError
  File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/lib/mailbox.py", line 102, in <module>
    class _MailboxSlot:
  File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/lib/mailbox.py", line 103, in _MailboxSlot
    _result: Optional[pb.Result]
AttributeError: module 'wandb.proto.wandb_internal_pb2' has no attribute 'Result'


--------------------------------------
C++ Traceback (most recent call last):
--------------------------------------
No stack trace in paddle, may be caused by external reasons.

----------------------
Error Message Summary:
----------------------
FatalError: `Termination signal` is detected by the operating system.
  [TimeInfo: *** Aborted at 1716129561 (unix time) try "date -d @1716129561" if you are using GNU date ***]
  [SignalInfo: *** SIGTERM (@0x30e) received by PID 796 (TID 0x7f5dae22e740) from PID 782 ***]

Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics
Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics
[2024-05-19 14:47:12,876] [    INFO] distributed_strategy.py:214 - distributed strategy initialized
======================= Modified FLAGS detected =======================
FLAGS(name='FLAGS_selected_gpus', current_value='0', default_value='')
=======================================================================
I0519 14:47:12.877843   177 tcp_utils.cc:181] The server starts to listen on IP_ANY:41929
I0519 14:47:12.877998   177 tcp_utils.cc:130] Successfully connected to 172.19.2.2:41929
I0519 14:47:15.903615   177 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000
[2024-05-19 14:47:15,904] [    INFO] topology.py:358 - Total 2 pipe comm group(s) create successfully!
W0519 14:47:15.904691   177 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 7.5, Driver API Version: 12.2, Runtime API Version: 11.8
W0519 14:47:15.905977   177 gpu_resources.cc:164] device: 0, cuDNN Version: 8.9.
I0519 14:47:16.034154   177 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000
[2024-05-19 14:47:16,034] [    INFO] topology.py:358 - Total 1 data comm group(s) create successfully!
[2024-05-19 14:47:16,034] [    INFO] topology.py:358 - Total 2 model comm group(s) create successfully!
[2024-05-19 14:47:16,034] [    INFO] topology.py:358 - Total 2 sharding comm group(s) create successfully!
I0519 14:47:16.034626   177 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000
[2024-05-19 14:47:16,034] [    INFO] topology.py:288 - HybridParallelInfo: rank_id: 0, mp_degree: 1, sharding_degree: 1, pp_degree: 1, dp_degree: 2, sep_degree: 1, mp_group: [0],  sharding_group: [0], pp_group: [0], dp_group: [0, 1], sep:group: None, check/clip group: [0]
loading annotations into memory...
Done (t=1.95s)
creating index...
index created!
[05/19 14:47:18] ppdet.data.source.coco WARNING: Found an invalid bbox in annotations: im_id: 201, area: 0.0 x1: 611, y1: 158, x2: 615, y2: 158.
[05/19 14:47:22] ppdet.data.source.coco INFO: Load [6471 samples valid, 0 samples invalid] in file datasets/VisDrone/annotations_VisDrone_train.json.
wandb: Currently logged in as: thanhtuit96 (thanhtuit). Use `wandb login --relogin` to force relogin
wandb: wandb version 0.17.0 is available!  To upgrade, please run:
wandb:  $ pip install wandb --upgrade
wandb: Tracking run with wandb version 0.16.6
wandb: Run data is saved locally in /kaggle/working/ObjectDetection/DETR/wandb/run-20240519_144730-7on4vywi
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run silvery-planet-1
wandb: ⭐️ View project at https://wandb.ai/thanhtuit/ObjectDetection-DETR_tools
wandb: 🚀 View run at https://wandb.ai/thanhtuit/ObjectDetection-DETR_tools/runs/7on4vywi
[05/19 14:47:46] ppdet.utils.download INFO: Downloading PPHGNetV2_X_ssld_pretrained.pdparams from https://bj.bcebos.com/v1/paddledet/models/pretrained/PPHGNetV2_X_ssld_pretrained.pdparams

  0%|          | 0/155004 [00:00<?, ?KB/s]
  0%|          | 3/155004 [00:00<3:53:33, 11.06KB/s]
  0%|          | 19/155004 [00:00<1:06:04, 39.10KB/s]
  0%|          | 35/155004 [00:00<53:47, 48.01KB/s]  
  0%|          | 51/155004 [00:01<49:29, 52.18KB/s]
  0%|          | 67/155004 [00:01<47:23, 54.50KB/s]
  0%|          | 83/155004 [00:01<46:11, 55.89KB/s]
  0%|          | 99/155004 [00:01<45:28, 56.78KB/s]
  0%|          | 115/155004 [00:02<38:18, 67.38KB/s]
  0%|          | 131/155004 [00:02<32:41, 78.95KB/s]
  0%|          | 147/155004 [00:02<36:14, 71.23KB/s]
  0%|          | 163/155004 [00:02<38:38, 66.80KB/s]
  0%|          | 195/155004 [00:03<30:50, 83.65KB/s]
  0%|          | 227/155004 [00:03<27:17, 94.50KB/s]
  0%|          | 259/155004 [00:03<25:22, 101.65KB/s]
  0%|          | 291/155004 [00:03<24:12, 106.48KB/s]
  0%|          | 339/155004 [00:04<20:11, 127.67KB/s]
  0%|          | 387/155004 [00:04<15:29, 166.29KB/s]
  0%|          | 406/155004 [00:04<15:51, 162.44KB/s]
  0%|          | 451/155004 [00:04<15:44, 163.70KB/s]
  0%|          | 515/155004 [00:04<11:27, 224.57KB/s]
  0%|          | 541/155004 [00:04<11:43, 219.60KB/s]
  0%|          | 595/155004 [00:05<12:10, 211.27KB/s]
  0%|          | 691/155004 [00:05<09:48, 262.14KB/s]
  1%|          | 787/155004 [00:05<08:46, 292.65KB/s]
  1%|          | 915/155004 [00:06<07:19, 350.39KB/s]
  1%|          | 1043/155004 [00:06<06:36, 388.16KB/s]
  1%|          | 1187/155004 [00:06<05:56, 431.79KB/s]
  1%|          | 1363/155004 [00:06<05:08, 497.62KB/s]
  1%|          | 1539/155004 [00:07<04:42, 542.70KB/s]
  1%|          | 1747/155004 [00:07<03:35, 710.48KB/s]
  1%|          | 1828/155004 [00:07<03:40, 693.35KB/s]
  1%|▏         | 1987/155004 [00:07<03:53, 654.77KB/s]
  1%|▏         | 2243/155004 [00:07<03:22, 752.58KB/s]
  2%|▏         | 2531/155004 [00:08<02:58, 852.63KB/s]
  2%|▏         | 2851/155004 [00:08<02:39, 955.07KB/s]
  2%|▏         | 3187/155004 [00:08<02:25, 1041.88KB/s]
  2%|▏         | 3571/155004 [00:09<02:11, 1154.81KB/s]
  3%|▎         | 3987/155004 [00:09<01:59, 1268.13KB/s]
  3%|▎         | 4435/155004 [00:09<01:33, 1609.98KB/s]
  3%|▎         | 4616/155004 [00:09<01:36, 1564.92KB/s]
  3%|▎         | 4931/155004 [00:09<01:45, 1421.19KB/s]
  4%|▎         | 5475/155004 [00:10<01:32, 1618.87KB/s]
  4%|▍         | 6051/155004 [00:10<01:23, 1781.33KB/s]
  4%|▍         | 6691/155004 [00:10<01:15, 1961.95KB/s]
  5%|▍         | 7363/155004 [00:10<00:59, 2481.11KB/s]
  5%|▍         | 7639/155004 [00:10<01:01, 2403.46KB/s]
  5%|▌         | 8099/155004 [00:11<00:56, 2605.03KB/s]
  5%|▌         | 8376/155004 [00:11<00:59, 2483.62KB/s]
  6%|▌         | 8899/155004 [00:11<01:04, 2263.90KB/s]
  6%|▋         | 9747/155004 [00:11<00:56, 2578.85KB/s]
  7%|▋         | 10659/155004 [00:12<00:50, 2843.63KB/s]
  8%|▊         | 11635/155004 [00:12<00:46, 3086.33KB/s]
  8%|▊         | 12675/155004 [00:12<00:42, 3318.70KB/s]
  9%|▉         | 13795/155004 [00:12<00:39, 3565.03KB/s]
 10%|▉         | 14979/155004 [00:12<00:31, 4447.62KB/s]
 10%|▉         | 15470/155004 [00:13<00:32, 4272.56KB/s]
 10%|█         | 16259/155004 [00:13<00:36, 3808.52KB/s]
 11%|█▏        | 17619/155004 [00:13<00:32, 4215.71KB/s]
 12%|█▏        | 19043/155004 [00:13<00:25, 5381.58KB/s]
 13%|█▎        | 19640/155004 [00:13<00:26, 5142.99KB/s]
 13%|█▎        | 20579/155004 [00:14<00:29, 4557.54KB/s]
 14%|█▍        | 22195/155004 [00:14<00:26, 5044.15KB/s]
 15%|█▌        | 23907/155004 [00:14<00:24, 5456.92KB/s]
 17%|█▋        | 25731/155004 [00:15<00:22, 5854.62KB/s]
 18%|█▊        | 27651/155004 [00:15<00:20, 6228.39KB/s]
 19%|█▉        | 29667/155004 [00:15<00:16, 7718.23KB/s]
 20%|█▉        | 30520/155004 [00:15<00:16, 7337.93KB/s]
 21%|██        | 31811/155004 [00:15<00:18, 6508.86KB/s]
 22%|██▏       | 34067/155004 [00:15<00:14, 8521.06KB/s]
 23%|██▎       | 35019/155004 [00:16<00:14, 8054.73KB/s]
 24%|██▎       | 36451/155004 [00:16<00:16, 7119.55KB/s]
 25%|██▌       | 38947/155004 [00:16<00:12, 9481.24KB/s]
 26%|██▌       | 40004/155004 [00:16<00:12, 8910.33KB/s]
 27%|██▋       | 41427/155004 [00:16<00:12, 9463.99KB/s]
 27%|██▋       | 42434/155004 [00:16<00:12, 8782.84KB/s]
 28%|██▊       | 43971/155004 [00:17<00:11, 9599.55KB/s]
 29%|██▉       | 44968/155004 [00:17<00:12, 8891.99KB/s]
 30%|███       | 46563/155004 [00:17<00:11, 9756.16KB/s]
 31%|███       | 47560/155004 [00:17<00:11, 9079.88KB/s]
 32%|███▏      | 49251/155004 [00:17<00:10, 10009.45KB/s]
 32%|███▏      | 50323/155004 [00:17<00:10, 9527.08KB/s] 
 34%|███▎      | 52083/155004 [00:17<00:09, 10364.61KB/s]
 35%|███▍      | 53523/155004 [00:18<00:09, 10627.41KB/s]
 36%|███▌      | 55059/155004 [00:18<00:09, 10603.28KB/s]
 36%|███▋      | 56547/155004 [00:18<00:09, 10906.87KB/s]
 38%|███▊      | 58243/155004 [00:18<00:08, 11144.78KB/s]
 38%|███▊      | 59635/155004 [00:18<00:08, 11074.93KB/s]
 40%|███▉      | 61299/155004 [00:18<00:08, 11191.20KB/s]
 40%|████      | 62675/155004 [00:18<00:08, 11076.91KB/s]
 42%|████▏     | 64451/155004 [00:18<00:07, 11436.07KB/s]
 42%|████▏     | 65747/155004 [00:19<00:08, 11075.79KB/s]
 44%|████▎     | 67427/155004 [00:19<00:07, 11224.53KB/s]
 44%|████▍     | 68867/155004 [00:19<00:07, 11243.62KB/s]
 46%|████▌     | 70611/155004 [00:19<00:07, 11471.77KB/s]
 46%|████▋     | 71907/155004 [00:19<00:07, 11105.80KB/s]
 48%|████▊     | 73715/155004 [00:19<00:07, 11514.27KB/s]
 48%|████▊     | 75027/155004 [00:19<00:07, 11176.59KB/s]
 49%|████▉     | 76643/155004 [00:20<00:07, 11142.23KB/s]
 50%|█████     | 78131/155004 [00:20<00:06, 11296.95KB/s]
 51%|█████▏    | 79747/155004 [00:20<00:06, 11233.49KB/s]
 52%|█████▏    | 81107/155004 [00:20<00:06, 11077.84KB/s]
 53%|█████▎    | 82819/155004 [00:20<00:06, 11290.46KB/s]
 54%|█████▍    | 84163/155004 [00:20<00:06, 11077.77KB/s]
 55%|█████▌    | 85795/155004 [00:20<00:06, 11119.61KB/s]
 56%|█████▋    | 87251/155004 [00:21<00:06, 11203.92KB/s]
 57%|█████▋    | 88819/155004 [00:21<00:05, 11070.74KB/s]
 58%|█████▊    | 90259/155004 [00:21<00:05, 11134.70KB/s]
 59%|█████▉    | 91907/155004 [00:21<00:05, 11192.90KB/s]
 60%|██████    | 93315/155004 [00:21<00:05, 11149.40KB/s]
 61%|██████    | 94931/155004 [00:21<00:05, 11133.73KB/s]
 62%|██████▏   | 96355/155004 [00:21<00:05, 11140.01KB/s]
 63%|██████▎   | 97827/155004 [00:21<00:05, 10821.26KB/s]
 64%|██████▍   | 99379/155004 [00:22<00:04, 11202.57KB/s]
 65%|██████▌   | 100947/155004 [00:22<00:04, 11069.48KB/s]
 66%|██████▌   | 102323/155004 [00:22<00:04, 10989.83KB/s]
 67%|██████▋   | 104067/155004 [00:22<00:04, 11304.69KB/s]
 68%|██████▊   | 105379/155004 [00:22<00:04, 11016.56KB/s]
 69%|██████▉   | 107043/155004 [00:22<00:04, 11148.05KB/s]
 70%|██████▉   | 108483/155004 [00:22<00:04, 11190.53KB/s]
 71%|███████   | 110067/155004 [00:23<00:04, 11092.27KB/s]
 72%|███████▏  | 111507/155004 [00:23<00:03, 11147.21KB/s]
 73%|███████▎  | 113139/155004 [00:23<00:03, 11165.15KB/s]
 74%|███████▍  | 114531/155004 [00:23<00:03, 11093.73KB/s]
 75%|███████▍  | 116211/155004 [00:23<00:03, 11235.95KB/s]
 76%|███████▌  | 117587/155004 [00:23<00:03, 11109.36KB/s]
 77%|███████▋  | 119075/155004 [00:23<00:03, 10828.94KB/s]
 78%|███████▊  | 120659/155004 [00:24<00:03, 11277.61KB/s]
 79%|███████▉  | 122259/155004 [00:24<00:02, 11190.63KB/s]
 80%|███████▉  | 123587/155004 [00:24<00:02, 10964.99KB/s]
 81%|████████  | 125171/155004 [00:24<00:02, 10940.09KB/s]
 82%|████████▏ | 126675/155004 [00:24<00:02, 11182.30KB/s]
 83%|████████▎ | 128275/155004 [00:24<00:02, 11120.34KB/s]
 84%|████████▎ | 129651/155004 [00:24<00:02, 11029.41KB/s]
 85%|████████▍ | 131203/155004 [00:24<00:02, 10905.51KB/s]
 86%|████████▌ | 132723/155004 [00:25<00:01, 11200.08KB/s]
 87%|████████▋ | 134323/155004 [00:25<00:01, 11129.25KB/s]
 88%|████████▊ | 135699/155004 [00:25<00:01, 11036.68KB/s]
 89%|████████▊ | 137235/155004 [00:25<00:01, 10881.33KB/s]
 90%|████████▉ | 138787/155004 [00:25<00:01, 11249.22KB/s]
 91%|█████████ | 140387/155004 [00:25<00:01, 11163.67KB/s]
 91%|█████████▏| 141699/155004 [00:25<00:01, 10920.22KB/s]
 93%|█████████▎| 143443/155004 [00:26<00:01, 11251.51KB/s]
 94%|█████████▎| 144931/155004 [00:26<00:00, 11374.19KB/s]
 94%|█████████▍| 146387/155004 [00:26<00:00, 10938.20KB/s]
 96%|█████████▌| 148067/155004 [00:26<00:00, 11575.25KB/s]
 96%|█████████▋| 149379/155004 [00:26<00:00, 10764.93KB/s]
 97%|█████████▋| 151091/155004 [00:26<00:00, 11527.60KB/s]
 98%|█████████▊| 152643/155004 [00:26<00:00, 11248.80KB/s]
 99%|█████████▉| 154051/155004 [00:27<00:00, 11197.84KB/s]
100%|██████████| 155004/155004 [00:27<00:00, 5730.75KB/s] 
[05/19 14:48:16] ppdet.utils.checkpoint INFO: ['fc.bias', 'fc.weight', 'last_conv.weight'] in pretrained weight is not used in the model, and its will not be loaded
[05/19 14:48:17] ppdet.utils.checkpoint INFO: Finish loading model weights: /root/.cache/paddle/weights/PPHGNetV2_X_ssld_pretrained.pdparams
W0519 14:48:20.122522   177 reducer.cc:721] All parameters are involved in the backward pass. It is recommended to set find_unused_parameters to False to improve performance. However, if unused parameters appear in subsequent iterative training, then an error will occur. Please make it clear that in the subsequent training, there will be no parameters that are not used in the backward pass, and then set find_unused_parameters
[05/19 14:48:21] ppdet.engine INFO: Epoch: [0] [  0/809] learning_rate: 0.000000 loss_class: 0.109804 loss_bbox: 0.697411 loss_giou: 2.439157 loss_class_aux: 0.583214 loss_bbox_aux: 4.228478 loss_giou_aux: 14.723276 loss_class_dn: 0.873400 loss_bbox_dn: 0.086687 loss_giou_dn: 1.583300 loss_class_aux_dn: 4.199629 loss_bbox_aux_dn: 0.433435 loss_giou_aux_dn: 7.916497 loss: 37.874287 eta: 3 days, 2:02:12 batch_cost: 3.2946 data_cost: 0.0005 ips: 1.2141 images/s
[05/19 14:53:36] ppdet.engine INFO: Epoch: [0] [200/809] learning_rate: 0.000005 loss_class: 0.115475 loss_bbox: 0.590497 loss_giou: 2.012227 loss_class_aux: 0.717756 loss_bbox_aux: 3.572585 loss_giou_aux: 12.202559 loss_class_dn: 0.650731 loss_bbox_dn: 0.123633 loss_giou_dn: 1.404296 loss_class_aux_dn: 3.379308 loss_bbox_aux_dn: 0.618320 loss_giou_aux_dn: 7.013911 loss: 32.793163 eta: 1 day, 10:03:56 batch_cost: 1.5108 data_cost: 0.0021 ips: 2.6476 images/s
[05/19 14:59:00] ppdet.engine INFO: Epoch: [0] [400/809] learning_rate: 0.000010 loss_class: 0.146548 loss_bbox: 0.500630 loss_giou: 1.904366 loss_class_aux: 0.854553 loss_bbox_aux: 3.079211 loss_giou_aux: 11.489814 loss_class_dn: 0.431924 loss_bbox_dn: 0.125420 loss_giou_dn: 1.394954 loss_class_aux_dn: 2.272492 loss_bbox_aux_dn: 0.632797 loss_giou_aux_dn: 6.903274 loss: 29.958066 eta: 1 day, 10:20:40 batch_cost: 1.5522 data_cost: 0.0026 ips: 2.5769 images/s
[05/19 15:04:19] ppdet.engine INFO: Epoch: [0] [600/809] learning_rate: 0.000015 loss_class: 0.183138 loss_bbox: 0.387057 loss_giou: 1.810114 loss_class_aux: 1.021730 loss_bbox_aux: 2.410990 loss_giou_aux: 10.909318 loss_class_dn: 0.391584 loss_bbox_dn: 0.111994 loss_giou_dn: 1.374636 loss_class_aux_dn: 1.907480 loss_bbox_aux_dn: 0.567154 loss_giou_aux_dn: 6.864466 loss: 28.483194 eta: 1 day, 10:11:01 batch_cost: 1.5257 data_cost: 0.0024 ips: 2.6217 images/s
[05/19 15:09:39] ppdet.engine INFO: Epoch: [0] [800/809] learning_rate: 0.000020 loss_class: 0.242067 loss_bbox: 0.331256 loss_giou: 1.752203 loss_class_aux: 1.358915 loss_bbox_aux: 2.035192 loss_giou_aux: 10.566290 loss_class_dn: 0.396643 loss_bbox_dn: 0.117754 loss_giou_dn: 1.427633 loss_class_aux_dn: 1.877571 loss_bbox_aux_dn: 0.592171 loss_giou_aux_dn: 7.151309 loss: 28.509894 eta: 1 day, 10:07:25 batch_cost: 1.5371 data_cost: 0.0023 ips: 2.6024 images/s
[05/19 15:09:55] ppdet.engine INFO: Epoch: [1] [  0/809] learning_rate: 0.000020 loss_class: 0.248910 loss_bbox: 0.327536 loss_giou: 1.748124 loss_class_aux: 1.434996 loss_bbox_aux: 1.983233 loss_giou_aux: 10.526444 loss_class_dn: 0.392683 loss_bbox_dn: 0.118762 loss_giou_dn: 1.427633 loss_class_aux_dn: 1.867389 loss_bbox_aux_dn: 0.594845 loss_giou_aux_dn: 7.151309 loss: 28.442179 eta: 1 day, 10:09:46 batch_cost: 1.5480 data_cost: 0.0170 ips: 2.5840 images/s
[05/19 15:15:20] ppdet.engine INFO: Epoch: [1] [200/809] learning_rate: 0.000025 loss_class: 0.325866 loss_bbox: 0.261831 loss_giou: 1.503312 loss_class_aux: 1.795064 loss_bbox_aux: 1.646755 loss_giou_aux: 9.197675 loss_class_dn: 0.405848 loss_bbox_dn: 0.116908 loss_giou_dn: 1.312363 loss_class_aux_dn: 1.899924 loss_bbox_aux_dn: 0.587281 loss_giou_aux_dn: 6.639198 loss: 26.055919 eta: 1 day, 10:09:46 batch_cost: 1.5550 data_cost: 0.0031 ips: 2.5723 images/s
[05/19 15:20:43] ppdet.engine INFO: Epoch: [1] [400/809] learning_rate: 0.000030 loss_class: 0.362999 loss_bbox: 0.262074 loss_giou: 1.574336 loss_class_aux: 2.030229 loss_bbox_aux: 1.632570 loss_giou_aux: 9.598309 loss_class_dn: 0.430456 loss_bbox_dn: 0.106288 loss_giou_dn: 1.295524 loss_class_aux_dn: 2.005164 loss_bbox_aux_dn: 0.540839 loss_giou_aux_dn: 6.537989 loss: 26.593936 eta: 1 day, 10:05:28 batch_cost: 1.5432 data_cost: 0.0030 ips: 2.5920 images/s
[05/19 15:26:05] ppdet.engine INFO: Epoch: [1] [600/809] learning_rate: 0.000035 loss_class: 0.413674 loss_bbox: 0.211832 loss_giou: 1.459727 loss_class_aux: 2.280157 loss_bbox_aux: 1.333013 loss_giou_aux: 8.954050 loss_class_dn: 0.430417 loss_bbox_dn: 0.092424 loss_giou_dn: 1.206493 loss_class_aux_dn: 2.017116 loss_bbox_aux_dn: 0.468860 loss_giou_aux_dn: 6.090151 loss: 25.308434 eta: 1 day, 10:00:33 batch_cost: 1.5412 data_cost: 0.0028 ips: 2.5953 images/s
[05/19 15:31:23] ppdet.engine INFO: Epoch: [1] [800/809] learning_rate: 0.000040 loss_class: 0.436514 loss_bbox: 0.195315 loss_giou: 1.378583 loss_class_aux: 2.423530 loss_bbox_aux: 1.234668 loss_giou_aux: 8.488070 loss_class_dn: 0.449268 loss_bbox_dn: 0.097788 loss_giou_dn: 1.121646 loss_class_aux_dn: 2.107834 loss_bbox_aux_dn: 0.496053 loss_giou_aux_dn: 5.793622 loss: 24.938890 eta: 1 day, 9:52:07 batch_cost: 1.5201 data_cost: 0.0033 ips: 2.6314 images/s
[05/19 15:31:45] ppdet.utils.checkpoint INFO: Save checkpoint: output/rtdetr_hgnetv2_x_6x_coco
loading annotations into memory...
Done (t=0.21s)
creating index...
index created!
[05/19 15:31:47] ppdet.data.source.coco INFO: Load [548 samples valid, 0 samples invalid] in file datasets/VisDrone/annotations_VisDrone_val.json.
[05/19 15:31:50] ppdet.engine INFO: Eval iter: 0
[05/19 15:32:52] ppdet.metrics.metrics INFO: The bbox result is saved to bbox.json.
loading annotations into memory...
Done (t=0.14s)
creating index...
index created!
[05/19 15:32:52] ppdet.metrics.coco_utils INFO: Start evaluate...
Loading and preparing results...
DONE (t=1.88s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=83.86s).
Accumulating evaluation results...
DONE (t=2.57s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.017
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.040
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.012
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.012
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.025
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.027
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.018
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.047
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.095
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.057
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.169
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.132
[05/19 15:34:21] ppdet.engine INFO: Total sample number: 548, average FPS: 9.180101473237684
[05/19 15:34:21] ppdet.engine INFO: Best test bbox ap is 0.017.
[05/19 15:34:24] ppdet.utils.checkpoint INFO: Save checkpoint: output/rtdetr_hgnetv2_x_6x_coco
[05/19 15:34:27] ppdet.engine INFO: Epoch: [2] [  0/809] learning_rate: 0.000040 loss_class: 0.436514 loss_bbox: 0.190228 loss_giou: 1.355289 loss_class_aux: 2.423530 loss_bbox_aux: 1.198317 loss_giou_aux: 8.289369 loss_class_dn: 0.446934 loss_bbox_dn: 0.095412 loss_giou_dn: 1.105881 loss_class_aux_dn: 2.087826 loss_bbox_aux_dn: 0.487604 loss_giou_aux_dn: 5.698340 loss: 24.563335 eta: 1 day, 9:50:52 batch_cost: 1.5107 data_cost: 0.0033 ips: 2.6478 images/s
[05/19 15:39:46] ppdet.engine INFO: Epoch: [2] [200/809] learning_rate: 0.000045 loss_class: 0.464409 loss_bbox: 0.185540 loss_giou: 1.370261 loss_class_aux: 2.655948 loss_bbox_aux: 1.131737 loss_giou_aux: 8.399027 loss_class_dn: 0.457022 loss_bbox_dn: 0.090482 loss_giou_dn: 1.139166 loss_class_aux_dn: 2.128933 loss_bbox_aux_dn: 0.460157 loss_giou_aux_dn: 5.741822 loss: 24.596806 eta: 1 day, 9:44:26 batch_cost: 1.5279 data_cost: 0.0032 ips: 2.6180 images/s
[05/19 15:45:04] ppdet.engine INFO: Epoch: [2] [400/809] learning_rate: 0.000050 loss_class: 0.446663 loss_bbox: 0.171631 loss_giou: 1.300038 loss_class_aux: 2.541440 loss_bbox_aux: 1.066781 loss_giou_aux: 7.996847 loss_class_dn: 0.429509 loss_bbox_dn: 0.085551 loss_giou_dn: 1.064214 loss_class_aux_dn: 2.030793 loss_bbox_aux_dn: 0.442241 loss_giou_aux_dn: 5.426585 loss: 23.793605 eta: 1 day, 9:37:41 batch_cost: 1.5235 data_cost: 0.0031 ips: 2.6256 images/s
[05/19 15:50:16] ppdet.engine INFO: Epoch: [2] [600/809] learning_rate: 0.000050 loss_class: 0.424061 loss_bbox: 0.181424 loss_giou: 1.369216 loss_class_aux: 2.372216 loss_bbox_aux: 1.124080 loss_giou_aux: 8.357028 loss_class_dn: 0.426770 loss_bbox_dn: 0.082361 loss_giou_dn: 1.057085 loss_class_aux_dn: 2.049204 loss_bbox_aux_dn: 0.425031 loss_giou_aux_dn: 5.420174 loss: 23.418367 eta: 1 day, 9:27:38 batch_cost: 1.4929 data_cost: 0.0033 ips: 2.6793 images/s
[05/19 15:55:31] ppdet.engine INFO: Epoch: [2] [800/809] learning_rate: 0.000050 loss_class: 0.445304 loss_bbox: 0.176066 loss_giou: 1.338526 loss_class_aux: 2.551275 loss_bbox_aux: 1.106353 loss_giou_aux: 8.285362 loss_class_dn: 0.432885 loss_bbox_dn: 0.083495 loss_giou_dn: 1.084231 loss_class_aux_dn: 2.087136 loss_bbox_aux_dn: 0.435047 loss_giou_aux_dn: 5.537897 loss: 24.292417 eta: 1 day, 9:20:08 batch_cost: 1.5088 data_cost: 0.0033 ips: 2.6511 images/s
[05/19 15:55:51] ppdet.engine INFO: Epoch: [3] [  0/809] learning_rate: 0.000050 loss_class: 0.449243 loss_bbox: 0.167929 loss_giou: 1.275381 loss_class_aux: 2.562489 loss_bbox_aux: 1.061722 loss_giou_aux: 7.823178 loss_class_dn: 0.435366 loss_bbox_dn: 0.083069 loss_giou_dn: 1.069175 loss_class_aux_dn: 2.081756 loss_bbox_aux_dn: 0.426329 loss_giou_aux_dn: 5.435639 loss: 23.467019 eta: 1 day, 9:22:50 batch_cost: 1.5387 data_cost: 0.0122 ips: 2.5995 images/s
[05/19 16:01:10] ppdet.engine INFO: Epoch: [3] [200/809] learning_rate: 0.000050 loss_class: 0.452845 loss_bbox: 0.172303 loss_giou: 1.308010 loss_class_aux: 2.620508 loss_bbox_aux: 1.070214 loss_giou_aux: 7.959267 loss_class_dn: 0.441857 loss_bbox_dn: 0.084600 loss_giou_dn: 1.045505 loss_class_aux_dn: 2.157980 loss_bbox_aux_dn: 0.444289 loss_giou_aux_dn: 5.335405 loss: 23.397141 eta: 1 day, 9:17:13 batch_cost: 1.5262 data_cost: 0.0029 ips: 2.6209 images/s
[05/19 16:06:26] ppdet.engine INFO: Epoch: [3] [400/809] learning_rate: 0.000050 loss_class: 0.431835 loss_bbox: 0.161333 loss_giou: 1.270525 loss_class_aux: 2.526102 loss_bbox_aux: 0.993447 loss_giou_aux: 7.729719 loss_class_dn: 0.411009 loss_bbox_dn: 0.076937 loss_giou_dn: 0.983667 loss_class_aux_dn: 2.022133 loss_bbox_aux_dn: 0.399821 loss_giou_aux_dn: 5.016749 loss: 21.984449 eta: 1 day, 9:10:35 batch_cost: 1.5145 data_cost: 0.0033 ips: 2.6412 images/s
[05/19 16:11:47] ppdet.engine INFO: Epoch: [3] [600/809] learning_rate: 0.000050 loss_class: 0.412699 loss_bbox: 0.167048 loss_giou: 1.332672 loss_class_aux: 2.418309 loss_bbox_aux: 1.020018 loss_giou_aux: 8.081558 loss_class_dn: 0.439995 loss_bbox_dn: 0.073425 loss_giou_dn: 1.049441 loss_class_aux_dn: 2.150877 loss_bbox_aux_dn: 0.380578 loss_giou_aux_dn: 5.389125 loss: 23.421144 eta: 1 day, 9:05:58 batch_cost: 1.5353 data_cost: 0.0035 ips: 2.6053 images/s
[05/19 16:17:09] ppdet.engine INFO: Epoch: [3] [800/809] learning_rate: 0.000050 loss_class: 0.425857 loss_bbox: 0.166994 loss_giou: 1.324059 loss_class_aux: 2.475139 loss_bbox_aux: 1.052269 loss_giou_aux: 8.147684 loss_class_dn: 0.442590 loss_bbox_dn: 0.073975 loss_giou_dn: 1.038611 loss_class_aux_dn: 2.196448 loss_bbox_aux_dn: 0.390584 loss_giou_aux_dn: 5.306982 loss: 24.024439 eta: 1 day, 9:02:07 batch_cost: 1.5458 data_cost: 0.0026 ips: 2.5877 images/s
[05/19 16:17:33] ppdet.utils.checkpoint INFO: Save checkpoint: output/rtdetr_hgnetv2_x_6x_coco
[05/19 16:17:35] ppdet.engine INFO: Eval iter: 0
[05/19 16:18:36] ppdet.metrics.metrics INFO: The bbox result is saved to bbox.json.
loading annotations into memory...
Done (t=0.36s)
creating index...
index created!
[05/19 16:18:37] ppdet.metrics.coco_utils INFO: Start evaluate...
Loading and preparing results...
DONE (t=1.68s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=86.92s).
Accumulating evaluation results...
DONE (t=2.91s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.079
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.151
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.074
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.044
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.119
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.182
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.055
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.153
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.220
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.142
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.334
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.368
[05/19 16:20:09] ppdet.engine INFO: Total sample number: 548, average FPS: 9.712129594793842
[05/19 16:20:09] ppdet.engine INFO: Best test bbox ap is 0.079.
[05/19 16:20:14] ppdet.utils.checkpoint INFO: Save checkpoint: output/rtdetr_hgnetv2_x_6x_coco
[05/19 16:20:17] ppdet.engine INFO: Epoch: [4] [  0/809] learning_rate: 0.000050 loss_class: 0.434871 loss_bbox: 0.164768 loss_giou: 1.302444 loss_class_aux: 2.546712 loss_bbox_aux: 1.026963 loss_giou_aux: 7.971454 loss_class_dn: 0.436979 loss_bbox_dn: 0.073928 loss_giou_dn: 1.015813 loss_class_aux_dn: 2.161131 loss_bbox_aux_dn: 0.388692 loss_giou_aux_dn: 5.209520 loss: 23.531901 eta: 1 day, 9:01:40 batch_cost: 1.5406 data_cost: 0.0026 ips: 2.5963 images/s
[05/19 16:25:35] ppdet.engine INFO: Epoch: [4] [200/809] learning_rate: 0.000050 loss_class: 0.407466 loss_bbox: 0.161341 loss_giou: 1.234918 loss_class_aux: 2.376428 loss_bbox_aux: 1.000166 loss_giou_aux: 7.586209 loss_class_dn: 0.397852 loss_bbox_dn: 0.071154 loss_giou_dn: 0.935707 loss_class_aux_dn: 1.969121 loss_bbox_aux_dn: 0.371115 loss_giou_aux_dn: 4.821790 loss: 22.030917 eta: 1 day, 8:55:41 batch_cost: 1.5193 data_cost: 0.0034 ips: 2.6328 images/s
[05/19 16:30:56] ppdet.engine INFO: Epoch: [4] [400/809] learning_rate: 0.000050 loss_class: 0.404732 loss_bbox: 0.159038 loss_giou: 1.277349 loss_class_aux: 2.376397 loss_bbox_aux: 0.997029 loss_giou_aux: 7.874786 loss_class_dn: 0.431166 loss_bbox_dn: 0.072206 loss_giou_dn: 0.959774 loss_class_aux_dn: 2.143662 loss_bbox_aux_dn: 0.382313 loss_giou_aux_dn: 4.960264 loss: 22.716452 eta: 1 day, 8:51:08 batch_cost: 1.5380 data_cost: 0.0032 ips: 2.6008 images/s
[05/19 16:36:11] ppdet.engine INFO: Epoch: [4] [600/809] learning_rate: 0.000050 loss_class: 0.439106 loss_bbox: 0.150066 loss_giou: 1.238603 loss_class_aux: 2.607087 loss_bbox_aux: 0.932903 loss_giou_aux: 7.576386 loss_class_dn: 0.433163 loss_bbox_dn: 0.072089 loss_giou_dn: 0.981932 loss_class_aux_dn: 2.159473 loss_bbox_aux_dn: 0.384373 loss_giou_aux_dn: 5.071344 loss: 22.863655 eta: 1 day, 8:44:23 batch_cost: 1.5062 data_cost: 0.0032 ips: 2.6556 images/s
[05/19 16:41:29] ppdet.engine INFO: Epoch: [4] [800/809] learning_rate: 0.000050 loss_class: 0.427356 loss_bbox: 0.144935 loss_giou: 1.246488 loss_class_aux: 2.510352 loss_bbox_aux: 0.921607 loss_giou_aux: 7.637641 loss_class_dn: 0.436988 loss_bbox_dn: 0.067377 loss_giou_dn: 0.941054 loss_class_aux_dn: 2.177366 loss_bbox_aux_dn: 0.353004 loss_giou_aux_dn: 4.842919 loss: 22.096016 eta: 1 day, 8:38:48 batch_cost: 1.5217 data_cost: 0.0031 ips: 2.6286 images/s
[05/19 16:41:48] ppdet.engine INFO: Epoch: [5] [  0/809] learning_rate: 0.000050 loss_class: 0.427356 loss_bbox: 0.144078 loss_giou: 1.225915 loss_class_aux: 2.510352 loss_bbox_aux: 0.900152 loss_giou_aux: 7.511260 loss_class_dn: 0.435205 loss_bbox_dn: 0.067458 loss_giou_dn: 0.932387 loss_class_aux_dn: 2.160869 loss_bbox_aux_dn: 0.354887 loss_giou_aux_dn: 4.822684 loss: 22.082722 eta: 1 day, 8:40:01 batch_cost: 1.5433 data_cost: 0.0206 ips: 2.5918 images/s
[05/19 16:47:03] ppdet.engine INFO: Epoch: [5] [200/809] learning_rate: 0.000050 loss_class: 0.416344 loss_bbox: 0.134254 loss_giou: 1.195670 loss_class_aux: 2.483057 loss_bbox_aux: 0.842103 loss_giou_aux: 7.363931 loss_class_dn: 0.427540 loss_bbox_dn: 0.063739 loss_giou_dn: 0.898471 loss_class_aux_dn: 2.130085 loss_bbox_aux_dn: 0.345421 loss_giou_aux_dn: 4.711903 loss: 21.641969 eta: 1 day, 8:33:22 batch_cost: 1.5045 data_cost: 0.0031 ips: 2.6587 images/s
[05/19 16:52:17] ppdet.engine INFO: Epoch: [5] [400/809] learning_rate: 0.000050 loss_class: 0.439757 loss_bbox: 0.147138 loss_giou: 1.250387 loss_class_aux: 2.605042 loss_bbox_aux: 0.934695 loss_giou_aux: 7.618828 loss_class_dn: 0.448039 loss_bbox_dn: 0.067709 loss_giou_dn: 0.956039 loss_class_aux_dn: 2.239351 loss_bbox_aux_dn: 0.360360 loss_giou_aux_dn: 4.945697 loss: 22.751977 eta: 1 day, 8:26:49 batch_cost: 1.5037 data_cost: 0.0034 ips: 2.6601 images/s
[05/19 16:57:35] ppdet.engine INFO: Epoch: [5] [600/809] learning_rate: 0.000050 loss_class: 0.415038 loss_bbox: 0.144688 loss_giou: 1.269066 loss_class_aux: 2.443092 loss_bbox_aux: 0.893336 loss_giou_aux: 7.799277 loss_class_dn: 0.429249 loss_bbox_dn: 0.061909 loss_giou_dn: 0.929492 loss_class_aux_dn: 2.148080 loss_bbox_aux_dn: 0.333044 loss_giou_aux_dn: 4.750091 loss: 21.959692 eta: 1 day, 8:21:20 batch_cost: 1.5206 data_cost: 0.0038 ips: 2.6305 images/s
[05/19 17:02:53] ppdet.engine INFO: Epoch: [5] [800/809] learning_rate: 0.000050 loss_class: 0.418301 loss_bbox: 0.143759 loss_giou: 1.313402 loss_class_aux: 2.504145 loss_bbox_aux: 0.902042 loss_giou_aux: 8.041492 loss_class_dn: 0.444108 loss_bbox_dn: 0.063485 loss_giou_dn: 1.000754 loss_class_aux_dn: 2.241478 loss_bbox_aux_dn: 0.344049 loss_giou_aux_dn: 5.151427 loss: 23.099792 eta: 1 day, 8:15:59 batch_cost: 1.5225 data_cost: 0.0032 ips: 2.6272 images/s
[05/19 17:03:14] ppdet.utils.checkpoint INFO: Save checkpoint: output/rtdetr_hgnetv2_x_6x_coco
[05/19 17:03:16] ppdet.engine INFO: Eval iter: 0
[05/19 17:04:19] ppdet.metrics.metrics INFO: The bbox result is saved to bbox.json.
loading annotations into memory...
Done (t=0.18s)
creating index...
index created!
[05/19 17:04:19] ppdet.metrics.coco_utils INFO: Start evaluate...
Loading and preparing results...
DONE (t=2.16s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=88.10s).
Accumulating evaluation results...
DONE (t=2.84s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.106
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.199
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.100
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.061
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.164
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.233
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.067
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.178
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.252
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.169
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.364
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.481
[05/19 17:05:52] ppdet.engine INFO: Total sample number: 548, average FPS: 9.477230135378782
[05/19 17:05:52] ppdet.engine INFO: Best test bbox ap is 0.106.
[05/19 17:05:57] ppdet.utils.checkpoint INFO: Save checkpoint: output/rtdetr_hgnetv2_x_6x_coco
[05/19 17:06:00] ppdet.engine INFO: Epoch: [6] [  0/809] learning_rate: 0.000050 loss_class: 0.425980 loss_bbox: 0.143759 loss_giou: 1.310089 loss_class_aux: 2.541620 loss_bbox_aux: 0.902042 loss_giou_aux: 8.041492 loss_class_dn: 0.449372 loss_bbox_dn: 0.065208 loss_giou_dn: 1.000754 loss_class_aux_dn: 2.258618 loss_bbox_aux_dn: 0.350981 loss_giou_aux_dn: 5.151427 loss: 23.164706 eta: 1 day, 8:15:17 batch_cost: 1.5171 data_cost: 0.0031 ips: 2.6366 images/s
[05/19 17:11:18] ppdet.engine INFO: Epoch: [6] [200/809] learning_rate: 0.000050 loss_class: 0.429866 loss_bbox: 0.137231 loss_giou: 1.197447 loss_class_aux: 2.546910 loss_bbox_aux: 0.854785 loss_giou_aux: 7.314628 loss_class_dn: 0.422318 loss_bbox_dn: 0.063673 loss_giou_dn: 0.872409 loss_class_aux_dn: 2.126632 loss_bbox_aux_dn: 0.339933 loss_giou_aux_dn: 4.482355 loss: 21.209758 eta: 1 day, 8:09:54 batch_cost: 1.5211 data_cost: 0.0031 ips: 2.6296 images/s
[05/19 17:16:31] ppdet.engine INFO: Epoch: [6] [400/809] learning_rate: 0.000050 loss_class: 0.419559 loss_bbox: 0.137924 loss_giou: 1.287495 loss_class_aux: 2.465828 loss_bbox_aux: 0.869333 loss_giou_aux: 7.882264 loss_class_dn: 0.432698 loss_bbox_dn: 0.063649 loss_giou_dn: 0.958573 loss_class_aux_dn: 2.178363 loss_bbox_aux_dn: 0.337747 loss_giou_aux_dn: 4.890231 loss: 22.252794 eta: 1 day, 8:03:36 batch_cost: 1.5012 data_cost: 0.0029 ips: 2.6645 images/s
[05/19 17:21:45] ppdet.engine INFO: Epoch: [6] [600/809] learning_rate: 0.000050 loss_class: 0.421520 loss_bbox: 0.134290 loss_giou: 1.202854 loss_class_aux: 2.511013 loss_bbox_aux: 0.848709 loss_giou_aux: 7.436869 loss_class_dn: 0.443853 loss_bbox_dn: 0.059378 loss_giou_dn: 0.895513 loss_class_aux_dn: 2.209766 loss_bbox_aux_dn: 0.322337 loss_giou_aux_dn: 4.637143 loss: 21.512446 eta: 1 day, 7:57:23 batch_cost: 1.5013 data_cost: 0.0029 ips: 2.6643 images/s
[05/19 17:27:00] ppdet.engine INFO: Epoch: [6] [800/809] learning_rate: 0.000050 loss_class: 0.422504 loss_bbox: 0.139663 loss_giou: 1.201944 loss_class_aux: 2.516703 loss_bbox_aux: 0.878355 loss_giou_aux: 7.341618 loss_class_dn: 0.434101 loss_bbox_dn: 0.061803 loss_giou_dn: 0.913200 loss_class_aux_dn: 2.177452 loss_bbox_aux_dn: 0.330588 loss_giou_aux_dn: 4.753887 loss: 21.829653 eta: 1 day, 7:51:44 batch_cost: 1.5122 data_cost: 0.0028 ips: 2.6451 images/s
[05/19 17:27:18] ppdet.engine INFO: Epoch: [7] [  0/809] learning_rate: 0.000050 loss_class: 0.418821 loss_bbox: 0.134234 loss_giou: 1.173768 loss_class_aux: 2.496694 loss_bbox_aux: 0.836058 loss_giou_aux: 7.246898 loss_class_dn: 0.427160 loss_bbox_dn: 0.060520 loss_giou_dn: 0.901536 loss_class_aux_dn: 2.147980 loss_bbox_aux_dn: 0.322598 loss_giou_aux_dn: 4.598039 loss: 21.291823 eta: 1 day, 7:52:13 batch_cost: 1.5312 data_cost: 0.0213 ips: 2.6124 images/s
[05/19 17:32:29] ppdet.engine INFO: Epoch: [7] [200/809] learning_rate: 0.000050 loss_class: 0.432405 loss_bbox: 0.147793 loss_giou: 1.254605 loss_class_aux: 2.561363 loss_bbox_aux: 0.950911 loss_giou_aux: 7.790249 loss_class_dn: 0.464233 loss_bbox_dn: 0.062773 loss_giou_dn: 0.935233 loss_class_aux_dn: 2.316103 loss_bbox_aux_dn: 0.333921 loss_giou_aux_dn: 4.866642 loss: 22.919467 eta: 1 day, 7:45:32 batch_cost: 1.4878 data_cost: 0.0034 ips: 2.6886 images/s
[05/19 17:37:54] ppdet.engine INFO: Epoch: [7] [400/809] learning_rate: 0.000050 loss_class: 0.434620 loss_bbox: 0.132677 loss_giou: 1.220665 loss_class_aux: 2.531556 loss_bbox_aux: 0.820142 loss_giou_aux: 7.452224 loss_class_dn: 0.448270 loss_bbox_dn: 0.056691 loss_giou_dn: 0.911016 loss_class_aux_dn: 2.263125 loss_bbox_aux_dn: 0.299916 loss_giou_aux_dn: 4.722756 loss: 22.180026 eta: 1 day, 7:41:41 batch_cost: 1.5535 data_cost: 0.0031 ips: 2.5748 images/s
[05/19 17:43:14] ppdet.engine INFO: Epoch: [7] [600/809] learning_rate: 0.000050 loss_class: 0.407909 loss_bbox: 0.126105 loss_giou: 1.207649 loss_class_aux: 2.422357 loss_bbox_aux: 0.793632 loss_giou_aux: 7.468938 loss_class_dn: 0.439111 loss_bbox_dn: 0.060689 loss_giou_dn: 0.891654 loss_class_aux_dn: 2.183531 loss_bbox_aux_dn: 0.320604 loss_giou_aux_dn: 4.623901 loss: 21.439321 eta: 1 day, 7:36:56 batch_cost: 1.5330 data_cost: 0.0031 ips: 2.6092 images/s
[05/19 17:48:33] ppdet.engine INFO: Epoch: [7] [800/809] learning_rate: 0.000050 loss_class: 0.421114 loss_bbox: 0.126030 loss_giou: 1.142547 loss_class_aux: 2.488523 loss_bbox_aux: 0.797455 loss_giou_aux: 6.968175 loss_class_dn: 0.430925 loss_bbox_dn: 0.057291 loss_giou_dn: 0.856886 loss_class_aux_dn: 2.153721 loss_bbox_aux_dn: 0.308376 loss_giou_aux_dn: 4.423200 loss: 20.382988 eta: 1 day, 7:32:06 batch_cost: 1.5315 data_cost: 0.0031 ips: 2.6118 images/s
[05/19 17:48:56] ppdet.utils.checkpoint INFO: Save checkpoint: output/rtdetr_hgnetv2_x_6x_coco
[05/19 17:48:58] ppdet.engine INFO: Eval iter: 0
[05/19 17:50:00] ppdet.metrics.metrics INFO: The bbox result is saved to bbox.json.
loading annotations into memory...
Done (t=0.39s)
creating index...
index created!
[05/19 17:50:00] ppdet.metrics.coco_utils INFO: Start evaluate...
Loading and preparing results...
DONE (t=1.83s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=85.94s).
Accumulating evaluation results...
DONE (t=2.99s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.134
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.246
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.126
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.073
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.203
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.298
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.078
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.213
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.296
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.197
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.426
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.583
[05/19 17:51:31] ppdet.engine INFO: Total sample number: 548, average FPS: 9.574730507462528
[05/19 17:51:31] ppdet.engine INFO: Best test bbox ap is 0.134.
[05/19 17:51:35] ppdet.utils.checkpoint INFO: Save checkpoint: output/rtdetr_hgnetv2_x_6x_coco
[05/19 17:51:38] ppdet.engine INFO: Epoch: [8] [  0/809] learning_rate: 0.000050 loss_class: 0.421921 loss_bbox: 0.126030 loss_giou: 1.142547 loss_class_aux: 2.501134 loss_bbox_aux: 0.797455 loss_giou_aux: 6.968175 loss_class_dn: 0.433434 loss_bbox_dn: 0.057708 loss_giou_dn: 0.856063 loss_class_aux_dn: 2.164366 loss_bbox_aux_dn: 0.309142 loss_giou_aux_dn: 4.410548 loss: 20.345675 eta: 1 day, 7:31:42 batch_cost: 1.5211 data_cost: 0.0030 ips: 2.6296 images/s
[05/19 17:56:55] ppdet.engine INFO: Epoch: [8] [200/809] learning_rate: 0.000050 loss_class: 0.428098 loss_bbox: 0.148262 loss_giou: 1.285808 loss_class_aux: 2.538441 loss_bbox_aux: 0.935736 loss_giou_aux: 7.906093 loss_class_dn: 0.459021 loss_bbox_dn: 0.059681 loss_giou_dn: 0.943503 loss_class_aux_dn: 2.303048 loss_bbox_aux_dn: 0.316242 loss_giou_aux_dn: 4.815533 loss: 23.035442 eta: 1 day, 7:26:13 batch_cost: 1.5139 data_cost: 0.0029 ips: 2.6421 images/s
[05/19 18:02:12] ppdet.engine INFO: Epoch: [8] [400/809] learning_rate: 0.000050 loss_class: 0.426870 loss_bbox: 0.132041 loss_giou: 1.282875 loss_class_aux: 2.545745 loss_bbox_aux: 0.821170 loss_giou_aux: 7.899350 loss_class_dn: 0.439323 loss_bbox_dn: 0.055548 loss_giou_dn: 0.888006 loss_class_aux_dn: 2.209943 loss_bbox_aux_dn: 0.299311 loss_giou_aux_dn: 4.656190 loss: 21.604054 eta: 1 day, 7:21:01 batch_cost: 1.5217 data_cost: 0.0030 ips: 2.6286 images/s
[05/19 18:07:27] ppdet.engine INFO: Epoch: [8] [600/809] learning_rate: 0.000050 loss_class: 0.410205 loss_bbox: 0.124859 loss_giou: 1.198613 loss_class_aux: 2.451552 loss_bbox_aux: 0.792151 loss_giou_aux: 7.406682 loss_class_dn: 0.432494 loss_bbox_dn: 0.055424 loss_giou_dn: 0.854781 loss_class_aux_dn: 2.170806 loss_bbox_aux_dn: 0.305655 loss_giou_aux_dn: 4.446372 loss: 20.708385 eta: 1 day, 7:15:13 batch_cost: 1.5039 data_cost: 0.0035 ips: 2.6598 images/s
[05/19 18:12:50] ppdet.engine INFO: Epoch: [8] [800/809] learning_rate: 0.000050 loss_class: 0.412045 loss_bbox: 0.119026 loss_giou: 1.123606 loss_class_aux: 2.450212 loss_bbox_aux: 0.762955 loss_giou_aux: 6.888147 loss_class_dn: 0.433477 loss_bbox_dn: 0.056541 loss_giou_dn: 0.833215 loss_class_aux_dn: 2.153019 loss_bbox_aux_dn: 0.301764 loss_giou_aux_dn: 4.319128 loss: 20.280858 eta: 1 day, 7:10:56 batch_cost: 1.5474 data_cost: 0.0033 ips: 2.5850 images/s
[05/19 18:13:11] ppdet.engine INFO: Epoch: [9] [  0/809] learning_rate: 0.000050 loss_class: 0.412957 loss_bbox: 0.117196 loss_giou: 1.121859 loss_class_aux: 2.450212 loss_bbox_aux: 0.759194 loss_giou_aux: 6.859609 loss_class_dn: 0.433477 loss_bbox_dn: 0.055948 loss_giou_dn: 0.827433 loss_class_aux_dn: 2.153519 loss_bbox_aux_dn: 0.299549 loss_giou_aux_dn: 4.298714 loss: 20.280858 eta: 1 day, 7:11:38 batch_cost: 1.5731 data_cost: 0.0290 ips: 2.5428 images/s
[05/19 18:18:24] ppdet.engine INFO: Epoch: [9] [200/809] learning_rate: 0.000050 loss_class: 0.419633 loss_bbox: 0.122247 loss_giou: 1.128094 loss_class_aux: 2.489519 loss_bbox_aux: 0.771502 loss_giou_aux: 6.956671 loss_class_dn: 0.427235 loss_bbox_dn: 0.055364 loss_giou_dn: 0.829149 loss_class_aux_dn: 2.142063 loss_bbox_aux_dn: 0.295651 loss_giou_aux_dn: 4.248597 loss: 19.760186 eta: 1 day, 7:05:36 batch_cost: 1.4965 data_cost: 0.0035 ips: 2.6728 images/s
[05/19 18:23:44] ppdet.engine INFO: Epoch: [9] [400/809] learning_rate: 0.000050 loss_class: 0.417275 loss_bbox: 0.133259 loss_giou: 1.211051 loss_class_aux: 2.478012 loss_bbox_aux: 0.859325 loss_giou_aux: 7.513721 loss_class_dn: 0.447039 loss_bbox_dn: 0.055864 loss_giou_dn: 0.872577 loss_class_aux_dn: 2.226910 loss_bbox_aux_dn: 0.300551 loss_giou_aux_dn: 4.475599 loss: 21.658102 eta: 1 day, 7:00:42 batch_cost: 1.5303 data_cost: 0.0034 ips: 2.6139 images/s
[05/19 18:28:56] ppdet.engine INFO: Epoch: [9] [600/809] learning_rate: 0.000050 loss_class: 0.419077 loss_bbox: 0.125358 loss_giou: 1.100177 loss_class_aux: 2.504738 loss_bbox_aux: 0.803391 loss_giou_aux: 6.767038 loss_class_dn: 0.428888 loss_bbox_dn: 0.054014 loss_giou_dn: 0.816281 loss_class_aux_dn: 2.140075 loss_bbox_aux_dn: 0.292343 loss_giou_aux_dn: 4.207686 loss: 20.061877 eta: 1 day, 6:54:49 batch_cost: 1.4986 data_cost: 0.0031 ips: 2.6692 images/s
[05/19 18:34:14] ppdet.engine INFO: Epoch: [9] [800/809] learning_rate: 0.000050 loss_class: 0.438403 loss_bbox: 0.132871 loss_giou: 1.226569 loss_class_aux: 2.595404 loss_bbox_aux: 0.832750 loss_giou_aux: 7.607233 loss_class_dn: 0.461205 loss_bbox_dn: 0.058489 loss_giou_dn: 0.889123 loss_class_aux_dn: 2.326297 loss_bbox_aux_dn: 0.319436 loss_giou_aux_dn: 4.596327 loss: 22.043205 eta: 1 day, 6:49:41 batch_cost: 1.5223 data_cost: 0.0033 ips: 2.6276 images/s
[05/19 18:34:39] ppdet.utils.checkpoint INFO: Save checkpoint: output/rtdetr_hgnetv2_x_6x_coco
[05/19 18:34:41] ppdet.engine INFO: Eval iter: 0
[05/19 18:35:44] ppdet.metrics.metrics INFO: The bbox result is saved to bbox.json.
loading annotations into memory...
Done (t=0.37s)
creating index...
index created!
[05/19 18:35:44] ppdet.metrics.coco_utils INFO: Start evaluate...
Loading and preparing results...
DONE (t=1.83s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=86.59s).
Accumulating evaluation results...
DONE (t=2.96s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.150
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.271
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.142
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.084
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.228
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.344
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.082
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.226
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.311
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.209
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.450
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.587
[05/19 18:37:16] ppdet.engine INFO: Total sample number: 548, average FPS: 9.396855090036201
[05/19 18:37:16] ppdet.engine INFO: Best test bbox ap is 0.150.
[05/19 18:37:21] ppdet.utils.checkpoint INFO: Save checkpoint: output/rtdetr_hgnetv2_x_6x_coco
[05/19 18:37:24] ppdet.engine INFO: Epoch: [10] [  0/809] learning_rate: 0.000050 loss_class: 0.433882 loss_bbox: 0.137049 loss_giou: 1.237881 loss_class_aux: 2.556598 loss_bbox_aux: 0.870321 loss_giou_aux: 7.664057 loss_class_dn: 0.461205 loss_bbox_dn: 0.057546 loss_giou_dn: 0.889123 loss_class_aux_dn: 2.326297 loss_bbox_aux_dn: 0.315962 loss_giou_aux_dn: 4.623044 loss: 22.190327 eta: 1 day, 6:49:21 batch_cost: 1.5119 data_cost: 0.0032 ips: 2.6457 images/s
[05/19 18:42:45] ppdet.engine INFO: Epoch: [10] [200/809] learning_rate: 0.000050 loss_class: 0.411245 loss_bbox: 0.128297 loss_giou: 1.247882 loss_class_aux: 2.427258 loss_bbox_aux: 0.793331 loss_giou_aux: 7.672417 loss_class_dn: 0.449199 loss_bbox_dn: 0.054674 loss_giou_dn: 0.888197 loss_class_aux_dn: 2.236542 loss_bbox_aux_dn: 0.290721 loss_giou_aux_dn: 4.608476 loss: 21.281130 eta: 1 day, 6:44:36 batch_cost: 1.5353 data_cost: 0.0032 ips: 2.6054 images/s
[05/19 18:48:00] ppdet.engine INFO: Epoch: [10] [400/809] learning_rate: 0.000050 loss_class: 0.420967 loss_bbox: 0.121505 loss_giou: 1.103981 loss_class_aux: 2.521424 loss_bbox_aux: 0.769328 loss_giou_aux: 6.820882 loss_class_dn: 0.451324 loss_bbox_dn: 0.057121 loss_giou_dn: 0.841659 loss_class_aux_dn: 2.258183 loss_bbox_aux_dn: 0.303207 loss_giou_aux_dn: 4.330127 loss: 20.266923 eta: 1 day, 6:39:04 batch_cost: 1.5085 data_cost: 0.0032 ips: 2.6516 images/s
[05/19 18:53:18] ppdet.engine INFO: Epoch: [10] [600/809] learning_rate: 0.000050 loss_class: 0.433765 loss_bbox: 0.137020 loss_giou: 1.247487 loss_class_aux: 2.554791 loss_bbox_aux: 0.872201 loss_giou_aux: 7.693666 loss_class_dn: 0.467692 loss_bbox_dn: 0.055276 loss_giou_dn: 0.896859 loss_class_aux_dn: 2.325897 loss_bbox_aux_dn: 0.293640 loss_giou_aux_dn: 4.618666 loss: 22.343904 eta: 1 day, 6:34:03 batch_cost: 1.5264 data_cost: 0.0029 ips: 2.6206 images/s
[05/19 18:58:38] ppdet.engine INFO: Epoch: [10] [800/809] learning_rate: 0.000050 loss_class: 0.437803 loss_bbox: 0.112498 loss_giou: 1.152090 loss_class_aux: 2.598303 loss_bbox_aux: 0.729612 loss_giou_aux: 7.129819 loss_class_dn: 0.444354 loss_bbox_dn: 0.053378 loss_giou_dn: 0.837471 loss_class_aux_dn: 2.234759 loss_bbox_aux_dn: 0.280448 loss_giou_aux_dn: 4.345734 loss: 20.765598 eta: 1 day, 6:29:15 batch_cost: 1.5340 data_cost: 0.0031 ips: 2.6076 images/s
[05/19 18:58:55] ppdet.engine INFO: Epoch: [11] [  0/809] learning_rate: 0.000050 loss_class: 0.437803 loss_bbox: 0.111698 loss_giou: 1.112460 loss_class_aux: 2.598303 loss_bbox_aux: 0.721548 loss_giou_aux: 6.790368 loss_class_dn: 0.444354 loss_bbox_dn: 0.053378 loss_giou_dn: 0.821370 loss_class_aux_dn: 2.234759 loss_bbox_aux_dn: 0.280448 loss_giou_aux_dn: 4.222326 loss: 20.520793 eta: 1 day, 6:29:22 batch_cost: 1.5479 data_cost: 0.0224 ips: 2.5841 images/s
[05/19 19:04:19] ppdet.engine INFO: Epoch: [11] [200/809] learning_rate: 0.000050 loss_class: 0.413518 loss_bbox: 0.126762 loss_giou: 1.238147 loss_class_aux: 2.453129 loss_bbox_aux: 0.800819 loss_giou_aux: 7.574897 loss_class_dn: 0.437790 loss_bbox_dn: 0.052165 loss_giou_dn: 0.868072 loss_class_aux_dn: 2.193814 loss_bbox_aux_dn: 0.277616 loss_giou_aux_dn: 4.449801 loss: 21.187895 eta: 1 day, 6:25:01 batch_cost: 1.5518 data_cost: 0.0030 ips: 2.5776 images/s
[05/19 19:09:39] ppdet.engine INFO: Epoch: [11] [400/809] learning_rate: 0.000050 loss_class: 0.444280 loss_bbox: 0.123001 loss_giou: 1.190689 loss_class_aux: 2.643560 loss_bbox_aux: 0.770710 loss_giou_aux: 7.380497 loss_class_dn: 0.457527 loss_bbox_dn: 0.054102 loss_giou_dn: 0.849708 loss_class_aux_dn: 2.316647 loss_bbox_aux_dn: 0.292224 loss_giou_aux_dn: 4.338295 loss: 21.024324 eta: 1 day, 6:20:13 batch_cost: 1.5364 data_cost: 0.0028 ips: 2.6034 images/s
[05/19 19:14:57] ppdet.engine INFO: Epoch: [11] [600/809] learning_rate: 0.000050 loss_class: 0.400358 loss_bbox: 0.114629 loss_giou: 1.168890 loss_class_aux: 2.407441 loss_bbox_aux: 0.745361 loss_giou_aux: 7.136606 loss_class_dn: 0.420148 loss_bbox_dn: 0.049797 loss_giou_dn: 0.806122 loss_class_aux_dn: 2.120821 loss_bbox_aux_dn: 0.268329 loss_giou_aux_dn: 4.149249 loss: 20.257504 eta: 1 day, 6:15:05 batch_cost: 1.5232 data_cost: 0.0032 ips: 2.6261 images/s
[05/19 19:20:12] ppdet.engine INFO: Epoch: [11] [800/809] learning_rate: 0.000050 loss_class: 0.406874 loss_bbox: 0.122262 loss_giou: 1.189215 loss_class_aux: 2.403627 loss_bbox_aux: 0.767229 loss_giou_aux: 7.309916 loss_class_dn: 0.434928 loss_bbox_dn: 0.050924 loss_giou_dn: 0.835017 loss_class_aux_dn: 2.187048 loss_bbox_aux_dn: 0.270047 loss_giou_aux_dn: 4.262701 loss: 20.650830 eta: 1 day, 6:09:41 batch_cost: 1.5127 data_cost: 0.0029 ips: 2.6443 images/s
[05/19 19:20:34] ppdet.utils.checkpoint INFO: Save checkpoint: output/rtdetr_hgnetv2_x_6x_coco
[05/19 19:20:37] ppdet.engine INFO: Eval iter: 0
[05/19 19:21:40] ppdet.metrics.metrics INFO: The bbox result is saved to bbox.json.
loading annotations into memory...
Done (t=0.17s)
creating index...
index created!
[05/19 19:21:40] ppdet.metrics.coco_utils INFO: Start evaluate...
Loading and preparing results...
DONE (t=1.88s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=86.79s).
Accumulating evaluation results...
DONE (t=2.98s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.166
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.297
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.159
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.093
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.252
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.368
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.085
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.239
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.323
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.222
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.465
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.620
[05/19 19:23:12] ppdet.engine INFO: Total sample number: 548, average FPS: 9.346074973435716
[05/19 19:23:12] ppdet.engine INFO: Best test bbox ap is 0.166.
[05/19 19:23:16] ppdet.utils.checkpoint INFO: Save checkpoint: output/rtdetr_hgnetv2_x_6x_coco
[05/19 19:23:19] ppdet.engine INFO: Epoch: [12] [  0/809] learning_rate: 0.000050 loss_class: 0.408708 loss_bbox: 0.124154 loss_giou: 1.210589 loss_class_aux: 2.426672 loss_bbox_aux: 0.786332 loss_giou_aux: 7.376902 loss_class_dn: 0.437742 loss_bbox_dn: 0.051092 loss_giou_dn: 0.845257 loss_class_aux_dn: 2.204078 loss_bbox_aux_dn: 0.273276 loss_giou_aux_dn: 4.363117 loss: 20.807848 eta: 1 day, 6:09:17 batch_cost: 1.5011 data_cost: 0.0029 ips: 2.6647 images/s
[05/19 19:28:38] ppdet.engine INFO: Epoch: [12] [200/809] learning_rate: 0.000050 loss_class: 0.402408 loss_bbox: 0.120926 loss_giou: 1.180716 loss_class_aux: 2.430677 loss_bbox_aux: 0.766148 loss_giou_aux: 7.369250 loss_class_dn: 0.445641 loss_bbox_dn: 0.050263 loss_giou_dn: 0.844366 loss_class_aux_dn: 2.231785 loss_bbox_aux_dn: 0.269592 loss_giou_aux_dn: 4.345338 loss: 20.905730 eta: 1 day, 6:04:18 batch_cost: 1.5294 data_cost: 0.0034 ips: 2.6154 images/s
[05/19 19:34:02] ppdet.engine INFO: Epoch: [12] [400/809] learning_rate: 0.000050 loss_class: 0.417045 loss_bbox: 0.118820 loss_giou: 1.155528 loss_class_aux: 2.502793 loss_bbox_aux: 0.780337 loss_giou_aux: 7.148621 loss_class_dn: 0.440636 loss_bbox_dn: 0.049272 loss_giou_dn: 0.813913 loss_class_aux_dn: 2.214595 loss_bbox_aux_dn: 0.271029 loss_giou_aux_dn: 4.212848 loss: 20.077355 eta: 1 day, 5:59:44 batch_cost: 1.5472 data_cost: 0.0037 ips: 2.5852 images/s
[05/19 19:39:23] ppdet.engine INFO: Epoch: [12] [600/809] learning_rate: 0.000050 loss_class: 0.414683 loss_bbox: 0.109308 loss_giou: 1.077818 loss_class_aux: 2.480036 loss_bbox_aux: 0.724133 loss_giou_aux: 6.682855 loss_class_dn: 0.412185 loss_bbox_dn: 0.049228 loss_giou_dn: 0.789800 loss_class_aux_dn: 2.073653 loss_bbox_aux_dn: 0.267586 loss_giou_aux_dn: 4.066059 loss: 19.510634 eta: 1 day, 5:54:58 batch_cost: 1.5390 data_cost: 0.0033 ips: 2.5990 images/s
[05/19 19:44:43] ppdet.engine INFO: Epoch: [12] [800/809] learning_rate: 0.000050 loss_class: 0.426378 loss_bbox: 0.114944 loss_giou: 1.162720 loss_class_aux: 2.551194 loss_bbox_aux: 0.734740 loss_giou_aux: 7.179102 loss_class_dn: 0.456591 loss_bbox_dn: 0.051404 loss_giou_dn: 0.842747 loss_class_aux_dn: 2.275122 loss_bbox_aux_dn: 0.284089 loss_giou_aux_dn: 4.335885 loss: 20.697582 eta: 1 day, 5:50:05 batch_cost: 1.5347 data_cost: 0.0030 ips: 2.6064 images/s
[05/19 19:45:03] ppdet.engine INFO: Epoch: [13] [  0/809] learning_rate: 0.000050 loss_class: 0.427116 loss_bbox: 0.116767 loss_giou: 1.164382 loss_class_aux: 2.539717 loss_bbox_aux: 0.741391 loss_giou_aux: 7.232064 loss_class_dn: 0.459500 loss_bbox_dn: 0.051865 loss_giou_dn: 0.846921 loss_class_aux_dn: 2.316904 loss_bbox_aux_dn: 0.284089 loss_giou_aux_dn: 4.353246 loss: 20.796409 eta: 1 day, 5:50:23 batch_cost: 1.5546 data_cost: 0.0162 ips: 2.5731 images/s
[05/19 19:50:18] ppdet.engine INFO: Epoch: [13] [200/809] learning_rate: 0.000050 loss_class: 0.428441 loss_bbox: 0.121475 loss_giou: 1.134263 loss_class_aux: 2.555169 loss_bbox_aux: 0.762793 loss_giou_aux: 7.003014 loss_class_dn: 0.456872 loss_bbox_dn: 0.055164 loss_giou_dn: 0.828898 loss_class_aux_dn: 2.300616 loss_bbox_aux_dn: 0.296098 loss_giou_aux_dn: 4.290641 loss: 20.802574 eta: 1 day, 5:44:52 batch_cost: 1.5068 data_cost: 0.0053 ips: 2.6546 images/s
[05/19 19:55:39] ppdet.engine INFO: Epoch: [13] [400/809] learning_rate: 0.000050 loss_class: 0.413615 loss_bbox: 0.106920 loss_giou: 1.093483 loss_class_aux: 2.473093 loss_bbox_aux: 0.689737 loss_giou_aux: 6.777129 loss_class_dn: 0.433800 loss_bbox_dn: 0.049559 loss_giou_dn: 0.813229 loss_class_aux_dn: 2.202834 loss_bbox_aux_dn: 0.267072 loss_giou_aux_dn: 4.167961 loss: 20.185640 eta: 1 day, 5:40:02 batch_cost: 1.5378 data_cost: 0.0030 ips: 2.6011 images/s
[05/19 20:00:53] ppdet.engine INFO: Epoch: [13] [600/809] learning_rate: 0.000050 loss_class: 0.436987 loss_bbox: 0.108302 loss_giou: 1.113934 loss_class_aux: 2.589442 loss_bbox_aux: 0.697655 loss_giou_aux: 6.887698 loss_class_dn: 0.451806 loss_bbox_dn: 0.049909 loss_giou_dn: 0.821313 loss_class_aux_dn: 2.273696 loss_bbox_aux_dn: 0.262377 loss_giou_aux_dn: 4.231777 loss: 20.379865 eta: 1 day, 5:34:30 batch_cost: 1.5042 data_cost: 0.0028 ips: 2.6593 images/s
[05/19 20:06:04] ppdet.engine INFO: Epoch: [13] [800/809] learning_rate: 0.000050 loss_class: 0.439010 loss_bbox: 0.118632 loss_giou: 1.116608 loss_class_aux: 2.650831 loss_bbox_aux: 0.756636 loss_giou_aux: 6.957331 loss_class_dn: 0.461011 loss_bbox_dn: 0.053521 loss_giou_dn: 0.825408 loss_class_aux_dn: 2.306534 loss_bbox_aux_dn: 0.287625 loss_giou_aux_dn: 4.225593 loss: 20.670030 eta: 1 day, 5:28:35 batch_cost: 1.4856 data_cost: 0.0034 ips: 2.6924 images/s
[05/19 20:06:28] ppdet.utils.checkpoint INFO: Save checkpoint: output/rtdetr_hgnetv2_x_6x_coco
[05/19 20:06:30] ppdet.engine INFO: Eval iter: 0
[05/19 20:07:32] ppdet.metrics.metrics INFO: The bbox result is saved to bbox.json.
loading annotations into memory...
Done (t=0.16s)
creating index...
index created!
[05/19 20:07:32] ppdet.metrics.coco_utils INFO: Start evaluate...
Loading and preparing results...
DONE (t=1.82s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=86.93s).
Accumulating evaluation results...
DONE (t=3.00s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.175
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.313
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.168
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.100
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.266
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.388
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.089
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.244
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.330
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.226
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.473
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.628
[05/19 20:09:05] ppdet.engine INFO: Total sample number: 548, average FPS: 9.486486334111424
[05/19 20:09:05] ppdet.engine INFO: Best test bbox ap is 0.175.
[05/19 20:09:09] ppdet.utils.checkpoint INFO: Save checkpoint: output/rtdetr_hgnetv2_x_6x_coco
[05/19 20:09:12] ppdet.engine INFO: Epoch: [14] [  0/809] learning_rate: 0.000050 loss_class: 0.443421 loss_bbox: 0.118318 loss_giou: 1.118818 loss_class_aux: 2.675296 loss_bbox_aux: 0.752874 loss_giou_aux: 6.957331 loss_class_dn: 0.463401 loss_bbox_dn: 0.054398 loss_giou_dn: 0.828187 loss_class_aux_dn: 2.313867 loss_bbox_aux_dn: 0.290985 loss_giou_aux_dn: 4.259701 loss: 20.836994 eta: 1 day, 5:28:11 batch_cost: 1.4816 data_cost: 0.0032 ips: 2.6998 images/s
[05/19 20:14:32] ppdet.engine INFO: Epoch: [14] [200/809] learning_rate: 0.000050 loss_class: 0.433989 loss_bbox: 0.110526 loss_giou: 1.135960 loss_class_aux: 2.565366 loss_bbox_aux: 0.711596 loss_giou_aux: 7.120257 loss_class_dn: 0.444007 loss_bbox_dn: 0.051595 loss_giou_dn: 0.847752 loss_class_aux_dn: 2.231608 loss_bbox_aux_dn: 0.274991 loss_giou_aux_dn: 4.383162 loss: 21.062346 eta: 1 day, 5:23:17 batch_cost: 1.5341 data_cost: 0.0033 ips: 2.6074 images/s
[05/19 20:19:53] ppdet.engine INFO: Epoch: [14] [400/809] learning_rate: 0.000050 loss_class: 0.423749 loss_bbox: 0.100904 loss_giou: 1.112726 loss_class_aux: 2.520124 loss_bbox_aux: 0.646458 loss_giou_aux: 6.877255 loss_class_dn: 0.455057 loss_bbox_dn: 0.048857 loss_giou_dn: 0.816254 loss_class_aux_dn: 2.277025 loss_bbox_aux_dn: 0.263608 loss_giou_aux_dn: 4.222300 loss: 20.566430 eta: 1 day, 5:18:22 batch_cost: 1.5334 data_cost: 0.0031 ips: 2.6086 images/s
[05/19 20:25:11] ppdet.engine INFO: Epoch: [14] [600/809] learning_rate: 0.000050 loss_class: 0.436003 loss_bbox: 0.110788 loss_giou: 1.108626 loss_class_aux: 2.626604 loss_bbox_aux: 0.715302 loss_giou_aux: 6.865783 loss_class_dn: 0.465062 loss_bbox_dn: 0.048340 loss_giou_dn: 0.823329 loss_class_aux_dn: 2.370601 loss_bbox_aux_dn: 0.261763 loss_giou_aux_dn: 4.255716 loss: 20.608557 eta: 1 day, 5:13:14 batch_cost: 1.5228 data_cost: 0.0029 ips: 2.6267 images/s
[05/19 20:30:34] ppdet.engine INFO: Epoch: [14] [800/809] learning_rate: 0.000050 loss_class: 0.436935 loss_bbox: 0.097704 loss_giou: 1.025382 loss_class_aux: 2.624734 loss_bbox_aux: 0.626063 loss_giou_aux: 6.304000 loss_class_dn: 0.435591 loss_bbox_dn: 0.051322 loss_giou_dn: 0.774359 loss_class_aux_dn: 2.190924 loss_bbox_aux_dn: 0.277409 loss_giou_aux_dn: 3.990137 loss: 18.826330 eta: 1 day, 5:08:31 batch_cost: 1.5446 data_cost: 0.0031 ips: 2.5897 images/s
[05/19 20:30:54] ppdet.engine INFO: Epoch: [15] [  0/809] learning_rate: 0.000050 loss_class: 0.432994 loss_bbox: 0.097962 loss_giou: 1.032987 loss_class_aux: 2.600545 loss_bbox_aux: 0.631921 loss_giou_aux: 6.402852 loss_class_dn: 0.436455 loss_bbox_dn: 0.051322 loss_giou_dn: 0.788301 loss_class_aux_dn: 2.194492 loss_bbox_aux_dn: 0.277596 loss_giou_aux_dn: 4.076251 loss: 19.183054 eta: 1 day, 5:08:47 batch_cost: 1.5663 data_cost: 0.0354 ips: 2.5538 images/s
[05/19 20:36:16] ppdet.engine INFO: Epoch: [15] [200/809] learning_rate: 0.000050 loss_class: 0.432954 loss_bbox: 0.105631 loss_giou: 1.136143 loss_class_aux: 2.595827 loss_bbox_aux: 0.710525 loss_giou_aux: 7.031455 loss_class_dn: 0.459465 loss_bbox_dn: 0.048378 loss_giou_dn: 0.824429 loss_class_aux_dn: 2.300311 loss_bbox_aux_dn: 0.260195 loss_giou_aux_dn: 4.245862 loss: 20.661277 eta: 1 day, 5:03:55 batch_cost: 1.5377 data_cost: 0.0030 ips: 2.6013 images/s
[05/19 20:41:31] ppdet.engine INFO: Epoch: [15] [400/809] learning_rate: 0.000050 loss_class: 0.415921 loss_bbox: 0.108511 loss_giou: 1.093254 loss_class_aux: 2.483230 loss_bbox_aux: 0.720151 loss_giou_aux: 6.695865 loss_class_dn: 0.431579 loss_bbox_dn: 0.050198 loss_giou_dn: 0.795518 loss_class_aux_dn: 2.199084 loss_bbox_aux_dn: 0.273365 loss_giou_aux_dn: 4.152126 loss: 19.420871 eta: 1 day, 4:58:28 batch_cost: 1.5065 data_cost: 0.0031 ips: 2.6552 images/s
[05/19 20:46:59] ppdet.engine INFO: Epoch: [15] [600/809] learning_rate: 0.000050 loss_class: 0.425130 loss_bbox: 0.113561 loss_giou: 1.111710 loss_class_aux: 2.519997 loss_bbox_aux: 0.741051 loss_giou_aux: 6.814844 loss_class_dn: 0.433441 loss_bbox_dn: 0.050302 loss_giou_dn: 0.800254 loss_class_aux_dn: 2.205805 loss_bbox_aux_dn: 0.270467 loss_giou_aux_dn: 4.121264 loss: 19.883920 eta: 1 day, 4:54:02 batch_cost: 1.5623 data_cost: 0.0032 ips: 2.5603 images/s
[05/19 20:52:27] ppdet.engine INFO: Epoch: [15] [800/809] learning_rate: 0.000050 loss_class: 0.424899 loss_bbox: 0.109709 loss_giou: 1.178687 loss_class_aux: 2.530547 loss_bbox_aux: 0.715606 loss_giou_aux: 7.307207 loss_class_dn: 0.461841 loss_bbox_dn: 0.049727 loss_giou_dn: 0.859830 loss_class_aux_dn: 2.320712 loss_bbox_aux_dn: 0.266774 loss_giou_aux_dn: 4.428397 loss: 21.355911 eta: 1 day, 4:49:41 batch_cost: 1.5681 data_cost: 0.0037 ips: 2.5509 images/s
[05/19 20:52:49] ppdet.utils.checkpoint INFO: Save checkpoint: output/rtdetr_hgnetv2_x_6x_coco
[05/19 20:52:52] ppdet.engine INFO: Eval iter: 0
[05/19 20:53:54] ppdet.metrics.metrics INFO: The bbox result is saved to bbox.json.
loading annotations into memory...
Done (t=0.37s)
creating index...
index created!
[05/19 20:53:54] ppdet.metrics.coco_utils INFO: Start evaluate...
Loading and preparing results...
DONE (t=1.81s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=89.58s).
Accumulating evaluation results...
DONE (t=3.40s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.183
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.327
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.177
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.103
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.280
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.405
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.091
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.251
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.337
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.233
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.482
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.659
[05/19 20:55:29] ppdet.engine INFO: Total sample number: 548, average FPS: 9.618830273725877
[05/19 20:55:29] ppdet.engine INFO: Best test bbox ap is 0.183.
[05/19 20:55:33] ppdet.utils.checkpoint INFO: Save checkpoint: output/rtdetr_hgnetv2_x_6x_coco
[05/19 20:55:37] ppdet.engine INFO: Epoch: [16] [  0/809] learning_rate: 0.000050 loss_class: 0.426735 loss_bbox: 0.109709 loss_giou: 1.180696 loss_class_aux: 2.534021 loss_bbox_aux: 0.716107 loss_giou_aux: 7.349123 loss_class_dn: 0.460419 loss_bbox_dn: 0.050031 loss_giou_dn: 0.858318 loss_class_aux_dn: 2.321493 loss_bbox_aux_dn: 0.266986 loss_giou_aux_dn: 4.422057 loss: 21.355911 eta: 1 day, 4:49:18 batch_cost: 1.5559 data_cost: 0.0036 ips: 2.5709 images/s
[05/19 21:01:03] ppdet.engine INFO: Epoch: [16] [200/809] learning_rate: 0.000050 loss_class: 0.411606 loss_bbox: 0.104770 loss_giou: 1.080903 loss_class_aux: 2.448182 loss_bbox_aux: 0.673638 loss_giou_aux: 6.731490 loss_class_dn: 0.435598 loss_bbox_dn: 0.047385 loss_giou_dn: 0.785369 loss_class_aux_dn: 2.187655 loss_bbox_aux_dn: 0.255229 loss_giou_aux_dn: 4.049860 loss: 19.391649 eta: 1 day, 4:44:44 batch_cost: 1.5572 data_cost: 0.0035 ips: 2.5687 images/s
[05/19 21:06:26] ppdet.engine INFO: Epoch: [16] [400/809] learning_rate: 0.000050 loss_class: 0.442544 loss_bbox: 0.103178 loss_giou: 1.094572 loss_class_aux: 2.601705 loss_bbox_aux: 0.693714 loss_giou_aux: 6.799632 loss_class_dn: 0.442692 loss_bbox_dn: 0.050167 loss_giou_dn: 0.794994 loss_class_aux_dn: 2.231438 loss_bbox_aux_dn: 0.270942 loss_giou_aux_dn: 4.126547 loss: 20.015273 eta: 1 day, 4:39:56 batch_cost: 1.5443 data_cost: 0.0030 ips: 2.5901 images/s
[05/19 21:11:50] ppdet.engine INFO: Epoch: [16] [600/809] learning_rate: 0.000050 loss_class: 0.399524 loss_bbox: 0.104324 loss_giou: 1.059883 loss_class_aux: 2.402314 loss_bbox_aux: 0.700221 loss_giou_aux: 6.605078 loss_class_dn: 0.430299 loss_bbox_dn: 0.046994 loss_giou_dn: 0.777476 loss_class_aux_dn: 2.164647 loss_bbox_aux_dn: 0.251543 loss_giou_aux_dn: 3.990280 loss: 19.484631 eta: 1 day, 4:35:10 batch_cost: 1.5472 data_cost: 0.0033 ips: 2.5853 images/s
[05/19 21:17:12] ppdet.engine INFO: Epoch: [16] [800/809] learning_rate: 0.000050 loss_class: 0.434314 loss_bbox: 0.103913 loss_giou: 1.091605 loss_class_aux: 2.655689 loss_bbox_aux: 0.667386 loss_giou_aux: 6.745966 loss_class_dn: 0.463212 loss_bbox_dn: 0.049267 loss_giou_dn: 0.805917 loss_class_aux_dn: 2.324829 loss_bbox_aux_dn: 0.259756 loss_giou_aux_dn: 4.162534 loss: 20.212687 eta: 1 day, 4:30:11 batch_cost: 1.5352 data_cost: 0.0033 ips: 2.6055 images/s
[05/19 21:17:32] ppdet.engine INFO: Epoch: [17] [  0/809] learning_rate: 0.000050 loss_class: 0.452119 loss_bbox: 0.102735 loss_giou: 1.085239 loss_class_aux: 2.704806 loss_bbox_aux: 0.660034 loss_giou_aux: 6.709478 loss_class_dn: 0.463212 loss_bbox_dn: 0.048194 loss_giou_dn: 0.801145 loss_class_aux_dn: 2.320064 loss_bbox_aux_dn: 0.258953 loss_giou_aux_dn: 4.160968 loss: 20.157332 eta: 1 day, 4:30:23 batch_cost: 1.5598 data_cost: 0.0285 ips: 2.5644 images/s
[05/19 21:22:58] ppdet.engine INFO: Epoch: [17] [200/809] learning_rate: 0.000050 loss_class: 0.431330 loss_bbox: 0.109550 loss_giou: 1.130962 loss_class_aux: 2.568758 loss_bbox_aux: 0.720181 loss_giou_aux: 7.025277 loss_class_dn: 0.451790 loss_bbox_dn: 0.048315 loss_giou_dn: 0.835046 loss_class_aux_dn: 2.278407 loss_bbox_aux_dn: 0.259711 loss_giou_aux_dn: 4.311138 loss: 20.499879 eta: 1 day, 4:25:46 batch_cost: 1.5580 data_cost: 0.0035 ips: 2.5673 images/s
[05/19 21:28:25] ppdet.engine INFO: Epoch: [17] [400/809] learning_rate: 0.000050 loss_class: 0.425439 loss_bbox: 0.106358 loss_giou: 1.046946 loss_class_aux: 2.549324 loss_bbox_aux: 0.688005 loss_giou_aux: 6.473017 loss_class_dn: 0.440469 loss_bbox_dn: 0.048794 loss_giou_dn: 0.753860 loss_class_aux_dn: 2.237640 loss_bbox_aux_dn: 0.263235 loss_giou_aux_dn: 3.909724 loss: 19.159451 eta: 1 day, 4:21:10 batch_cost: 1.5607 data_cost: 0.0033 ips: 2.5629 images/s
[05/19 21:33:49] ppdet.engine INFO: Epoch: [17] [600/809] learning_rate: 0.000050 loss_class: 0.428414 loss_bbox: 0.111100 loss_giou: 1.135722 loss_class_aux: 2.583326 loss_bbox_aux: 0.707586 loss_giou_aux: 7.031208 loss_class_dn: 0.454695 loss_bbox_dn: 0.048686 loss_giou_dn: 0.828941 loss_class_aux_dn: 2.318829 loss_bbox_aux_dn: 0.262727 loss_giou_aux_dn: 4.286703 loss: 20.690388 eta: 1 day, 4:16:23 batch_cost: 1.5491 data_cost: 0.0032 ips: 2.5822 images/s
[05/19 21:39:08] ppdet.engine INFO: Epoch: [17] [800/809] learning_rate: 0.000050 loss_class: 0.406409 loss_bbox: 0.106793 loss_giou: 1.053476 loss_class_aux: 2.460137 loss_bbox_aux: 0.705035 loss_giou_aux: 6.529109 loss_class_dn: 0.437559 loss_bbox_dn: 0.048573 loss_giou_dn: 0.771413 loss_class_aux_dn: 2.194560 loss_bbox_aux_dn: 0.262258 loss_giou_aux_dn: 3.955232 loss: 19.440296 eta: 1 day, 4:11:15 batch_cost: 1.5276 data_cost: 0.0036 ips: 2.6185 images/s
[05/19 21:39:31] ppdet.utils.checkpoint INFO: Save checkpoint: output/rtdetr_hgnetv2_x_6x_coco
[05/19 21:39:33] ppdet.engine INFO: Eval iter: 0
[05/19 21:40:35] ppdet.metrics.metrics INFO: The bbox result is saved to bbox.json.
loading annotations into memory...
Done (t=0.16s)
creating index...
index created!
[05/19 21:40:36] ppdet.metrics.coco_utils INFO: Start evaluate...
Loading and preparing results...
DONE (t=1.91s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=88.22s).
Accumulating evaluation results...
DONE (t=3.04s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.189
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.336
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.184
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.108
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.286
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.413
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.094
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.256
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.345
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.241
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.490
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.664
[05/19 21:42:09] ppdet.engine INFO: Total sample number: 548, average FPS: 9.462510802996558
[05/19 21:42:10] ppdet.engine INFO: Best test bbox ap is 0.189.
[05/19 21:42:14] ppdet.utils.checkpoint INFO: Save checkpoint: output/rtdetr_hgnetv2_x_6x_coco
[05/19 21:42:18] ppdet.engine INFO: Epoch: [18] [  0/809] learning_rate: 0.000050 loss_class: 0.410969 loss_bbox: 0.107800 loss_giou: 1.089760 loss_class_aux: 2.500042 loss_bbox_aux: 0.717631 loss_giou_aux: 6.604051 loss_class_dn: 0.437285 loss_bbox_dn: 0.048573 loss_giou_dn: 0.775267 loss_class_aux_dn: 2.194560 loss_bbox_aux_dn: 0.262258 loss_giou_aux_dn: 3.993620 loss: 19.995757 eta: 1 day, 4:10:57 batch_cost: 1.5200 data_cost: 0.0035 ips: 2.6316 images/s
[05/19 21:47:33] ppdet.engine INFO: Epoch: [18] [200/809] learning_rate: 0.000050 loss_class: 0.422713 loss_bbox: 0.108138 loss_giou: 1.183187 loss_class_aux: 2.530434 loss_bbox_aux: 0.711213 loss_giou_aux: 7.300576 loss_class_dn: 0.450628 loss_bbox_dn: 0.047853 loss_giou_dn: 0.827177 loss_class_aux_dn: 2.284361 loss_bbox_aux_dn: 0.253581 loss_giou_aux_dn: 4.239742 loss: 20.545209 eta: 1 day, 4:05:30 batch_cost: 1.5066 data_cost: 0.0033 ips: 2.6549 images/s
[05/19 21:52:51] ppdet.engine INFO: Epoch: [18] [400/809] learning_rate: 0.000050 loss_class: 0.434604 loss_bbox: 0.103709 loss_giou: 0.999924 loss_class_aux: 2.644462 loss_bbox_aux: 0.664847 loss_giou_aux: 6.280287 loss_class_dn: 0.447816 loss_bbox_dn: 0.047631 loss_giou_dn: 0.759216 loss_class_aux_dn: 2.239352 loss_bbox_aux_dn: 0.255536 loss_giou_aux_dn: 3.932020 loss: 19.567024 eta: 1 day, 4:00:18 batch_cost: 1.5218 data_cost: 0.0031 ips: 2.6285 images/s
[05/19 21:58:18] ppdet.engine INFO: Epoch: [18] [600/809] learning_rate: 0.000050 loss_class: 0.429189 loss_bbox: 0.101569 loss_giou: 1.102965 loss_class_aux: 2.601172 loss_bbox_aux: 0.654243 loss_giou_aux: 6.777876 loss_class_dn: 0.435827 loss_bbox_dn: 0.047668 loss_giou_dn: 0.791285 loss_class_aux_dn: 2.188233 loss_bbox_aux_dn: 0.252379 loss_giou_aux_dn: 4.101030 loss: 19.875921 eta: 1 day, 3:55:47 batch_cost: 1.5695 data_cost: 0.0031 ips: 2.5486 images/s
[05/19 22:03:36] ppdet.engine INFO: Epoch: [18] [800/809] learning_rate: 0.000050 loss_class: 0.429323 loss_bbox: 0.099922 loss_giou: 1.071170 loss_class_aux: 2.569081 loss_bbox_aux: 0.670620 loss_giou_aux: 6.606223 loss_class_dn: 0.439011 loss_bbox_dn: 0.050705 loss_giou_dn: 0.815065 loss_class_aux_dn: 2.237349 loss_bbox_aux_dn: 0.270902 loss_giou_aux_dn: 4.158551 loss: 20.343513 eta: 1 day, 3:50:31 batch_cost: 1.5169 data_cost: 0.0032 ips: 2.6369 images/s
[05/19 22:03:57] ppdet.engine INFO: Epoch: [19] [  0/809] learning_rate: 0.000050 loss_class: 0.427018 loss_bbox: 0.100664 loss_giou: 1.081433 loss_class_aux: 2.568219 loss_bbox_aux: 0.673250 loss_giou_aux: 6.703489 loss_class_dn: 0.440891 loss_bbox_dn: 0.049977 loss_giou_dn: 0.818203 loss_class_aux_dn: 2.237349 loss_bbox_aux_dn: 0.261052 loss_giou_aux_dn: 4.247322 loss: 20.544723 eta: 1 day, 3:50:44 batch_cost: 1.5471 data_cost: 0.0393 ips: 2.5855 images/s
[05/19 22:09:19] ppdet.engine INFO: Epoch: [19] [200/809] learning_rate: 0.000050 loss_class: 0.439534 loss_bbox: 0.102574 loss_giou: 1.070171 loss_class_aux: 2.609414 loss_bbox_aux: 0.660989 loss_giou_aux: 6.600355 loss_class_dn: 0.431788 loss_bbox_dn: 0.048873 loss_giou_dn: 0.774051 loss_class_aux_dn: 2.190632 loss_bbox_aux_dn: 0.255413 loss_giou_aux_dn: 3.996566 loss: 19.466007 eta: 1 day, 3:45:45 batch_cost: 1.5382 data_cost: 0.0030 ips: 2.6004 images/s
[05/19 22:14:36] ppdet.engine INFO: Epoch: [19] [400/809] learning_rate: 0.000050 loss_class: 0.419104 loss_bbox: 0.102088 loss_giou: 0.984127 loss_class_aux: 2.509048 loss_bbox_aux: 0.644495 loss_giou_aux: 6.106618 loss_class_dn: 0.424034 loss_bbox_dn: 0.046654 loss_giou_dn: 0.726435 loss_class_aux_dn: 2.141857 loss_bbox_aux_dn: 0.251824 loss_giou_aux_dn: 3.787300 loss: 18.612819 eta: 1 day, 3:40:31 batch_cost: 1.5205 data_cost: 0.0031 ips: 2.6307 images/s
[05/19 22:19:58] ppdet.engine INFO: Epoch: [19] [600/809] learning_rate: 0.000050 loss_class: 0.403948 loss_bbox: 0.109618 loss_giou: 1.115691 loss_class_aux: 2.419821 loss_bbox_aux: 0.694635 loss_giou_aux: 6.927968 loss_class_dn: 0.442674 loss_bbox_dn: 0.048133 loss_giou_dn: 0.808397 loss_class_aux_dn: 2.229757 loss_bbox_aux_dn: 0.256502 loss_giou_aux_dn: 4.142843 loss: 20.483940 eta: 1 day, 3:35:35 batch_cost: 1.5418 data_cost: 0.0029 ips: 2.5943 images/s
[05/19 22:25:16] ppdet.engine INFO: Epoch: [19] [800/809] learning_rate: 0.000050 loss_class: 0.432844 loss_bbox: 0.106970 loss_giou: 1.101762 loss_class_aux: 2.599699 loss_bbox_aux: 0.697411 loss_giou_aux: 6.798186 loss_class_dn: 0.431534 loss_bbox_dn: 0.049994 loss_giou_dn: 0.795808 loss_class_aux_dn: 2.180136 loss_bbox_aux_dn: 0.262189 loss_giou_aux_dn: 4.073469 loss: 20.073274 eta: 1 day, 3:30:24 batch_cost: 1.5234 data_cost: 0.0031 ips: 2.6257 images/s
[05/19 22:25:37] ppdet.utils.checkpoint INFO: Save checkpoint: output/rtdetr_hgnetv2_x_6x_coco
[05/19 22:25:40] ppdet.engine INFO: Eval iter: 0
[05/19 22:26:42] ppdet.metrics.metrics INFO: The bbox result is saved to bbox.json.
loading annotations into memory...
Done (t=0.16s)
creating index...
index created!
[05/19 22:26:42] ppdet.metrics.coco_utils INFO: Start evaluate...
Loading and preparing results...
DONE (t=1.91s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=87.19s).
Accumulating evaluation results...
DONE (t=2.97s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.194
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.343
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.187
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.112
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.292
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.435
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.095
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.262
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.350
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.246
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.496
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.667
[05/19 22:28:15] ppdet.engine INFO: Total sample number: 548, average FPS: 9.448450950165796
[05/19 22:28:15] ppdet.engine INFO: Best test bbox ap is 0.194.
[05/19 22:28:19] ppdet.utils.checkpoint INFO: Save checkpoint: output/rtdetr_hgnetv2_x_6x_coco
[05/19 22:28:23] ppdet.engine INFO: Epoch: [20] [  0/809] learning_rate: 0.000050 loss_class: 0.427192 loss_bbox: 0.106564 loss_giou: 1.101762 loss_class_aux: 2.568571 loss_bbox_aux: 0.695910 loss_giou_aux: 6.798186 loss_class_dn: 0.428964 loss_bbox_dn: 0.049484 loss_giou_dn: 0.792619 loss_class_aux_dn: 2.172684 loss_bbox_aux_dn: 0.259722 loss_giou_aux_dn: 4.073469 loss: 19.972837 eta: 1 day, 3:30:04 batch_cost: 1.5215 data_cost: 0.0031 ips: 2.6291 images/s
[05/19 22:33:45] ppdet.engine INFO: Epoch: [20] [200/809] learning_rate: 0.000050 loss_class: 0.424908 loss_bbox: 0.098015 loss_giou: 0.994017 loss_class_aux: 2.583952 loss_bbox_aux: 0.627881 loss_giou_aux: 6.223616 loss_class_dn: 0.433273 loss_bbox_dn: 0.046753 loss_giou_dn: 0.729472 loss_class_aux_dn: 2.188259 loss_bbox_aux_dn: 0.250058 loss_giou_aux_dn: 3.764834 loss: 19.101990 eta: 1 day, 3:25:07 batch_cost: 1.5417 data_cost: 0.0031 ips: 2.5946 images/s
[05/19 22:39:09] ppdet.engine INFO: Epoch: [20] [400/809] learning_rate: 0.000050 loss_class: 0.439074 loss_bbox: 0.101627 loss_giou: 1.078354 loss_class_aux: 2.597941 loss_bbox_aux: 0.661219 loss_giou_aux: 6.710746 loss_class_dn: 0.442014 loss_bbox_dn: 0.046792 loss_giou_dn: 0.796045 loss_class_aux_dn: 2.234278 loss_bbox_aux_dn: 0.254977 loss_giou_aux_dn: 4.057546 loss: 19.896277 eta: 1 day, 3:20:18 batch_cost: 1.5512 data_cost: 0.0034 ips: 2.5787 images/s
[05/19 22:44:28] ppdet.engine INFO: Epoch: [20] [600/809] learning_rate: 0.000050 loss_class: 0.427562 loss_bbox: 0.105193 loss_giou: 1.106953 loss_class_aux: 2.609574 loss_bbox_aux: 0.680214 loss_giou_aux: 6.873988 loss_class_dn: 0.444409 loss_bbox_dn: 0.045926 loss_giou_dn: 0.786736 loss_class_aux_dn: 2.241605 loss_bbox_aux_dn: 0.249607 loss_giou_aux_dn: 4.024057 loss: 19.764440 eta: 1 day, 3:15:09 batch_cost: 1.5261 data_cost: 0.0033 ips: 2.6211 images/s
[05/19 22:49:51] ppdet.engine INFO: Epoch: [20] [800/809] learning_rate: 0.000050 loss_class: 0.424098 loss_bbox: 0.099278 loss_giou: 1.094360 loss_class_aux: 2.501024 loss_bbox_aux: 0.653616 loss_giou_aux: 6.887167 loss_class_dn: 0.419855 loss_bbox_dn: 0.045780 loss_giou_dn: 0.801813 loss_class_aux_dn: 2.124833 loss_bbox_aux_dn: 0.242010 loss_giou_aux_dn: 4.126681 loss: 20.347675 eta: 1 day, 3:10:16 batch_cost: 1.5476 data_cost: 0.0031 ips: 2.5847 images/s
[05/19 22:50:14] ppdet.engine INFO: Epoch: [21] [  0/809] learning_rate: 0.000050 loss_class: 0.424926 loss_bbox: 0.099867 loss_giou: 1.102390 loss_class_aux: 2.511484 loss_bbox_aux: 0.658278 loss_giou_aux: 6.887167 loss_class_dn: 0.421665 loss_bbox_dn: 0.045101 loss_giou_dn: 0.801813 loss_class_aux_dn: 2.128756 loss_bbox_aux_dn: 0.240279 loss_giou_aux_dn: 4.126681 loss: 20.211712 eta: 1 day, 3:10:30 batch_cost: 1.5842 data_cost: 0.0364 ips: 2.5249 images/s
[05/19 22:55:40] ppdet.engine INFO: Epoch: [21] [200/809] learning_rate: 0.000050 loss_class: 0.415993 loss_bbox: 0.099920 loss_giou: 1.088938 loss_class_aux: 2.464078 loss_bbox_aux: 0.641980 loss_giou_aux: 6.758847 loss_class_dn: 0.432034 loss_bbox_dn: 0.046324 loss_giou_dn: 0.802070 loss_class_aux_dn: 2.167834 loss_bbox_aux_dn: 0.247427 loss_giou_aux_dn: 4.140310 loss: 20.152469 eta: 1 day, 3:05:43 batch_cost: 1.5571 data_cost: 0.0040 ips: 2.5689 images/s
[05/19 23:01:01] ppdet.engine INFO: Epoch: [21] [400/809] learning_rate: 0.000050 loss_class: 0.425511 loss_bbox: 0.092980 loss_giou: 1.034090 loss_class_aux: 2.536182 loss_bbox_aux: 0.605063 loss_giou_aux: 6.440016 loss_class_dn: 0.441849 loss_bbox_dn: 0.047424 loss_giou_dn: 0.773020 loss_class_aux_dn: 2.234788 loss_bbox_aux_dn: 0.260104 loss_giou_aux_dn: 3.981696 loss: 19.291510 eta: 1 day, 3:00:38 batch_cost: 1.5331 data_cost: 0.0031 ips: 2.6092 images/s
[05/19 23:06:23] ppdet.engine INFO: Epoch: [21] [600/809] learning_rate: 0.000050 loss_class: 0.411013 loss_bbox: 0.095642 loss_giou: 1.042207 loss_class_aux: 2.466769 loss_bbox_aux: 0.613877 loss_giou_aux: 6.486618 loss_class_dn: 0.410911 loss_bbox_dn: 0.043694 loss_giou_dn: 0.749082 loss_class_aux_dn: 2.057820 loss_bbox_aux_dn: 0.233101 loss_giou_aux_dn: 3.823772 loss: 18.642427 eta: 1 day, 2:55:38 batch_cost: 1.5390 data_cost: 0.0034 ips: 2.5992 images/s
[05/19 23:11:46] ppdet.engine INFO: Epoch: [21] [800/809] learning_rate: 0.000050 loss_class: 0.404156 loss_bbox: 0.105445 loss_giou: 1.132116 loss_class_aux: 2.452860 loss_bbox_aux: 0.678006 loss_giou_aux: 6.993553 loss_class_dn: 0.446227 loss_bbox_dn: 0.046570 loss_giou_dn: 0.799029 loss_class_aux_dn: 2.233168 loss_bbox_aux_dn: 0.245926 loss_giou_aux_dn: 4.094008 loss: 19.763546 eta: 1 day, 2:50:39 batch_cost: 1.5421 data_cost: 0.0030 ips: 2.5939 images/s
[05/19 23:12:09] ppdet.utils.checkpoint INFO: Save checkpoint: output/rtdetr_hgnetv2_x_6x_coco
[05/19 23:12:12] ppdet.engine INFO: Eval iter: 0
[05/19 23:13:15] ppdet.metrics.metrics INFO: The bbox result is saved to bbox.json.
loading annotations into memory...
Done (t=0.38s)
creating index...
index created!
[05/19 23:13:15] ppdet.metrics.coco_utils INFO: Start evaluate...
Loading and preparing results...
DONE (t=1.87s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=88.76s).
Accumulating evaluation results...
DONE (t=3.47s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.197
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.348
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.191
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.115
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.295
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.427
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.097
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.264
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.353
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.249
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.499
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.656
[05/19 23:14:50] ppdet.engine INFO: Total sample number: 548, average FPS: 9.430969055928884
[05/19 23:14:50] ppdet.engine INFO: Best test bbox ap is 0.197.
[05/19 23:14:54] ppdet.utils.checkpoint INFO: Save checkpoint: output/rtdetr_hgnetv2_x_6x_coco
[05/19 23:14:57] ppdet.engine INFO: Epoch: [22] [  0/809] learning_rate: 0.000050 loss_class: 0.412819 loss_bbox: 0.103075 loss_giou: 1.052239 loss_class_aux: 2.479814 loss_bbox_aux: 0.660950 loss_giou_aux: 6.559781 loss_class_dn: 0.444406 loss_bbox_dn: 0.047018 loss_giou_dn: 0.780373 loss_class_aux_dn: 2.233168 loss_bbox_aux_dn: 0.249130 loss_giou_aux_dn: 4.008662 loss: 19.544210 eta: 1 day, 2:50:23 batch_cost: 1.5402 data_cost: 0.0030 ips: 2.5971 images/s


--------------------------------------
C++ Traceback (most recent call last):
--------------------------------------
0   phi::backends::gpu::GpuMemcpySync(void*, void const*, unsigned long, cudaMemcpyKind)

----------------------
Error Message Summary:
----------------------
FatalError: `Termination signal` is detected by the operating system.
  [TimeInfo: *** Aborted at 1716160565 (unix time) try "date -d @1716160565" if you are using GNU date ***]
  [SignalInfo: *** SIGTERM (@0xa3) received by PID 177 (TID 0x7fe2823f4740) from PID 163 ***]

Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics
Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics
[2024-05-20 01:06:18,200] [    INFO] distributed_strategy.py:214 - distributed strategy initialized
======================= Modified FLAGS detected =======================
FLAGS(name='FLAGS_selected_gpus', current_value='0', default_value='')
=======================================================================
I0520 01:06:18.201602   212 tcp_utils.cc:181] The server starts to listen on IP_ANY:45378
I0520 01:06:18.201762   212 tcp_utils.cc:130] Successfully connected to 172.19.2.2:45378
I0520 01:06:21.322988   212 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000
[2024-05-20 01:06:21,323] [    INFO] topology.py:358 - Total 2 pipe comm group(s) create successfully!
W0520 01:06:21.324384   212 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 7.5, Driver API Version: 12.2, Runtime API Version: 11.8
W0520 01:06:21.326066   212 gpu_resources.cc:164] device: 0, cuDNN Version: 8.9.
I0520 01:06:21.454342   212 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000
[2024-05-20 01:06:21,454] [    INFO] topology.py:358 - Total 1 data comm group(s) create successfully!
[2024-05-20 01:06:21,454] [    INFO] topology.py:358 - Total 2 model comm group(s) create successfully!
[2024-05-20 01:06:21,454] [    INFO] topology.py:358 - Total 2 sharding comm group(s) create successfully!
I0520 01:06:21.454846   212 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000
[2024-05-20 01:06:21,454] [    INFO] topology.py:288 - HybridParallelInfo: rank_id: 0, mp_degree: 1, sharding_degree: 1, pp_degree: 1, dp_degree: 2, sep_degree: 1, mp_group: [0],  sharding_group: [0], pp_group: [0], dp_group: [0, 1], sep:group: None, check/clip group: [0]
loading annotations into memory...
Done (t=2.11s)
creating index...
index created!
[05/20 01:06:23] ppdet.data.source.coco WARNING: Found an invalid bbox in annotations: im_id: 201, area: 0.0 x1: 611, y1: 158, x2: 615, y2: 158.
[05/20 01:06:28] ppdet.data.source.coco INFO: Load [6471 samples valid, 0 samples invalid] in file datasets/VisDrone/annotations_VisDrone_train.json.
wandb: Currently logged in as: thanhtuit96 (thanhtuit). Use `wandb login --relogin` to force relogin


--------------------------------------
C++ Traceback (most recent call last):
--------------------------------------
No stack trace in paddle, may be caused by external reasons.

----------------------
Error Message Summary:
----------------------
FatalError: `Termination signal` is detected by the operating system.
  [TimeInfo: *** Aborted at 1716167199 (unix time) try "date -d @1716167199" if you are using GNU date ***]
  [SignalInfo: *** SIGTERM (@0xc6) received by PID 212 (TID 0x7b2abb0a7740) from PID 198 ***]

Warning: Unable to use MOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics
Warning: Unable to use MCMOT metric, please install motmetrics, for example: `pip install motmetrics`, see https://github.com/longcw/py-motmetrics
[2024-05-20 01:07:01,878] [    INFO] distributed_strategy.py:214 - distributed strategy initialized
======================= Modified FLAGS detected =======================
FLAGS(name='FLAGS_selected_gpus', current_value='0', default_value='')
=======================================================================
I0520 01:07:01.880188   362 tcp_utils.cc:181] The server starts to listen on IP_ANY:36395
I0520 01:07:01.880379   362 tcp_utils.cc:130] Successfully connected to 172.19.2.2:36395
I0520 01:07:01.961936   362 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000
[2024-05-20 01:07:01,962] [    INFO] topology.py:358 - Total 2 pipe comm group(s) create successfully!
W0520 01:07:01.966106   362 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 7.5, Driver API Version: 12.2, Runtime API Version: 11.8
W0520 01:07:01.967388   362 gpu_resources.cc:164] device: 0, cuDNN Version: 8.9.
I0520 01:07:02.126286   362 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000
[2024-05-20 01:07:02,126] [    INFO] topology.py:358 - Total 1 data comm group(s) create successfully!
[2024-05-20 01:07:02,126] [    INFO] topology.py:358 - Total 2 model comm group(s) create successfully!
[2024-05-20 01:07:02,126] [    INFO] topology.py:358 - Total 2 sharding comm group(s) create successfully!
I0520 01:07:02.126700   362 process_group_nccl.cc:129] ProcessGroupNCCL pg_timeout_ 1800000
[2024-05-20 01:07:02,126] [    INFO] topology.py:288 - HybridParallelInfo: rank_id: 0, mp_degree: 1, sharding_degree: 1, pp_degree: 1, dp_degree: 2, sep_degree: 1, mp_group: [0],  sharding_group: [0], pp_group: [0], dp_group: [0, 1], sep:group: None, check/clip group: [0]
loading annotations into memory...
Done (t=1.99s)
creating index...
index created!
[05/20 01:07:04] ppdet.data.source.coco WARNING: Found an invalid bbox in annotations: im_id: 201, area: 0.0 x1: 611, y1: 158, x2: 615, y2: 158.
[05/20 01:07:09] ppdet.data.source.coco INFO: Load [6471 samples valid, 0 samples invalid] in file datasets/VisDrone/annotations_VisDrone_train.json.
wandb: Currently logged in as: thanhtuit96 (thanhtuit). Use `wandb login --relogin` to force relogin
wandb: wandb version 0.17.0 is available!  To upgrade, please run:
wandb:  $ pip install wandb --upgrade
wandb: Tracking run with wandb version 0.16.6
wandb: Run data is saved locally in /kaggle/working/ObjectDetection/DETR/wandb/run-20240520_010717-kwpy1yme
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run daily-dream-3
wandb: ⭐️ View project at https://wandb.ai/thanhtuit/ObjectDetection-DETR_tools
wandb: 🚀 View run at https://wandb.ai/thanhtuit/ObjectDetection-DETR_tools/runs/kwpy1yme
[05/20 01:07:33] ppdet.utils.checkpoint INFO: Exchange model and ema_model to load:
[05/20 01:07:33] ppdet.utils.checkpoint INFO: Loading ema_model weights from /kaggle/working/ObjectDetection/DETR/output/rtdetr_hgnetv2_x_6x_coco/21.pdparams
[05/20 01:07:34] ppdet.utils.checkpoint INFO: Loading model weights from /kaggle/working/ObjectDetection/DETR/output/rtdetr_hgnetv2_x_6x_coco/21.pdema
[05/20 01:07:34] ppdet.utils.checkpoint INFO: Finish resuming model weights: /kaggle/working/ObjectDetection/DETR/output/rtdetr_hgnetv2_x_6x_coco/21.pdparams
W0520 01:07:37.870548   362 reducer.cc:721] All parameters are involved in the backward pass. It is recommended to set find_unused_parameters to False to improve performance. However, if unused parameters appear in subsequent iterative training, then an error will occur. Please make it clear that in the subsequent training, there will be no parameters that are not used in the backward pass, and then set find_unused_parameters
[05/20 01:07:39] ppdet.engine INFO: Epoch: [22] [  0/809] learning_rate: 0.000050 loss_class: 0.446477 loss_bbox: 0.065953 loss_giou: 0.940175 loss_class_aux: 2.608018 loss_bbox_aux: 0.413131 loss_giou_aux: 5.876606 loss_class_dn: 0.397860 loss_bbox_dn: 0.029102 loss_giou_dn: 0.730097 loss_class_aux_dn: 2.018472 loss_bbox_aux_dn: 0.155799 loss_giou_aux_dn: 3.747177 loss: 17.428867 eta: 2 days, 19:16:41 batch_cost: 3.8383 data_cost: 0.0013 ips: 1.0421 images/s
[05/20 01:13:05] ppdet.engine INFO: Epoch: [22] [200/809] learning_rate: 0.000050 loss_class: 0.429664 loss_bbox: 0.094655 loss_giou: 1.071689 loss_class_aux: 2.598499 loss_bbox_aux: 0.632869 loss_giou_aux: 6.639613 loss_class_dn: 0.451413 loss_bbox_dn: 0.045535 loss_giou_dn: 0.782645 loss_class_aux_dn: 2.272191 loss_bbox_aux_dn: 0.244173 loss_giou_aux_dn: 4.023020 loss: 19.582922 eta: 1 day, 3:25:50 batch_cost: 1.5586 data_cost: 0.0027 ips: 2.5665 images/s
[05/20 01:18:30] ppdet.engine INFO: Epoch: [22] [400/809] learning_rate: 0.000050 loss_class: 0.404321 loss_bbox: 0.090119 loss_giou: 1.027519 loss_class_aux: 2.440328 loss_bbox_aux: 0.590902 loss_giou_aux: 6.387892 loss_class_dn: 0.433662 loss_bbox_dn: 0.045400 loss_giou_dn: 0.758714 loss_class_aux_dn: 2.195279 loss_bbox_aux_dn: 0.244623 loss_giou_aux_dn: 3.916644 loss: 18.919592 eta: 1 day, 3:11:14 batch_cost: 1.5520 data_cost: 0.0025 ips: 2.5774 images/s
[05/20 01:23:51] ppdet.engine INFO: Epoch: [22] [600/809] learning_rate: 0.000050 loss_class: 0.420328 loss_bbox: 0.096082 loss_giou: 0.985225 loss_class_aux: 2.529186 loss_bbox_aux: 0.621062 loss_giou_aux: 6.092589 loss_class_dn: 0.414886 loss_bbox_dn: 0.045393 loss_giou_dn: 0.725521 loss_class_aux_dn: 2.091394 loss_bbox_aux_dn: 0.243571 loss_giou_aux_dn: 3.724197 loss: 18.096030 eta: 1 day, 2:58:08 batch_cost: 1.5382 data_cost: 0.0026 ips: 2.6005 images/s
[05/20 01:29:14] ppdet.engine INFO: Epoch: [22] [800/809] learning_rate: 0.000050 loss_class: 0.428570 loss_bbox: 0.094058 loss_giou: 1.046203 loss_class_aux: 2.583212 loss_bbox_aux: 0.619184 loss_giou_aux: 6.499679 loss_class_dn: 0.439085 loss_bbox_dn: 0.045976 loss_giou_dn: 0.773637 loss_class_aux_dn: 2.223101 loss_bbox_aux_dn: 0.246341 loss_giou_aux_dn: 3.975849 loss: 19.547108 eta: 1 day, 2:49:18 batch_cost: 1.5392 data_cost: 0.0030 ips: 2.5987 images/s
[05/20 01:29:32] ppdet.engine INFO: Epoch: [23] [  0/809] learning_rate: 0.000050 loss_class: 0.438351 loss_bbox: 0.095685 loss_giou: 1.060056 loss_class_aux: 2.612251 loss_bbox_aux: 0.621058 loss_giou_aux: 6.575186 loss_class_dn: 0.442999 loss_bbox_dn: 0.045976 loss_giou_dn: 0.777679 loss_class_aux_dn: 2.243878 loss_bbox_aux_dn: 0.247875 loss_giou_aux_dn: 4.013665 loss: 19.686118 eta: 1 day, 2:53:29 batch_cost: 1.5588 data_cost: 0.0293 ips: 2.5661 images/s
[05/20 01:34:54] ppdet.engine INFO: Epoch: [23] [200/809] learning_rate: 0.000050 loss_class: 0.448456 loss_bbox: 0.089951 loss_giou: 0.993556 loss_class_aux: 2.711142 loss_bbox_aux: 0.592143 loss_giou_aux: 6.161139 loss_class_dn: 0.439973 loss_bbox_dn: 0.047723 loss_giou_dn: 0.773649 loss_class_aux_dn: 2.228480 loss_bbox_aux_dn: 0.255004 loss_giou_aux_dn: 4.014326 loss: 19.260250 eta: 1 day, 2:45:06 batch_cost: 1.5385 data_cost: 0.0033 ips: 2.6000 images/s
[05/20 01:40:26] ppdet.engine INFO: Epoch: [23] [400/809] learning_rate: 0.000050 loss_class: 0.438013 loss_bbox: 0.102586 loss_giou: 1.084841 loss_class_aux: 2.610192 loss_bbox_aux: 0.683048 loss_giou_aux: 6.741956 loss_class_dn: 0.447043 loss_bbox_dn: 0.046622 loss_giou_dn: 0.776518 loss_class_aux_dn: 2.264635 loss_bbox_aux_dn: 0.245076 loss_giou_aux_dn: 3.979445 loss: 19.947770 eta: 1 day, 2:46:13 batch_cost: 1.5878 data_cost: 0.0031 ips: 2.5191 images/s
[05/20 01:45:49] ppdet.engine INFO: Epoch: [23] [600/809] learning_rate: 0.000050 loss_class: 0.408636 loss_bbox: 0.098624 loss_giou: 1.059442 loss_class_aux: 2.469717 loss_bbox_aux: 0.636532 loss_giou_aux: 6.563957 loss_class_dn: 0.453998 loss_bbox_dn: 0.043167 loss_giou_dn: 0.794411 loss_class_aux_dn: 2.269770 loss_bbox_aux_dn: 0.230240 loss_giou_aux_dn: 4.098908 loss: 19.680844 eta: 1 day, 2:39:18 batch_cost: 1.5453 data_cost: 0.0029 ips: 2.5886 images/s
[05/20 01:51:02] ppdet.engine INFO: Epoch: [23] [800/809] learning_rate: 0.000050 loss_class: 0.420670 loss_bbox: 0.101640 loss_giou: 1.000911 loss_class_aux: 2.489718 loss_bbox_aux: 0.661662 loss_giou_aux: 6.233522 loss_class_dn: 0.438699 loss_bbox_dn: 0.047968 loss_giou_dn: 0.754076 loss_class_aux_dn: 2.215141 loss_bbox_aux_dn: 0.256399 loss_giou_aux_dn: 3.876837 loss: 19.287801 eta: 1 day, 2:26:35 batch_cost: 1.4963 data_cost: 0.0025 ips: 2.6732 images/s
[05/20 01:51:25] ppdet.utils.checkpoint INFO: Save checkpoint: output/rtdetr_hgnetv2_x_6x_coco
loading annotations into memory...
Done (t=0.15s)
creating index...
index created!
[05/20 01:51:27] ppdet.data.source.coco INFO: Load [548 samples valid, 0 samples invalid] in file datasets/VisDrone/annotations_VisDrone_val.json.
[05/20 01:51:30] ppdet.engine INFO: Eval iter: 0
[05/20 01:52:33] ppdet.metrics.metrics INFO: The bbox result is saved to bbox.json.
loading annotations into memory...
Done (t=0.14s)
creating index...
index created!
[05/20 01:52:33] ppdet.metrics.coco_utils INFO: Start evaluate...
Loading and preparing results...
DONE (t=2.12s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=92.60s).
Accumulating evaluation results...
DONE (t=3.05s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.201
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.353
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.194
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.117
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.301
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.427
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.098
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.267
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.356
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.252
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.503
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.652
[05/20 01:54:11] ppdet.engine INFO: Total sample number: 548, average FPS: 9.057458823565387
[05/20 01:54:11] ppdet.engine INFO: Best test bbox ap is 0.201.
[05/20 01:54:16] ppdet.utils.checkpoint INFO: Save checkpoint: output/rtdetr_hgnetv2_x_6x_coco
[05/20 01:54:20] ppdet.engine INFO: Epoch: [24] [  0/809] learning_rate: 0.000050 loss_class: 0.421911 loss_bbox: 0.096516 loss_giou: 0.984546 loss_class_aux: 2.507176 loss_bbox_aux: 0.654192 loss_giou_aux: 6.150344 loss_class_dn: 0.431456 loss_bbox_dn: 0.047713 loss_giou_dn: 0.738790 loss_class_aux_dn: 2.186306 loss_bbox_aux_dn: 0.256399 loss_giou_aux_dn: 3.827082 loss: 19.033889 eta: 1 day, 2:26:09 batch_cost: 1.4967 data_cost: 0.0025 ips: 2.6726 images/s
[05/20 01:59:42] ppdet.engine INFO: Epoch: [24] [200/809] learning_rate: 0.000050 loss_class: 0.432100 loss_bbox: 0.099944 loss_giou: 1.039651 loss_class_aux: 2.617023 loss_bbox_aux: 0.658402 loss_giou_aux: 6.486967 loss_class_dn: 0.440116 loss_bbox_dn: 0.045505 loss_giou_dn: 0.758473 loss_class_aux_dn: 2.236161 loss_bbox_aux_dn: 0.247869 loss_giou_aux_dn: 3.911072 loss: 19.377412 eta: 1 day, 2:19:51 batch_cost: 1.5378 data_cost: 0.0033 ips: 2.6012 images/s
[05/20 02:05:06] ppdet.engine INFO: Epoch: [24] [400/809] learning_rate: 0.000050 loss_class: 0.405096 loss_bbox: 0.103687 loss_giou: 1.065689 loss_class_aux: 2.410601 loss_bbox_aux: 0.700636 loss_giou_aux: 6.618891 loss_class_dn: 0.442195 loss_bbox_dn: 0.044589 loss_giou_dn: 0.794662 loss_class_aux_dn: 2.225750 loss_bbox_aux_dn: 0.237968 loss_giou_aux_dn: 4.117761 loss: 19.472183 eta: 1 day, 2:14:33 batch_cost: 1.5453 data_cost: 0.0037 ips: 2.5884 images/s
[05/20 02:10:35] ppdet.engine INFO: Epoch: [24] [600/809] learning_rate: 0.000050 loss_class: 0.408509 loss_bbox: 0.106035 loss_giou: 1.130732 loss_class_aux: 2.474089 loss_bbox_aux: 0.706209 loss_giou_aux: 6.970690 loss_class_dn: 0.430122 loss_bbox_dn: 0.044908 loss_giou_dn: 0.788633 loss_class_aux_dn: 2.174001 loss_bbox_aux_dn: 0.241166 loss_giou_aux_dn: 4.069491 loss: 19.907720 eta: 1 day, 2:11:29 batch_cost: 1.5694 data_cost: 0.0034 ips: 2.5488 images/s
[05/20 02:15:56] ppdet.engine INFO: Epoch: [24] [800/809] learning_rate: 0.000050 loss_class: 0.416026 loss_bbox: 0.095421 loss_giou: 1.084998 loss_class_aux: 2.481690 loss_bbox_aux: 0.627860 loss_giou_aux: 6.755666 loss_class_dn: 0.451382 loss_bbox_dn: 0.045890 loss_giou_dn: 0.795073 loss_class_aux_dn: 2.275539 loss_bbox_aux_dn: 0.246767 loss_giou_aux_dn: 4.092684 loss: 19.865692 eta: 1 day, 2:04:56 batch_cost: 1.5321 data_cost: 0.0034 ips: 2.6108 images/s
[05/20 02:16:16] ppdet.engine INFO: Epoch: [25] [  0/809] learning_rate: 0.000050 loss_class: 0.420622 loss_bbox: 0.093712 loss_giou: 1.084998 loss_class_aux: 2.490678 loss_bbox_aux: 0.617911 loss_giou_aux: 6.755666 loss_class_dn: 0.451382 loss_bbox_dn: 0.046875 loss_giou_dn: 0.798392 loss_class_aux_dn: 2.275477 loss_bbox_aux_dn: 0.251645 loss_giou_aux_dn: 4.110934 loss: 19.865692 eta: 1 day, 2:06:39 batch_cost: 1.5659 data_cost: 0.0150 ips: 2.5545 images/s
[05/20 02:21:39] ppdet.engine INFO: Epoch: [25] [200/809] learning_rate: 0.000050 loss_class: 0.447892 loss_bbox: 0.097573 loss_giou: 1.016019 loss_class_aux: 2.675176 loss_bbox_aux: 0.629812 loss_giou_aux: 6.421303 loss_class_dn: 0.443949 loss_bbox_dn: 0.047494 loss_giou_dn: 0.762737 loss_class_aux_dn: 2.250313 loss_bbox_aux_dn: 0.257079 loss_giou_aux_dn: 3.924872 loss: 19.315125 eta: 1 day, 2:00:41 batch_cost: 1.5386 data_cost: 0.0034 ips: 2.5998 images/s
[05/20 02:26:56] ppdet.engine INFO: Epoch: [25] [400/809] learning_rate: 0.000050 loss_class: 0.436080 loss_bbox: 0.088141 loss_giou: 1.005762 loss_class_aux: 2.636475 loss_bbox_aux: 0.583201 loss_giou_aux: 6.273939 loss_class_dn: 0.435520 loss_bbox_dn: 0.045516 loss_giou_dn: 0.719083 loss_class_aux_dn: 2.202116 loss_bbox_aux_dn: 0.239283 loss_giou_aux_dn: 3.726342 loss: 18.493905 eta: 1 day, 1:53:35 batch_cost: 1.5211 data_cost: 0.0029 ips: 2.6296 images/s
[05/20 02:32:14] ppdet.engine INFO: Epoch: [25] [600/809] learning_rate: 0.000050 loss_class: 0.419062 loss_bbox: 0.104297 loss_giou: 1.140784 loss_class_aux: 2.480239 loss_bbox_aux: 0.694249 loss_giou_aux: 7.181141 loss_class_dn: 0.431588 loss_bbox_dn: 0.043284 loss_giou_dn: 0.790712 loss_class_aux_dn: 2.204810 loss_bbox_aux_dn: 0.230885 loss_giou_aux_dn: 4.037150 loss: 19.922297 eta: 1 day, 1:46:30 batch_cost: 1.5174 data_cost: 0.0033 ips: 2.6361 images/s
[05/20 02:37:36] ppdet.engine INFO: Epoch: [25] [800/809] learning_rate: 0.000050 loss_class: 0.422725 loss_bbox: 0.092268 loss_giou: 1.030645 loss_class_aux: 2.529808 loss_bbox_aux: 0.604828 loss_giou_aux: 6.422277 loss_class_dn: 0.442596 loss_bbox_dn: 0.048461 loss_giou_dn: 0.759510 loss_class_aux_dn: 2.237485 loss_bbox_aux_dn: 0.259714 loss_giou_aux_dn: 3.912983 loss: 19.075337 eta: 1 day, 1:40:53 batch_cost: 1.5370 data_cost: 0.0034 ips: 2.6024 images/s
[05/20 02:37:59] ppdet.utils.checkpoint INFO: Save checkpoint: output/rtdetr_hgnetv2_x_6x_coco
[05/20 02:38:01] ppdet.engine INFO: Eval iter: 0
[05/20 02:39:04] ppdet.metrics.metrics INFO: The bbox result is saved to bbox.json.
loading annotations into memory...
Done (t=0.35s)
creating index...
index created!
[05/20 02:39:05] ppdet.metrics.coco_utils INFO: Start evaluate...
Loading and preparing results...
DONE (t=1.75s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=89.34s).
Accumulating evaluation results...
DONE (t=3.10s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.204
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.358
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.197
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.120
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.305
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.437
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.100
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.269
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.360
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.257
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.505
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.666
[05/20 02:40:39] ppdet.engine INFO: Total sample number: 548, average FPS: 9.341785756306578
[05/20 02:40:39] ppdet.engine INFO: Best test bbox ap is 0.204.
[05/20 02:40:46] ppdet.utils.checkpoint INFO: Save checkpoint: output/rtdetr_hgnetv2_x_6x_coco
[05/20 02:40:49] ppdet.engine INFO: Epoch: [26] [  0/809] learning_rate: 0.000050 loss_class: 0.422649 loss_bbox: 0.092268 loss_giou: 1.030645 loss_class_aux: 2.519339 loss_bbox_aux: 0.603741 loss_giou_aux: 6.422277 loss_class_dn: 0.440762 loss_bbox_dn: 0.048913 loss_giou_dn: 0.754095 loss_class_aux_dn: 2.223611 loss_bbox_aux_dn: 0.260096 loss_giou_aux_dn: 3.906237 loss: 19.051765 eta: 1 day, 1:40:23 batch_cost: 1.5381 data_cost: 0.0033 ips: 2.6006 images/s
[05/20 02:46:12] ppdet.engine INFO: Epoch: [26] [200/809] learning_rate: 0.000050 loss_class: 0.398639 loss_bbox: 0.084247 loss_giou: 0.989320 loss_class_aux: 2.400950 loss_bbox_aux: 0.557671 loss_giou_aux: 6.041494 loss_class_dn: 0.416189 loss_bbox_dn: 0.042604 loss_giou_dn: 0.728332 loss_class_aux_dn: 2.105899 loss_bbox_aux_dn: 0.230175 loss_giou_aux_dn: 3.741843 loss: 18.169376 eta: 1 day, 1:35:02 batch_cost: 1.5404 data_cost: 0.0031 ips: 2.5968 images/s
[05/20 02:51:33] ppdet.engine INFO: Epoch: [26] [400/809] learning_rate: 0.000050 loss_class: 0.429308 loss_bbox: 0.093192 loss_giou: 1.043643 loss_class_aux: 2.606146 loss_bbox_aux: 0.610051 loss_giou_aux: 6.404592 loss_class_dn: 0.441386 loss_bbox_dn: 0.043499 loss_giou_dn: 0.750361 loss_class_aux_dn: 2.257509 loss_bbox_aux_dn: 0.233638 loss_giou_aux_dn: 3.882820 loss: 18.736143 eta: 1 day, 1:29:34 batch_cost: 1.5377 data_cost: 0.0034 ips: 2.6013 images/s
[05/20 02:56:54] ppdet.engine INFO: Epoch: [26] [600/809] learning_rate: 0.000050 loss_class: 0.423373 loss_bbox: 0.095576 loss_giou: 1.046554 loss_class_aux: 2.570421 loss_bbox_aux: 0.638840 loss_giou_aux: 6.597026 loss_class_dn: 0.447536 loss_bbox_dn: 0.045395 loss_giou_dn: 0.771885 loss_class_aux_dn: 2.266781 loss_bbox_aux_dn: 0.241593 loss_giou_aux_dn: 3.992731 loss: 19.512135 eta: 1 day, 1:24:02 batch_cost: 1.5356 data_cost: 0.0031 ips: 2.6048 images/s
[05/20 03:02:16] ppdet.engine INFO: Epoch: [26] [800/809] learning_rate: 0.000050 loss_class: 0.460945 loss_bbox: 0.099489 loss_giou: 1.079220 loss_class_aux: 2.792340 loss_bbox_aux: 0.659646 loss_giou_aux: 6.664953 loss_class_dn: 0.445151 loss_bbox_dn: 0.046101 loss_giou_dn: 0.785122 loss_class_aux_dn: 2.270517 loss_bbox_aux_dn: 0.249150 loss_giou_aux_dn: 4.018325 loss: 20.236510 eta: 1 day, 1:18:46 batch_cost: 1.5406 data_cost: 0.0034 ips: 2.5964 images/s
[05/20 03:02:34] ppdet.engine INFO: Epoch: [27] [  0/809] learning_rate: 0.000050 loss_class: 0.460037 loss_bbox: 0.095382 loss_giou: 1.075235 loss_class_aux: 2.789576 loss_bbox_aux: 0.638182 loss_giou_aux: 6.627815 loss_class_dn: 0.444551 loss_bbox_dn: 0.046513 loss_giou_dn: 0.783770 loss_class_aux_dn: 2.256770 loss_bbox_aux_dn: 0.249982 loss_giou_aux_dn: 4.016215 loss: 19.416531 eta: 1 day, 1:19:10 batch_cost: 1.5523 data_cost: 0.0126 ips: 2.5768 images/s
[05/20 03:07:59] ppdet.engine INFO: Epoch: [27] [200/809] learning_rate: 0.000050 loss_class: 0.422255 loss_bbox: 0.093790 loss_giou: 1.074865 loss_class_aux: 2.524622 loss_bbox_aux: 0.624999 loss_giou_aux: 6.759279 loss_class_dn: 0.439193 loss_bbox_dn: 0.044145 loss_giou_dn: 0.785396 loss_class_aux_dn: 2.223360 loss_bbox_aux_dn: 0.235044 loss_giou_aux_dn: 4.045792 loss: 19.369226 eta: 1 day, 1:14:18 batch_cost: 1.5493 data_cost: 0.0074 ips: 2.5817 images/s
[05/20 03:13:23] ppdet.engine INFO: Epoch: [27] [400/809] learning_rate: 0.000050 loss_class: 0.410127 loss_bbox: 0.102794 loss_giou: 1.071961 loss_class_aux: 2.460820 loss_bbox_aux: 0.699997 loss_giou_aux: 6.685116 loss_class_dn: 0.438249 loss_bbox_dn: 0.045365 loss_giou_dn: 0.760312 loss_class_aux_dn: 2.232153 loss_bbox_aux_dn: 0.245794 loss_giou_aux_dn: 3.972182 loss: 19.504269 eta: 1 day, 1:09:22 batch_cost: 1.5486 data_cost: 0.0031 ips: 2.5830 images/s
[05/20 03:18:53] ppdet.engine INFO: Epoch: [27] [600/809] learning_rate: 0.000050 loss_class: 0.412483 loss_bbox: 0.088479 loss_giou: 1.005398 loss_class_aux: 2.483102 loss_bbox_aux: 0.587250 loss_giou_aux: 6.348157 loss_class_dn: 0.423629 loss_bbox_dn: 0.044325 loss_giou_dn: 0.744445 loss_class_aux_dn: 2.137327 loss_bbox_aux_dn: 0.235497 loss_giou_aux_dn: 3.811221 loss: 18.352384 eta: 1 day, 1:05:32 batch_cost: 1.5753 data_cost: 0.0032 ips: 2.5392 images/s
[05/20 03:24:19] ppdet.engine INFO: Epoch: [27] [800/809] learning_rate: 0.000050 loss_class: 0.442641 loss_bbox: 0.095203 loss_giou: 1.106010 loss_class_aux: 2.687094 loss_bbox_aux: 0.634511 loss_giou_aux: 6.906728 loss_class_dn: 0.477181 loss_bbox_dn: 0.044863 loss_giou_dn: 0.797152 loss_class_aux_dn: 2.416907 loss_bbox_aux_dn: 0.237408 loss_giou_aux_dn: 4.092210 loss: 20.227622 eta: 1 day, 1:01:04 batch_cost: 1.5621 data_cost: 0.0035 ips: 2.5606 images/s
[05/20 03:24:42] ppdet.utils.checkpoint INFO: Save checkpoint: output/rtdetr_hgnetv2_x_6x_coco
[05/20 03:24:44] ppdet.engine INFO: Eval iter: 0
[05/20 03:25:46] ppdet.metrics.metrics INFO: The bbox result is saved to bbox.json.
loading annotations into memory...
Done (t=0.39s)
creating index...
index created!
[05/20 03:25:47] ppdet.metrics.coco_utils INFO: Start evaluate...
Loading and preparing results...
DONE (t=1.95s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=90.82s).
Accumulating evaluation results...
DONE (t=3.09s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.207
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.362
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.201
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.121
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.308
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.443
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.102
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.272
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.362
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.259
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.507
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.655
[05/20 03:27:23] ppdet.engine INFO: Total sample number: 548, average FPS: 9.547780867777037
[05/20 03:27:23] ppdet.engine INFO: Best test bbox ap is 0.207.
[05/20 03:27:27] ppdet.utils.checkpoint INFO: Save checkpoint: output/rtdetr_hgnetv2_x_6x_coco
[05/20 03:27:30] ppdet.engine INFO: Epoch: [28] [  0/809] learning_rate: 0.000050 loss_class: 0.445168 loss_bbox: 0.096459 loss_giou: 1.106010 loss_class_aux: 2.691352 loss_bbox_aux: 0.645278 loss_giou_aux: 6.906728 loss_class_dn: 0.480670 loss_bbox_dn: 0.045386 loss_giou_dn: 0.801215 loss_class_aux_dn: 2.423730 loss_bbox_aux_dn: 0.240282 loss_giou_aux_dn: 4.114364 loss: 20.363426 eta: 1 day, 1:00:32 batch_cost: 1.5573 data_cost: 0.0034 ips: 2.5686 images/s
[05/20 03:32:55] ppdet.engine INFO: Epoch: [28] [200/809] learning_rate: 0.000050 loss_class: 0.417849 loss_bbox: 0.099040 loss_giou: 1.074988 loss_class_aux: 2.529439 loss_bbox_aux: 0.636302 loss_giou_aux: 6.706931 loss_class_dn: 0.441951 loss_bbox_dn: 0.044839 loss_giou_dn: 0.767254 loss_class_aux_dn: 2.226955 loss_bbox_aux_dn: 0.240002 loss_giou_aux_dn: 3.932291 loss: 19.174661 eta: 1 day, 0:55:35 batch_cost: 1.5508 data_cost: 0.0031 ips: 2.5793 images/s
[05/20 03:38:17] ppdet.engine INFO: Epoch: [28] [400/809] learning_rate: 0.000050 loss_class: 0.441212 loss_bbox: 0.089940 loss_giou: 0.991819 loss_class_aux: 2.662115 loss_bbox_aux: 0.590811 loss_giou_aux: 6.139363 loss_class_dn: 0.455001 loss_bbox_dn: 0.045923 loss_giou_dn: 0.746032 loss_class_aux_dn: 2.282391 loss_bbox_aux_dn: 0.243271 loss_giou_aux_dn: 3.871369 loss: 19.236920 eta: 1 day, 0:50:05 batch_cost: 1.5367 data_cost: 0.0031 ips: 2.6031 images/s
[05/20 03:43:42] ppdet.engine INFO: Epoch: [28] [600/809] learning_rate: 0.000050 loss_class: 0.405407 loss_bbox: 0.093424 loss_giou: 1.029349 loss_class_aux: 2.450657 loss_bbox_aux: 0.596528 loss_giou_aux: 6.351947 loss_class_dn: 0.436078 loss_bbox_dn: 0.043040 loss_giou_dn: 0.741872 loss_class_aux_dn: 2.208496 loss_bbox_aux_dn: 0.231316 loss_giou_aux_dn: 3.834755 loss: 18.901896 eta: 1 day, 0:44:57 batch_cost: 1.5462 data_cost: 0.0033 ips: 2.5871 images/s
[05/20 03:49:05] ppdet.engine INFO: Epoch: [28] [800/809] learning_rate: 0.000050 loss_class: 0.407842 loss_bbox: 0.103826 loss_giou: 1.109148 loss_class_aux: 2.487351 loss_bbox_aux: 0.666483 loss_giou_aux: 6.962270 loss_class_dn: 0.452334 loss_bbox_dn: 0.045428 loss_giou_dn: 0.806462 loss_class_aux_dn: 2.308534 loss_bbox_aux_dn: 0.238884 loss_giou_aux_dn: 4.136123 loss: 20.150183 eta: 1 day, 0:39:25 batch_cost: 1.5340 data_cost: 0.0039 ips: 2.6076 images/s
[05/20 03:49:25] ppdet.engine INFO: Epoch: [29] [  0/809] learning_rate: 0.000050 loss_class: 0.407842 loss_bbox: 0.101948 loss_giou: 1.089732 loss_class_aux: 2.487351 loss_bbox_aux: 0.634806 loss_giou_aux: 6.816726 loss_class_dn: 0.449145 loss_bbox_dn: 0.044700 loss_giou_dn: 0.801095 loss_class_aux_dn: 2.290331 loss_bbox_aux_dn: 0.237711 loss_giou_aux_dn: 4.081280 loss: 20.007434 eta: 1 day, 0:40:04 batch_cost: 1.5632 data_cost: 0.0257 ips: 2.5589 images/s
[05/20 03:54:47] ppdet.engine INFO: Epoch: [29] [200/809] learning_rate: 0.000050 loss_class: 0.418533 loss_bbox: 0.093623 loss_giou: 1.036756 loss_class_aux: 2.524105 loss_bbox_aux: 0.631502 loss_giou_aux: 6.471950 loss_class_dn: 0.449492 loss_bbox_dn: 0.044268 loss_giou_dn: 0.769049 loss_class_aux_dn: 2.273319 loss_bbox_aux_dn: 0.236426 loss_giou_aux_dn: 3.949852 loss: 19.383165 eta: 1 day, 0:34:30 batch_cost: 1.5337 data_cost: 0.0035 ips: 2.6081 images/s
[05/20 04:00:10] ppdet.engine INFO: Epoch: [29] [400/809] learning_rate: 0.000050 loss_class: 0.418387 loss_bbox: 0.093137 loss_giou: 1.042843 loss_class_aux: 2.516371 loss_bbox_aux: 0.630432 loss_giou_aux: 6.505157 loss_class_dn: 0.436962 loss_bbox_dn: 0.041422 loss_giou_dn: 0.753281 loss_class_aux_dn: 2.224197 loss_bbox_aux_dn: 0.224101 loss_giou_aux_dn: 3.909313 loss: 19.064836 eta: 1 day, 0:29:18 batch_cost: 1.5438 data_cost: 0.0032 ips: 2.5910 images/s
[05/20 04:05:31] ppdet.engine INFO: Epoch: [29] [600/809] learning_rate: 0.000050 loss_class: 0.430081 loss_bbox: 0.096337 loss_giou: 1.028504 loss_class_aux: 2.598295 loss_bbox_aux: 0.630192 loss_giou_aux: 6.326237 loss_class_dn: 0.446232 loss_bbox_dn: 0.045271 loss_giou_dn: 0.746718 loss_class_aux_dn: 2.244449 loss_bbox_aux_dn: 0.243143 loss_giou_aux_dn: 3.839064 loss: 19.112411 eta: 1 day, 0:23:51 batch_cost: 1.5357 data_cost: 0.0030 ips: 2.6046 images/s
[05/20 04:10:53] ppdet.engine INFO: Epoch: [29] [800/809] learning_rate: 0.000050 loss_class: 0.420406 loss_bbox: 0.091793 loss_giou: 1.043133 loss_class_aux: 2.540378 loss_bbox_aux: 0.589786 loss_giou_aux: 6.459545 loss_class_dn: 0.431390 loss_bbox_dn: 0.044942 loss_giou_dn: 0.736633 loss_class_aux_dn: 2.182666 loss_bbox_aux_dn: 0.240383 loss_giou_aux_dn: 3.782522 loss: 18.534068 eta: 1 day, 0:18:27 batch_cost: 1.5370 data_cost: 0.0036 ips: 2.6024 images/s
Traceback (most recent call last):
  File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 183, in <module>
    main()
  File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 179, in main
    run(FLAGS, cfg)
  File "/kaggle/working/ObjectDetection/DETR/tools/train.py", line 135, in run
    trainer.train(FLAGS.eval)
  File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/trainer.py", line 404, in train
    self._compose_callback.on_epoch_end(self.status)
  File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/callbacks.py", line 86, in on_epoch_end
    c.on_epoch_end(status)
  File "/kaggle/working/ObjectDetection/DETR/ppdet/engine/callbacks.py", line 216, in on_epoch_end
    save_model(
  File "/kaggle/working/ObjectDetection/DETR/ppdet/utils/checkpoint.py", line 324, in save_model
    paddle.save(state_dict, save_path + ".pdopt")
  File "/root/.local/lib/python3.10/site-packages/paddle/framework/io.py", line 902, in save
    _pickle_save(obj, f, protocol)
  File "/root/.local/lib/python3.10/site-packages/paddle/framework/io.py", line 428, in _pickle_save
    pickler.dump(obj)
OSError: [Errno 28] No space left on device
I0520 04:11:19.231618   362 process_group_nccl.cc:132] ProcessGroupNCCL destruct 
I0520 04:11:19.231832   362 process_group_nccl.cc:132] ProcessGroupNCCL destruct 
I0520 04:11:19.231868   362 process_group_nccl.cc:132] ProcessGroupNCCL destruct 
I0520 04:11:19.707178   388 tcp_store.cc:289] receive shutdown event and so quit from MasterDaemon run loop