init

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2020 Jie Lei
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,3 +1,38 @@
----
-license: mit
----
+# XML_RVMR
+
+This repository contains the XML model for the baseline of the Ranked Video Moment Retrieval (RVMR) task. The associated paper is titled "Video Moment Retrieval in Practical Setting: A Dataset of Ranked Moments for Imprecise Queries."
+
+The main repository of the paper is [TVR-Ranking](https://huggingface.co/axgroup/TVR-Ranking), and this model is adapted from [TVRetrieval](https://github.com/jayleicn/TVRetrieval.git).
+
+Annotations and features can be downloaded from [TVR-Ranking](https://huggingface.co/axgroup/TVR-Ranking). The environment setup is the same as for RelocNet_RVMR, as detailed in the [TVR-Ranking](https://huggingface.co/axgroup/TVR-Ranking) repository.
+
+
+## Performance
+
+| **Model** | **Train Set Top N** | **IoU=0.3**  | **IoU=0.5**  | **IoU=0.7**  |
+|-----------|---------------------|--------------|--------------|--------------|
+|           |                     | **Val** | **Test** | **Val** | **Test** | **Val** | **Test** |
+| **NDCG@10** |                     |              |              |              |
+| XML       | 1                   | 0.1016 | 0.0917 | 0.0747 | 0.0660 | 0.0244 | 0.0268 |
+| XML       | 20                  | 0.2226 | 0.2135 | 0.1623 | 0.1567 | 0.0580 | 0.0627 |
+| XML       | 40                  | 0.2002 | 0.2044 | 0.1461 | 0.1502 | 0.0541 | 0.0589 |
+| **NDCG@20** |                     |              |              |              |
+| XML       | 1                   | 0.1010 | 0.0923 | 0.0737 | 0.0662 | 0.0258 | 0.0269 |
+| XML       | 20                  | 0.2331 | 0.2243 | 0.1700 | 0.1650 | 0.0627 | 0.0664 |
+| XML       | 40                  | 0.2114 | 0.2167 | 0.1530 | 0.1590 | 0.0583 | 0.0635 |
+| **NDCG@40** |                     |              |              |              |
+| XML       | 1                   | 0.1077 | 0.1016 | 0.0775 | 0.0727 | 0.0273 | 0.0294 |
+| XML       | 20                  | 0.2580 | 0.2512 | 0.1874 | 0.1853 | 0.0705 | 0.0753 |
+| XML       | 40                  | 0.2408 | 0.2432 | 0.1740 | 0.1791 | 0.0666 | 0.0720 |
+
+
+
+## Quick Start
+
+Modify the path in `run_top20.sh` and then execute the script:
+
+```sh
+sh run_top20.sh
+```
+
+Feel free to contribute or raise issues for any problems encountered.

run_top01.sh

@@ -0,0 +1,25 @@
+python baselines/crossmodal_moment_localization/train.py \
+   --train_path      data/TVR_Ranking/train_top01.json \
+   --val_path        data/TVR_Ranking/val.json \
+   --test_path       data/TVR_Ranking/test.json \
+   --corpus_path     data/TVR_Ranking/video_corpus.json \
+   --desc_bert_path  data/features/query_bert.h5 \
+   --vid_feat_path   data/features/tvr_i3d_rgb600_avg_cl-1.5.h5 \
+   --sub_bert_path   data/features/tvr_sub_pretrained_w_sub_query_max_cl-1.5.h5\
+   --dset_name=tvr \
+   --eval_split_name=val \
+   --nms_thd=-1 \
+   --results_root=results \
+   --clip_length=1.5 \
+   --vid_feat_size=1024 \
+   --ctx_mode=video_sub_tef \
+   --max_ctx_l=128 \
+   --max_pred_l=16 \
+   --eval_num_per_epoch=0.05 \
+   --n_epoch=4000 \
+   --exp_id=top01 \
+   --model_name=XML \
+   --lr=0.001
+
+   # qsub -I -l select=1:ngpus=1 -P gs_slab -q gpu8
+   # cd 11_TVR-Ranking/TVRetrieval/; conda activate py11; sh run_top01.sh 

run_top20.sh

@@ -0,0 +1,24 @@
+python baselines/crossmodal_moment_localization/train.py \
+   --train_path      data/TVR_Ranking/train_top20.json \
+   --val_path        data/TVR_Ranking/val.json \
+   --test_path       data/TVR_Ranking/test.json \
+   --corpus_path     data/TVR_Ranking/video_corpus.json \
+   --desc_bert_path  data/features/query_bert.h5 \
+   --vid_feat_path   data/features/tvr_i3d_rgb600_avg_cl-1.5.h5 \
+   --sub_bert_path   data/features/tvr_sub_pretrained_w_sub_query_max_cl-1.5.h5\
+   --dset_name=tvr \
+   --eval_split_name=val \
+   --nms_thd=-1 \
+   --results_root=results \
+   --clip_length=1.5 \
+   --vid_feat_size=1024 \
+   --ctx_mode=video_sub_tef \
+   --max_ctx_l=128 \
+   --max_pred_l=16 \
+   --eval_num_per_epoch=1 \
+   --n_epoch=200 \
+   --exp_id=top20 \
+   --model_name=XML
+
+   # qsub -I -l select=1:ngpus=1 -P gs_slab -q gpu8
+   # cd 11_TVR-Ranking/TVRetrieval/; conda activate py11; sh run_top20.sh 

run_top40.sh

@@ -0,0 +1,24 @@
+python baselines/crossmodal_moment_localization/train.py \
+   --train_path      data/TVR_Ranking/train_top40.json \
+   --val_path        data/TVR_Ranking/val.json \
+   --test_path       data/TVR_Ranking/test.json \
+   --corpus_path     data/TVR_Ranking/video_corpus.json \
+   --desc_bert_path  data/features/query_bert.h5 \
+   --vid_feat_path   data/features/tvr_i3d_rgb600_avg_cl-1.5.h5 \
+   --sub_bert_path   data/features/tvr_sub_pretrained_w_sub_query_max_cl-1.5.h5\
+   --dset_name=tvr \
+   --eval_split_name=val \
+   --nms_thd=-1 \
+   --results_root=results \
+   --clip_length=1.5 \
+   --vid_feat_size=1024 \
+   --ctx_mode=video_sub_tef \
+   --max_ctx_l=128 \
+   --max_pred_l=16 \
+   --eval_num_per_epoch=2 \
+   --n_epoch=100 \
+   --exp_id=top40 \
+   --model_name=XML
+
+   # qsub -I -l select=1:ngpus=1 -P gs_slab -q gpu8
+   # cd 11_TVR-Ranking/TVRetrieval/; conda activate py11; sh run_top40.sh