All experimental runs shown in the above table can be programmatically executed based on the instructions below. To list all the experimental conditions:
python -m pyserini.2cr.mrtydi --list-conditions
Run all languages for a specific condition and show commands:
python -m pyserini.2cr.mrtydi --condition bm25 --display-commands
Run a particular language for a specific condition and show commands:
python -m pyserini.2cr.mrtydi --condition bm25 --language ko --display-commands
Run all languages for all conditions and show commands:
python -m pyserini.2cr.mrtydi --all --display-commands
With the above commands, run files will be placed in the current directory. Use the option --directory runs to place the runs in a sub-directory.
For a specific condition, just show the commands and do not run:
python -m pyserini.2cr.mrtydi --condition bm25 --display-commands --dry-run
This will generate exactly the commands for a specific condition above (corresponding to a row in the table).
For a specific condition and language, just show the commands and do not run:
python -m pyserini.2cr.mrtydi --condition bm25 --language ko --display-commands --dry-run
For all conditions, just show the commands and do not run and skip evaluation:
python -m pyserini.2cr.mrtydi --all --display-commands --dry-run --skip-eval
Finally, to generate this page:
python -m pyserini.2cr.mrtydi --generate-report --output docs/2cr/mrtydi.html
The output file mrtydi.html should be identical to this page.