Update README.md

README.md

@@ -1,3 +1,271 @@
 ---
 license: apache-2.0
 ---
+# MetricX-23
+
+*This is not an officially supported Google product.*
+
+**GitHub repository: [https://github.com/google-research/metricx](https://github.com/google-research/metricx)**
+
+This repository contains the MetricX-23 models,
+a family of models for automatic evaluation of translations that were proposed
+in the WMT'23 Metrics Shared Task submission
+[MetricX-23: The Google Submission to the WMT 2023 Metrics Shared Task](https://aclanthology.org/2023.wmt-1.63/).
+The models were trained in [T5X](https://github.com/google-research/t5x) and
+then converted for use in PyTorch.
+
+## Available Models
+There are 6 models available on HuggingFace that vary in the number of
+parameters and whether or not the model is reference-based or reference-free
+(also known as quality estimation, or QE):
+
+* [MetricX-23-XXL](https://huggingface.co/google/metricx-23-large-v2p0)
+* [MetricX-23-XL](https://huggingface.co/google/metricx-23-xl-v2p0)
+* [MetricX-23-Large](https://huggingface.co/google/metricx-23-xxl-v2p0)
+* [MetricX-23-QE-XXL](https://huggingface.co/google/metricx-23-qe-large-v2p0)
+* [MetricX-23-QE-XL](https://huggingface.co/google/metricx-23-qe-xl-v2p0)
+* [MetricX-23-QE-Large](https://huggingface.co/google/metricx-23-qe-xxl-v2p0)
+
+We recommend using the XXL model versions for the best agreement with human
+judgments of translation quality, the Large versions for best speed, and the
+XL for an intermediate use case.
+
+
+## Changes to the WMT'23 Submission
+
+These models available here are most similar to the primary submission to the WMT'23 Metrics
+Shared Task. They are initialized with [mT5](https://aclanthology.org/2021.naacl-main.41/)
+then fine-tuned on a combination of direct assessment and MQM data. However,
+we made some changes that make these models different from the WMT'23 submissions.
+
+First, the models are trained to regress the actual MQM score rather than a
+normalized score between 0 and 1. **That means the output from the MetricX-23
+models is a score in the range [0, 25] where lower is better (i.e., it predicts
+an error score).**
+
+Second, these models were trained with a larger variety of synthetic data that
+makes them more robust to translation edge cases like over- and undertranslation,
+described in more detail in the following section.
+
+### Synthetic Data
+
+In order for our MetricX models to learn to identify certain types of bad
+translations that are not sufficiently (or at all) represented in the regular
+training data, we created synthetic examples and mixed them in during training.
+The synthetic training data was generated from the DA datasets ranging from
+WMT15 to WMT21 (~ 43 language pairs). In most cases, the synthetic examples have
+the candidate translation manipulated so as to turn it into a bad translation
+with a specific issue commonly unrecognized by learned metrics.
+
+The table below provides an overview of the various failure modes that we
+considered, including brief descriptions of how we prepared the synthetic data
+to address them.
+
+| Failure mode | Synthetic example description |
+| ----------- | ----------- |
+| Undertranslation | Candidate translation with an arbitrary sentence removed (if multi-sentence); alternatively, candidate with a certain proportion of words removed from the end. |
+| Overtranslation | Candidate translation duplicated (with space in between). |
+| Fluent but unrelated translation | Arbitrary reference of a similar length from the dataset. |
+| Gibberish | Text of a similar length as the reference, generated by sampling words from the reference translation vocabulary (built from all references in the data). |
+| Missing punctuation | Reference translation with the end punctuation removed (11 punctuation symbols considered). |
+| Latin instead of Chinese/Japanese or Hindi/Bengali punctuation | Candidate translation with the language-specific punctuation symbol at the end replaced with the Latin equivalent (e.g., "." instead of "。" or "।"); alternatively, the punctuation symbol is replaced with the Latin equivalent in the reference, keeping the correct one in the candidate. |
+| Reference-matching translation | Reference translation copied as the candidate translation (unlike the rest of the synthetic data, these examples are meant to train the metric to predict a perfect score for candidates matching the reference). |
+
+Examples from the first 4 categories were assigned a label corresponding to the
+worst score on the given rating scale (e.g., 25 when mixed with MQM training
+data), whereas the reference-matching translation examples are assigned the best
+score (e.g., 0 when used with MQM data). The missing/incorrect punctuation
+examples were labeled with a score slightly worse than perfect.
+
+Note that some of the synthetic datasets are only meaningful in the
+reference-based scenario, and we thus excluded them when training a QE variant
+of MetricX. These are the Latin-vs-special punctuation and the
+reference-matching translation examples.
+
+Most of the synthetic training sets were created using stratified sampling
+across target languages, taking 500 examples per target language. One exception
+is the missing punctuation set, which used a stratified sample across different
+punctuation symbols instead.
+
+When training MetricX, a small proportion of the synthetic examples was mixed
+with the regular training examples. During the first-stage fine-tuning on DA
+data, each synthetic training set constituted between 0.1% and 1% of all
+training examples, whereas in the second-stage fine-tuning on MQM data we used
+an even smaller proportion, around 0.05%.
+
+As for evaluating the effect of the synthetic training data on the model's
+performance, the DEMETR challenge set - which we originally used to evaluate the
+models submitted to the WMT23 Metrics Shared Task - was not adequate anymore. We
+therefore created a new DEMETR-style test set based on the WMT22 DA data, with
+examples constructed analogically to the synthetic training examples, as
+described above. This test set helped us determine the right proportions of
+synthetic data for fine-tuning in order to make MetricX robust for the failure
+modes in consideration, without sacrificing the system- and segment-level
+correlations with human ratings.
+
+## Usage
+
+The code for using MetricX models can be found at [https://github.com/google-research/metricx](https://github.com/google-research/metricx).
+The repository contains example prediction scripts, described below.
+
+The `metricx23/predict.py` script contains an example for how to run inference
+on the models.
+
+### Reference-Based
+Example usage for a reference-based model:
+
+```bash
+python -m metricx23.predict \
+  --tokenizer google/mt5-xl \
+  --model_name_or_path google/metricx-23-xl-v2p0 \
+  --max_input_length 1024 \
+  --batch_size 1 \
+  --input_file input.jsonl \
+  --output_file output.jsonl
+```
+
+`input.jsonl` is expected to have 1 serialized JSON object per line with
+`"reference"` and `"hypothesis"` fields. The output jsonl will be parallel
+to `input.jsonl` but additionally contain a `"prediction"` field with the predicted score.
+
+Note that the model was trained with a maximum input length of 1024 tokens, so
+significantly increasing that value may lead to unpredictable behavior.
+
+### Reference-Free
+Example usage for a reference-free model:
+
+```bash
+python -m metricx23.predict \
+  --tokenizer google/mt5-xl \
+  --model_name_or_path google/metricx-23-qe-xl-v2p0 \
+  --max_input_length 1024 \
+  --batch_size 1 \
+  --input_file input.jsonl \
+  --output_file output.jsonl \
+  --qe
+```
+
+`input.jsonl` is expected to have 1 serialized JSON object per line with
+`"source"` and `"hypothesis"` fields. The output jsonl will be parallel
+to `input.jsonl` but additionally contain a `"prediction"` field with the predicted score.
+
+
+## Meta-Evaluation
+The `metricx23/evaluate.py` script contains code to calculate various correlations
+between the MetricX-23 scores and MQM ratings of translation quality using the
+[MT Metrics Eval](https://github.com/google-research/mt-metrics-eval) library.
+
+Example usage:
+
+```bash
+python -m metricx23.evaluate \
+  --dataset wmt22 \
+  --lp en-de \
+  --input_file input.jsonl \
+  --output_file output.json
+```
+
+`input.jsonl` is expected to have one JSON object serialized per line.
+Each JSON object is expected to contain 4 fields:
+
+* `"system_id"`: The name of the system that generated the translation.
+* `"segment_id"`: The 0-based index of the corresponding segment in the MT
+Metrics Eval data.
+* `"label"`: The ground-truth translation quality score (with higher is better).
+* `"prediction"`: The model predicted translation quality score (with lower is
+better; the script negates the scores so higher is better).
+
+The script will calculate the 4 agreement/correlations that were used in the
+WMT'23 Shared Task. Below are the results for the MetricX-23 models on the
+WMT'22 Metrics Shared Task data:
+
+English-German:
+
+| Model      | System-Level Accuracy | System-Level Pearson | Segment-Level Pearson | Segment-Level Pairwise Acc |
+| ----------- | ----------- | ----------- | ----------- | ----------- |
+| MetricX-23-XXL      | 0.795       | 0.835       | 0.546       | 0.619       |
+| MetricX-23-XL   | 0.756        | 0.813       | 0.540       | 0.605       |
+| MetricX-23-Large   | 0.769        | 0.759       | 0.507       | 0.595       |
+| MetricX-23-QE-XXL   | 0.769        | 0.830       | 0.490       | 0.606       |
+| MetricX-23-QE-XL   | 0.718        | 0.684       | 0.421       | 0.594       |
+| MetricX-23-QE-Large   | 0.744        | 0.671       | 0.387       | 0.579       |
+
+English-Russian:
+
+| Model      | System-Level Accuracy | System-Level Pearson | Segment-Level Pearson | Segment-Level Pairwise Acc |
+| ----------- | ----------- | ----------- | ----------- | ----------- |
+| MetricX-23-XXL      | 0.905       | 0.943       | 0.477       | 0.609       |
+| MetricX-23-XL   | 0.876        | 0.906       | 0.498       | 0.589       |
+| MetricX-23-Large   | 0.876        | 0.841       | 0.474       | 0.569       |
+| MetricX-23-QE-XXL   | 0.895        | 0.940       | 0.470       | 0.602       |
+| MetricX-23-QE-XL   | 0.848        | 0.861       | 0.415       | 0.570       |
+| MetricX-23-QE-Large   | 0.819        | 0.778       | 0.411       | 0.551       |
+
+Chinese-English:
+
+| Model      | System-Level Accuracy | System-Level Pearson | Segment-Level Pearson | Segment-Level Pairwise Acc |
+| ----------- | ----------- | ----------- | ----------- | ----------- |
+| MetricX-23-XXL      | 0.868       | 0.919       | 0.605       | 0.551       |
+| MetricX-23-XL   | 0.868        | 0.924       | 0.584       | 0.543       |
+| MetricX-23-Large   | 0.857        | 0.919       | 0.555       | 0.539       |
+| MetricX-23-QE-XXL   | 0.857        | 0.928       | 0.573       | 0.544       |
+| MetricX-23-QE-XL   | 0.802        | 0.879       | 0.546       | 0.529       |
+| MetricX-23-QE-Large   | 0.758        | 0.904       | 0.522       | 0.529       |
+
+
+The `metricx23/evaluate_wmt23.py` script re-calculates the average correlation
+score that was used to rank submissions from the
+[WMT'23 Shared Task](https://www2.statmt.org/wmt23/pdf/2023.wmt-1.51.pdf).
+
+Example usage:
+
+```bash
+python -m metricx23.evaluate_wmt23 \
+  --en_de predictions_ende.jsonl \
+  --he_en predictions_heen.jsonl \
+  --zh_en predictions_zhen.jsonl \
+  --output_file output.json
+```
+
+Each of the 3 input files is expected to be in the same format as described
+above. Each file should correspond to running inference on each of the language
+pairs from the WMT'23 dataset.
+
+The results for each of the models is the following:
+
+| Model      | Average Correlation |
+| ----------- | ----------- |
+| MetricX-23-XXL      | 0.812       |
+| MetricX-23-XL   | 0.813        |
+| MetricX-23-Large   | 0.794        |
+| MetricX-23-QE-XXL   | 0.797        |
+| MetricX-23-QE-XL   | 0.767        |
+| MetricX-23-QE-Large   | 0.762        |
+
+
+## Citation
+If you use MetricX-23 in your research, please cite the following publication:
+
+```bibtex
+@inproceedings{juraska-etal-2023-metricx,
+    title = {{MetricX-23: The Google Submission to the WMT 2023 Metrics Shared Task}},
+    author = "Juraska, Juraj  and
+      Finkelstein, Mara  and
+      Deutsch, Daniel  and
+      Siddhant, Aditya  and
+      Mirzazadeh, Mehdi  and
+      Freitag, Markus",
+    editor = "Koehn, Philipp  and
+      Haddow, Barry  and
+      Kocmi, Tom  and
+      Monz, Christof",
+    booktitle = "Proceedings of the Eighth Conference on Machine Translation",
+    month = dec,
+    year = "2023",
+    address = "Singapore",
+    publisher = "Association for Computational Linguistics",
+    url = "https://aclanthology.org/2023.wmt-1.63",
+    doi = "10.18653/v1/2023.wmt-1.63",
+    pages = "756--767",
+}
+```