Upload folder using huggingface_hub

README.md

@@ -1,3 +1,100 @@
----
-license: cc-by-4.0
----
+---
+language:
+- en
+- fr
+tags:
+- translation
+license: cc-by-4.0
+datasets:
+- quickmt/quickmt-train.en-fr
+model-index:
+- name: quickmt-en-fr
+  results:
+  - task:
+      name: Translation fra-eng
+      type: translation
+      args: fra-eng
+    dataset:
+      name: flores101-devtest
+      type: flores_101
+      args: eng_Latn fra_Latn devtest
+    metrics:
+    - name: CHRF
+      type: chrf
+      value: 71.60
+    - name: BLEU
+      type: bleu
+      value: 50.79
+    - name: COMET
+      type: comet
+      value: 87.11
+---
+
+
+# `quickmt-en-fr` Neural Machine Translation Model 
+
+`quickmt-en-fr` is a reasonably fast and reasonably accurate neural machine translation model for translation from `en` into `fr`.
+
+
+## Model Information
+
+* Trained using [`eole`](https://github.com/eole-nlp/eole)
+* 185M parameter transformer 'big' with 8 encoder layers and 2 decoder layers
+* 50k joint Sentencepiece vocabulary
+* Exported for fast inference to [CTranslate2](https://github.com/OpenNMT/CTranslate2) format
+* Training data: https://huggingface.co/datasets/quickmt/quickmt-train.en-fr/tree/main
+
+See the `eole-config.yaml` model configuration in this repository for further details. 
+
+
+## Usage with `quickmt`
+
+You must install the Nvidia cuda toolkit first, if you want to do GPU inference.
+
+Next, install the `quickmt` python library and download the model:
+
+```bash
+git clone https://github.com/quickmt/quickmt.git
+pip install ./quickmt/
+
+# List available models
+quickmt-list
+
+# Download a model
+quickmt-model-download quickmt/quickmt-en-fr ./quickmt-en-fr
+```
+
+Finally use the model in python:
+
+```python
+from quickmt import Translator
+
+# Auto-detects GPU, set to "cpu" to force CPU inference
+t = Translator("./quickmt-en-fr/", device="auto")
+
+# Translate - set beam size to 5 for higher quality (but slower speed)
+sample_text = "The Virgo interferometer is a large-scale scientific instrument near Pisa, Italy, for detecting gravitational waves."
+t(sample_text, beam_size=1)
+
+# Get alternative translations by sampling
+# You can pass any cTranslate2 `translate_batch` arguments
+t([sample_text], sampling_temperature=1.2, beam_size=1, sampling_topk=50, sampling_topp=0.9)
+```
+
+The model is in `ctranslate2` format, and the tokenizers are `sentencepiece`, so you can use `ctranslate2` directly instead of through `quickmt`. It is also possible  to get this model to work with e.g. [LibreTranslate](https://libretranslate.com/) which also uses `ctranslate2` and `sentencepiece`.
+
+
+## Metrics
+
+`bleu` and `chrf2` are calculated with [sacrebleu](https://github.com/mjpost/sacrebleu) on the [Flores200 `devtest` test set](https://huggingface.co/datasets/facebook/flores) ("fra_Latn"->"eng_Latn"). `comet22` with the [`comet`](https://github.com/Unbabel/COMET) library and the [default model](https://huggingface.co/Unbabel/wmt22-comet-da). "Time (s)" is the time in seconds to translate (using `ctranslate2`) the flores-devtest dataset (1012 sentences) on an RTX 4070s GPU with batch size 32.
+
+| Model                            | chrf2 | bleu    | comet22 | Time (s) |
+| -------------------------------- | ----- | ------- | ------- | -------- |
+| quickmt/quickmt-en-fr            | 71.60 | 50.79   | 87.11   |  1.28    |
+| Helsinki-NLP/opus-mt-en-fr       | 69.98 | 47.97   | 86.29   |  4.13    |
+| facebook/m2m100_418M             | 63.29 | 39.52   | 82.11   | 22.4     |
+| facebook/m2m100_1.2B             | 68.31 | 45.39   | 86.50   | 44.0     |
+| facebook/nllb-200-distilled-600M | 70.36 | 48.71   | 87.63   | 27.8     |
+| facebook/nllb-200-distilled-1.3B | 71.95 | 51.10   | 88.50   | 47.8     |
+
+`quickmt-en-fr` is the fastest and is higher quality than `opus-mt-en-fr`, `m2m100_418m`, `m2m100_1.2B`.

config.json

@@ -0,0 +1,10 @@
+{
+  "add_source_bos": false,
+  "add_source_eos": true,
+  "bos_token": "<s>",
+  "decoder_start_token": "<s>",
+  "eos_token": "</s>",
+  "layer_norm_epsilon": 1e-06,
+  "multi_query_attention": false,
+  "unk_token": "<unk>"
+}

eole-config.yaml

@@ -0,0 +1,101 @@
+## IO
+save_data: enfr/data_spm
+overwrite: True
+seed: 1234
+report_every: 100
+valid_metrics: ["BLEU"]
+tensorboard: true
+tensorboard_log_dir: tensorboard
+
+### Vocab
+src_vocab: enfr/joint.eole.vocab
+tgt_vocab: enfr/joint.eole.vocab
+src_vocab_size: 50000
+tgt_vocab_size: 50000
+vocab_size_multiple: 8
+share_vocab: True
+n_sample: 0
+
+data:
+    corpus_1:
+        path_src: hf://quickmt/quickmt-train.fr-en/en
+        path_tgt: hf://quickmt/quickmt-train.fr-en/fr
+        path_sco: hf://quickmt/quickmt-train.fr-en/sco
+    valid:
+        path_src: enfr/dev.en
+        path_tgt: enfr/dev.fr
+
+transforms: [sentencepiece, filtertoolong]
+transforms_configs:
+  sentencepiece:
+    src_subword_model: "enfr/joint.spm.model"
+    tgt_subword_model: "enfr/joint.spm.model"
+  filtertoolong:
+    src_seq_length: 256
+    tgt_seq_length: 256
+
+training:
+    # Run configuration
+    model_path: enfr/model
+    train_from: enfr/model
+    keep_checkpoint: 4
+    save_checkpoint_steps: 2000
+    train_steps: 100000
+    valid_steps: 2000
+    
+    # Train on a single GPU
+    world_size: 1
+    gpu_ranks: [0]
+
+    # Batching
+    batch_type: "tokens"
+    batch_size: 16384
+    valid_batch_size: 16384
+    batch_size_multiple: 8
+    accum_count: [8]
+    accum_steps: [0]
+
+    # Optimizer & Compute
+    compute_dtype: "bf16"
+    optim: "pagedadamw8bit"
+    #optim: "adamw"
+    learning_rate: 2.0
+    warmup_steps: 10000
+    decay_method: "noam"
+    adam_beta2: 0.998
+
+    # Data loading
+    bucket_size: 128000
+    num_workers: 4
+    prefetch_factor: 100
+    
+    # Hyperparams
+    dropout_steps: [0]
+    dropout: [0.1]
+    attention_dropout: [0.1]
+    max_grad_norm: 2
+    label_smoothing: 0.1
+    average_decay: 0.0001
+    param_init_method: xavier_uniform
+    normalization: "tokens"
+
+model:
+    architecture: "transformer"
+    layer_norm: standard
+    share_embeddings: true
+    share_decoder_embeddings: true
+    add_ffnbias: true
+    mlp_activation_fn: gelu
+    add_estimator: false
+    add_qkvbias: false
+    norm_eps: 1e-6
+    hidden_size: 1024
+    encoder:
+        layers: 8
+    decoder:
+        layers: 2
+    heads: 8
+    transformer_ff: 4096
+    embeddings:
+        word_vec_size: 1024
+        position_encoding_type: "SinusoidalInterleaved"

eole-model/config.json

@@ -0,0 +1,150 @@
+{
+  "valid_metrics": [
+    "BLEU"
+  ],
+  "tensorboard_log_dir": "tensorboard",
+  "src_vocab_size": 50000,
+  "tgt_vocab": "enfr/joint.eole.vocab",
+  "seed": 1234,
+  "save_data": "enfr/data_spm",
+  "tgt_vocab_size": 50000,
+  "overwrite": true,
+  "share_vocab": true,
+  "vocab_size_multiple": 8,
+  "tensorboard": true,
+  "transforms": [
+    "sentencepiece",
+    "filtertoolong"
+  ],
+  "report_every": 100,
+  "tensorboard_log_dir_dated": "tensorboard/Feb-20_00-47-23",
+  "n_sample": 0,
+  "src_vocab": "enfr/joint.eole.vocab",
+  "training": {
+    "dropout_steps": [
+      0
+    ],
+    "batch_size_multiple": 8,
+    "bucket_size": 128000,
+    "adam_beta2": 0.998,
+    "label_smoothing": 0.1,
+    "dropout": [
+      0.1
+    ],
+    "compute_dtype": "torch.bfloat16",
+    "valid_batch_size": 16384,
+    "valid_steps": 2000,
+    "accum_count": [
+      8
+    ],
+    "world_size": 1,
+    "gpu_ranks": [
+      0
+    ],
+    "batch_size": 16384,
+    "train_steps": 100000,
+    "train_from": "enfr/model",
+    "average_decay": 0.0001,
+    "save_checkpoint_steps": 2000,
+    "accum_steps": [
+      0
+    ],
+    "max_grad_norm": 2.0,
+    "prefetch_factor": 100,
+    "batch_type": "tokens",
+    "keep_checkpoint": 4,
+    "attention_dropout": [
+      0.1
+    ],
+    "learning_rate": 2.0,
+    "optim": "pagedadamw8bit",
+    "num_workers": 0,
+    "model_path": "enfr/model",
+    "normalization": "tokens",
+    "decay_method": "noam",
+    "warmup_steps": 10000,
+    "param_init_method": "xavier_uniform"
+  },
+  "transforms_configs": {
+    "sentencepiece": {
+      "src_subword_model": "${MODEL_PATH}/joint.spm.model",
+      "tgt_subword_model": "${MODEL_PATH}/joint.spm.model"
+    },
+    "filtertoolong": {
+      "tgt_seq_length": 256,
+      "src_seq_length": 256
+    }
+  },
+  "model": {
+    "layer_norm": "standard",
+    "add_qkvbias": false,
+    "architecture": "transformer",
+    "add_estimator": false,
+    "position_encoding_type": "SinusoidalInterleaved",
+    "norm_eps": 1e-06,
+    "heads": 8,
+    "share_decoder_embeddings": true,
+    "hidden_size": 1024,
+    "mlp_activation_fn": "gelu",
+    "share_embeddings": true,
+    "transformer_ff": 4096,
+    "add_ffnbias": true,
+    "encoder": {
+      "layer_norm": "standard",
+      "encoder_type": "transformer",
+      "add_qkvbias": false,
+      "n_positions": null,
+      "hidden_size": 1024,
+      "mlp_activation_fn": "gelu",
+      "position_encoding_type": "SinusoidalInterleaved",
+      "norm_eps": 1e-06,
+      "transformer_ff": 4096,
+      "layers": 8,
+      "heads": 8,
+      "src_word_vec_size": 1024,
+      "add_ffnbias": true
+    },
+    "decoder": {
+      "layer_norm": "standard",
+      "n_positions": null,
+      "add_qkvbias": false,
+      "hidden_size": 1024,
+      "tgt_word_vec_size": 1024,
+      "mlp_activation_fn": "gelu",
+      "decoder_type": "transformer",
+      "position_encoding_type": "SinusoidalInterleaved",
+      "norm_eps": 1e-06,
+      "transformer_ff": 4096,
+      "layers": 2,
+      "heads": 8,
+      "add_ffnbias": true
+    },
+    "embeddings": {
+      "tgt_word_vec_size": 1024,
+      "src_word_vec_size": 1024,
+      "position_encoding_type": "SinusoidalInterleaved",
+      "word_vec_size": 1024
+    }
+  },
+  "data": {
+    "corpus_1": {
+      "path_tgt": "hf://quickmt/quickmt-train.fr-en/fr",
+      "transforms": [
+        "sentencepiece",
+        "filtertoolong"
+      ],
+      "path_align": null,
+      "path_sco": "hf://quickmt/quickmt-train.fr-en/sco",
+      "path_src": "hf://quickmt/quickmt-train.fr-en/en"
+    },
+    "valid": {
+      "path_tgt": "enfr/dev.fr",
+      "transforms": [
+        "sentencepiece",
+        "filtertoolong"
+      ],
+      "path_align": null,
+      "path_src": "enfr/dev.en"
+    }
+  }
+}

eole-model/joint.spm.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:19bab02bdbc41207bd3fabf86e20e691e978f78725d898c42de586b67cdaed02
+size 1146015

eole-model/model.00.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2044ba22f9b306c427eed360f189b63ca8172bddc4d6fe559593beac1188f7cf
+size 762769904

joint.spm.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:19bab02bdbc41207bd3fabf86e20e691e978f78725d898c42de586b67cdaed02
+size 1146015

model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8e9e6b469af7bbafdd6fbbf1cabfabcd278e74b5acb637d4c2ea3e41f004b023
+size 381336824