Model

This model restores punctuation, predicts full stops (sentence boundaries), and predicts true-casing (capitalization) for text in the 6 most popular Romance languages:

  • Spanish
  • French
  • Portuguese
  • Catalan
  • Italian
  • Romanian

Together, these languages cover approximately 97% of native speakers of the Romance language family.

The model comprises a SentencePiece tokenizer, a Transformer encoder, and MLP prediction heads.

This model predicts the following punctuation per input subtoken:

  • .
  • ,
  • ?
  • ¿
  • ACRONYM

Though rare in these languages (relative to English), the special token ACRONYM allows fully punctuating tokens such as "pm" → "p.m.".

Widget notes If you use the widget, it'll take a minute to load the model since a "generic" library is used. Further, the widget does not respect multi-line output, so fullstop predictions are annotated with "\n".

Usage

The model is released as a SentencePiece tokenizer and an ONNX graph.

The easy way to use this model is to install punctuators:

pip install punctuators

If this package is broken, please let me know in the community tab (I update it for each model and break it a lot!).

Example Usage
from typing import List

from punctuators.models import PunctCapSegModelONNX

# Instantiate this model
# This will download the ONNX and SPE models. To clean up, delete this model from your HF cache directory.
m = PunctCapSegModelONNX.from_pretrained("pcs_romance")

# Define some input texts to punctuate, at least one per language
input_texts: List[str] = [
    "este modelo fue entrenado en un gpu a100 en realidad no se que dice esta frase lo traduje con nmt",
    "hola amigo cómo estás es un día lluvioso hoy",
    "hola amic com va avui ha estat un dia plujós el català prediu massa puntuació per com s'ha entrenat",
    "ciao amico come va oggi è stata una giornata piovosa",
    "olá amigo como tá indo estava chuvoso hoje",
    "salut l'ami comment ça va il pleuvait aujourd'hui",
    "salut prietene cum stă treaba azi a fost ploios",
]
results: List[List[str]] = m.infer(input_texts)
for input_text, output_texts in zip(input_texts, results):
    print(f"Input: {input_text}")
    print(f"Outputs:")
    for text in output_texts:
        print(f"\t{text}")
    print()

Exact output may vary based on the model version; here is the current output:

Expected Output
Input: este modelo fue entrenado en un gpu a100 en realidad no se que dice esta frase lo traduje con nmt
Outputs:
    Este modelo fue entrenado en un GPU A100.
    En realidad, no se que dice esta frase lo traduje con NMT.

Input: hola amigo cómo estás es un día lluvioso hoy
Outputs:
    Hola, amigo.
    ¿Cómo estás?
    Es un día lluvioso hoy.

Input: hola amic com va avui ha estat un dia plujós el català prediu massa puntuació per com s'ha entrenat
Outputs:
    Hola, amic.
    Com va avui?
    Ha estat un dia plujós.
    El català prediu massa puntuació per com s'ha entrenat.

Input: ciao amico come va oggi è stata una giornata piovosa
Outputs:
    Ciao amico, come va?
    Oggi è stata una giornata piovosa.

Input: olá amigo como tá indo estava chuvoso hoje
Outputs:
    Olá, amigo, como tá indo?
    Estava chuvoso hoje.

Input: salut l'ami comment ça va il pleuvait aujourd'hui
Outputs:
    Salut l'ami.
    Comment ça va?
    Il pleuvait aujourd'hui.

Input: salut prietene cum stă treaba azi a fost ploios
Outputs:
    Salut prietene, cum stă treaba azi?
    A fost ploios.

If you prefer your output to not be broken into separate sentences, you can disable sentence boundary detection in the API call:

input_texts: List[str] = [
    "hola amigo cómo estás es un día lluvioso hoy",
]
results: List[str] = m.infer(input_texts, apply_sbd=False)
print(results[0])

Instead of a List[List[str]] (a list of output sentences for each input), we get a List[str] (one output sentence per input):

Hola, amigo. ¿Cómo estás? Es un día lluvioso hoy.

Training Data

For all languages except Catalan, this model was trained with ~10M lines of text per language from StatMT's News Crawl.

Catalan is not included in StatMT's News Crawl. For completeness of the Romance language family, ~500k lines of OpenSubtitles was used for Catalan. Due to this, Catalan performance may be sub-par and may over-predict punctuation and sentence breaks, which is typical of OpenSubtitles.

Training Parameters

This model was trained by concatenating between 1 and 14 random sentences. The concatenation points became sentence boundary targets, text was lower-cased to produce true-case targets, and punctuation was removed to create punctuation targets.

Batches were built by randomly sampling from each language. Each example is language homogenous (i.e., we only concatenate sentences from the same language). Batches were multilingual. Neither language tags nor language-specific paths are utilized in the graph.

The maximum length during training was 256 subtokens. The punctuators package can punctuate inputs of any length. This is accomplished behind the scenes by splitting the input into overlapping subsegments of 256 tokens, and combining the results.

If you use the raw ONNX graph, note that while the model will accept sequences up to 512 tokens, only 256 positional embeddings have been trained.

Contact

Contact me at [email protected] with requests or issues, or just let me know on the community tab.

Metrics

Test sets were generated with 3,000 lines of held-out data per language (OpenSubtitles for Catalan, News Crawl for all others). Examples were derived by concatenating 10 sentences per example, removing all punctuation, and lower-casing all letters.

Since punctuation is subjective (e.g., see "hello friend how's it going" in the above examples) punctuation metrics can be misleading.

Also, keep in mind that the data is noisy. Catalan is especially noisy, since it's OpenSubtitles (note how Catalan has a 50 instances of "¿" which should not appear).

Note that we call the label "¿" "pre-punctuation" since it is unique in that it appears before words, and thus we predict it separate from the other punctuation tokens.

Generally, periods are easy, commas are a harder, question marks are hard, and acronyms are rare and noisy.

Expand any of the following tabs to see metrics for that language.

Spanish metrics
Pre-punctuation report: 
    label                                                precision    recall       f1           support   
    <NULL> (label_id: 0)                                    99.92      99.97      99.95     572069
    ¿ (label_id: 1)                                         81.93      60.46      69.57       1095
    -------------------
    micro avg                                               99.90      99.90      99.90     573164
    macro avg                                               90.93      80.22      84.76     573164
    weighted avg                                            99.89      99.90      99.89     573164
    
Punctuation report:
    label                                                precision    recall       f1           support   
    <NULL> (label_id: 0)                                    98.70      98.44      98.57     517310
    <ACRONYM> (label_id: 1)                                 39.68      86.21      54.35         58
    . (label_id: 2)                                         87.72      90.41      89.04      29267
    , (label_id: 3)                                         73.17      74.68      73.92      25422
    ? (label_id: 4)                                         69.49      59.26      63.97       1107
    -------------------
    micro avg                                               96.90      96.90      96.90     573164
    macro avg                                               73.75      81.80      75.97     573164
    weighted avg                                            96.94      96.90      96.92     573164
    
True-casing report:
    label                                                precision    recall       f1           support   
    LOWER (label_id: 0)                                     99.85      99.73      99.79    2164982
    UPPER (label_id: 1)                                     92.01      95.32      93.64      69437
    -------------------
    micro avg                                               99.60      99.60      99.60    2234419
    macro avg                                               95.93      97.53      96.71    2234419
    weighted avg                                            99.61      99.60      99.60    2234419

Fullstop report:
    label                                                precision    recall       f1           support   
    NOSTOP (label_id: 0)                                   100.00      99.98      99.99     543228
    FULLSTOP (label_id: 1)                                  99.66      99.93      99.80      32931
    -------------------
    micro avg                                               99.98      99.98      99.98     576159
    macro avg                                               99.83      99.96      99.89     576159
    weighted avg                                            99.98      99.98      99.98     576159
Portuguese metrics
Pre-punctuation report:
    label                                                precision    recall       f1           support   
    <NULL> (label_id: 0)                                   100.00     100.00     100.00     539822
    ¿ (label_id: 1)                                          0.00       0.00       0.00          0
    -------------------
    micro avg                                              100.00     100.00     100.00     539822
    macro avg                                              100.00     100.00     100.00     539822
    weighted avg                                           100.00     100.00     100.00     539822

Punctuation report:
    label                                                precision    recall       f1           support   
    <NULL> (label_id: 0)                                    98.77      98.27      98.52     481148
    <ACRONYM> (label_id: 1)                                  0.00       0.00       0.00          0
    . (label_id: 2)                                         87.63      90.63      89.11      29090
    , (label_id: 3)                                         74.44      78.69      76.50      28549
    ? (label_id: 4)                                         66.30      52.27      58.45       1035
    -------------------
    micro avg                                               96.74      96.74      96.74     539822
    macro avg                                               81.79      79.96      80.65     539822
    weighted avg                                            96.82      96.74      96.77     539822

True-casing report:
    label                                                precision    recall       f1           support   
    LOWER (label_id: 0)                                     99.90      99.82      99.86    2082598
    UPPER (label_id: 1)                                     94.75      97.08      95.90      70555
    -------------------
    micro avg                                               99.73      99.73      99.73    2153153
    macro avg                                               97.32      98.45      97.88    2153153
    weighted avg                                            99.73      99.73      99.73    2153153

Fullstop report:
    label                                                precision    recall       f1           support   
    NOSTOP (label_id: 0)                                   100.00      99.98      99.99     509905
    FULLSTOP (label_id: 1)                                  99.72      99.98      99.85      32909
    -------------------
    micro avg                                               99.98      99.98      99.98     542814
    macro avg                                               99.86      99.98      99.92     542814
    weighted avg                                            99.98      99.98      99.98     542814
Romanian metrics
Pre-punctuation report:
    label                                                precision    recall       f1           support   
    <NULL> (label_id: 0)                                   100.00     100.00     100.00     580702
    ¿ (label_id: 1)                                          0.00       0.00       0.00          0
    -------------------
    micro avg                                              100.00     100.00     100.00     580702
    macro avg                                              100.00     100.00     100.00     580702
    weighted avg                                           100.00     100.00     100.00     580702

Punctuation report:
    label                                                precision    recall       f1           support   
    <NULL> (label_id: 0)                                    98.56      98.47      98.51     520647
    <ACRONYM> (label_id: 1)                                 52.00      79.89      63.00        179
    . (label_id: 2)                                         87.29      89.37      88.32      29852
    , (label_id: 3)                                         75.26      74.69      74.97      29218
    ? (label_id: 4)                                         60.73      55.46      57.98        806
    -------------------
    micro avg                                               96.74      96.74      96.74     580702
    macro avg                                               74.77      79.57      76.56     580702
    weighted avg                                            96.74      96.74      96.74     580702

Truecasing report:
    label                                                precision    recall       f1           support   
    LOWER (label_id: 0)                                     99.84      99.75      99.79    2047297
    UPPER (label_id: 1)                                     93.56      95.65      94.59      77424
    -------------------
    micro avg                                               99.60      99.60      99.60    2124721
    macro avg                                               96.70      97.70      97.19    2124721
    weighted avg                                            99.61      99.60      99.60    2124721

Fullstop report:
    label                                                precision    recall       f1           support   
    NOSTOP (label_id: 0)                                   100.00      99.96      99.98     550858
    FULLSTOP (label_id: 1)                                  99.26      99.94      99.60      32833
    -------------------
    micro avg                                               99.95      99.95      99.95     583691
    macro avg                                               99.63      99.95      99.79     583691
    weighted avg                                            99.96      99.95      99.96     583691
Italian metrics
Pre-punctuation report:
    label                                                precision    recall       f1           support   
    <NULL> (label_id: 0)                                   100.00     100.00     100.00     577636
    ¿ (label_id: 1)                                          0.00       0.00       0.00          0
    -------------------
    micro avg                                              100.00     100.00     100.00     577636
    macro avg                                              100.00     100.00     100.00     577636
    weighted avg                                           100.00     100.00     100.00     577636

Punctuation report: 
    label                                                precision    recall       f1           support   
    <NULL> (label_id: 0)                                    98.10      97.73      97.91     522727
    <ACRONYM> (label_id: 1)                                 41.76      48.72      44.97         78
    . (label_id: 2)                                         81.71      86.70      84.13      28881
    , (label_id: 3)                                         61.72      63.24      62.47      24703
    ? (label_id: 4)                                         62.55      41.78      50.10       1247
    -------------------
    micro avg                                               95.58      95.58      95.58     577636
    macro avg                                               69.17      67.63      67.92     577636
    weighted avg                                            95.64      95.58      95.60     577636

Truecasing report:
    label                                                precision    recall       f1           support   
    LOWER (label_id: 0)                                     99.76      99.70      99.73    2160781
    UPPER (label_id: 1)                                     91.18      92.76      91.96      72471
    -------------------
    micro avg                                               99.47      99.47      99.47    2233252
    macro avg                                               95.47      96.23      95.85    2233252
    weighted avg                                            99.48      99.47      99.48    2233252

Fullstop report:
    label                                                precision    recall       f1           support   
    NOSTOP (label_id: 0)                                    99.99      99.98      99.99     547875
    FULLSTOP (label_id: 1)                                  99.72      99.91      99.82      32742
    -------------------
    micro avg                                               99.98      99.98      99.98     580617
    macro avg                                               99.86      99.95      99.90     580617
    weighted avg                                            99.98      99.98      99.98     580617
French metrics
Pre-punctuation report:
    label                                                precision    recall       f1           support   
    <NULL> (label_id: 0)                                   100.00     100.00     100.00     614010
    ¿ (label_id: 1)                                          0.00       0.00       0.00          0
    -------------------
    micro avg                                              100.00     100.00     100.00     614010
    macro avg                                              100.00     100.00     100.00     614010
    weighted avg                                           100.00     100.00     100.00     614010

Punctuation report:
    label                                                precision    recall       f1           support   
    <NULL> (label_id: 0)                                    98.72      98.57      98.65     556366
    <ACRONYM> (label_id: 1)                                 38.46      71.43      50.00         49
    . (label_id: 2)                                         86.41      88.56      87.47      28969
    , (label_id: 3)                                         72.15      72.80      72.47      27183
    ? (label_id: 4)                                         75.81      67.78      71.57       1443
    -------------------
    micro avg                                               96.88      96.88      96.88     614010
    macro avg                                               74.31      79.83      76.03     614010
    weighted avg                                            96.91      96.88      96.89     614010

Truecasing report:
    label                                                precision    recall       f1           support   
    LOWER (label_id: 0)                                     99.84      99.80      99.82    2127174
    UPPER (label_id: 1)                                     93.72      94.73      94.22      66496
    -------------------
    micro avg                                               99.65      99.65      99.65    2193670
    macro avg                                               96.78      97.27      97.02    2193670
    weighted avg                                            99.65      99.65      99.65    2193670

Fullstop report:
    label                                                precision    recall       f1           support   
    NOSTOP (label_id: 0)                                    99.99      99.94      99.97     584331
    FULLSTOP (label_id: 1)                                  98.92      99.90      99.41      32661
    -------------------
    micro avg                                               99.94      99.94      99.94     616992
    macro avg                                               99.46      99.92      99.69     616992
    weighted avg                                            99.94      99.94      99.94     616992
Catalan metrics
Pre-punctuation report:
    label                                                precision    recall       f1           support   
    <NULL> (label_id: 0)                                    99.97     100.00      99.98     143817
    ¿ (label_id: 1)                                          0.00       0.00       0.00         50
    -------------------
    micro avg                                               99.97      99.97      99.97     143867
    macro avg                                               49.98      50.00      49.99     143867
    weighted avg                                            99.93      99.97      99.95     143867

Punctuation report:
    label                                                precision    recall       f1           support   
    <NULL> (label_id: 0)                                    97.61      97.73      97.67     119040
    <ACRONYM> (label_id: 1)                                  0.00       0.00       0.00         28
    . (label_id: 2)                                         74.02      79.46      76.65      15282
    , (label_id: 3)                                         60.88      50.75      55.36       5836
    ? (label_id: 4)                                         64.94      60.28      62.52       3681
    -------------------
    micro avg                                               92.90      92.90      92.90     143867
    macro avg                                               59.49      57.64      58.44     143867
    weighted avg                                            92.76      92.90      92.80     143867

Truecasing report:
    label                                                precision    recall       f1           support   
    LOWER (label_id: 0)                                     99.81      99.83      99.82     422395
    UPPER (label_id: 1)                                     97.09      96.81      96.95      24854
    -------------------
    micro avg                                               99.66      99.66      99.66     447249
    macro avg                                               98.45      98.32      98.39     447249
    weighted avg                                            99.66      99.66      99.66     447249

Fullstop report:
    label                                                precision    recall       f1           support   
    NOSTOP (label_id: 0)                                    99.93      99.63      99.78     123867
    FULLSTOP (label_id: 1)                                  97.97      99.59      98.77      22000
    -------------------
    micro avg                                               99.63      99.63      99.63     145867
    macro avg                                               98.95      99.61      99.28     145867
    weighted avg                                            99.63      99.63      99.63     145867
Downloads last month
235
Inference Examples
Inference API (serverless) does not yet support generic models for this pipeline type.