pipeline/nltk/lm/__init__.py

# Natural Language Toolkit: Language Models
#
# Copyright (C) 2001-2023 NLTK Project
# Authors: Ilia Kurenkov <[email protected]>
# URL: <https://www.nltk.org/
# For license information, see LICENSE.TXT
"""

NLTK Language Modeling Module.

------------------------------



Currently this module covers only ngram language models, but it should be easy

to extend to neural models.





Preparing Data

==============



Before we train our ngram models it is necessary to make sure the data we put in

them is in the right format.

Let's say we have a text that is a list of sentences, where each sentence is

a list of strings. For simplicity we just consider a text consisting of

characters instead of words.



    >>> text = [['a', 'b', 'c'], ['a', 'c', 'd', 'c', 'e', 'f']]



If we want to train a bigram model, we need to turn this text into bigrams.

Here's what the first sentence of our text would look like if we use a function

from NLTK for this.



    >>> from nltk.util import bigrams

    >>> list(bigrams(text[0]))

    [('a', 'b'), ('b', 'c')]



Notice how "b" occurs both as the first and second member of different bigrams

but "a" and "c" don't? Wouldn't it be nice to somehow indicate how often sentences

start with "a" and end with "c"?

A standard way to deal with this is to add special "padding" symbols to the

sentence before splitting it into ngrams.

Fortunately, NLTK also has a function for that, let's see what it does to the

first sentence.



    >>> from nltk.util import pad_sequence

    >>> list(pad_sequence(text[0],

    ... pad_left=True,

    ... left_pad_symbol="<s>",

    ... pad_right=True,

    ... right_pad_symbol="</s>",

    ... n=2))

    ['<s>', 'a', 'b', 'c', '</s>']



Note the `n` argument, that tells the function we need padding for bigrams.

Now, passing all these parameters every time is tedious and in most cases they

can be safely assumed as defaults anyway.

Thus our module provides a convenience function that has all these arguments

already set while the other arguments remain the same as for `pad_sequence`.



    >>> from nltk.lm.preprocessing import pad_both_ends

    >>> list(pad_both_ends(text[0], n=2))

    ['<s>', 'a', 'b', 'c', '</s>']



Combining the two parts discussed so far we get the following preparation steps

for one sentence.



    >>> list(bigrams(pad_both_ends(text[0], n=2)))

    [('<s>', 'a'), ('a', 'b'), ('b', 'c'), ('c', '</s>')]



To make our model more robust we could also train it on unigrams (single words)

as well as bigrams, its main source of information.

NLTK once again helpfully provides a function called `everygrams`.

While not the most efficient, it is conceptually simple.





    >>> from nltk.util import everygrams

    >>> padded_bigrams = list(pad_both_ends(text[0], n=2))

    >>> list(everygrams(padded_bigrams, max_len=2))

    [('<s>',), ('<s>', 'a'), ('a',), ('a', 'b'), ('b',), ('b', 'c'), ('c',), ('c', '</s>'), ('</s>',)]



We are almost ready to start counting ngrams, just one more step left.

During training and evaluation our model will rely on a vocabulary that

defines which words are "known" to the model.

To create this vocabulary we need to pad our sentences (just like for counting

ngrams) and then combine the sentences into one flat stream of words.



    >>> from nltk.lm.preprocessing import flatten

    >>> list(flatten(pad_both_ends(sent, n=2) for sent in text))

    ['<s>', 'a', 'b', 'c', '</s>', '<s>', 'a', 'c', 'd', 'c', 'e', 'f', '</s>']



In most cases we want to use the same text as the source for both vocabulary

and ngram counts.

Now that we understand what this means for our preprocessing, we can simply import

a function that does everything for us.



    >>> from nltk.lm.preprocessing import padded_everygram_pipeline

    >>> train, vocab = padded_everygram_pipeline(2, text)



So as to avoid re-creating the text in memory, both `train` and `vocab` are lazy

iterators. They are evaluated on demand at training time.





Training

========

Having prepared our data we are ready to start training a model.

As a simple example, let us train a Maximum Likelihood Estimator (MLE).

We only need to specify the highest ngram order to instantiate it.



    >>> from nltk.lm import MLE

    >>> lm = MLE(2)



This automatically creates an empty vocabulary...



    >>> len(lm.vocab)

    0



... which gets filled as we fit the model.



    >>> lm.fit(train, vocab)

    >>> print(lm.vocab)

    <Vocabulary with cutoff=1 unk_label='<UNK>' and 9 items>

    >>> len(lm.vocab)

    9



The vocabulary helps us handle words that have not occurred during training.



    >>> lm.vocab.lookup(text[0])

    ('a', 'b', 'c')

    >>> lm.vocab.lookup(["aliens", "from", "Mars"])

    ('<UNK>', '<UNK>', '<UNK>')



Moreover, in some cases we want to ignore words that we did see during training

but that didn't occur frequently enough, to provide us useful information.

You can tell the vocabulary to ignore such words.

To find out how that works, check out the docs for the `Vocabulary` class.





Using a Trained Model

=====================

When it comes to ngram models the training boils down to counting up the ngrams

from the training corpus.



    >>> print(lm.counts)

    <NgramCounter with 2 ngram orders and 24 ngrams>



This provides a convenient interface to access counts for unigrams...



    >>> lm.counts['a']

    2



...and bigrams (in this case "a b")



    >>> lm.counts[['a']]['b']

    1



And so on. However, the real purpose of training a language model is to have it

score how probable words are in certain contexts.

This being MLE, the model returns the item's relative frequency as its score.



    >>> lm.score("a")

    0.15384615384615385



Items that are not seen during training are mapped to the vocabulary's

"unknown label" token. This is "<UNK>" by default.



    >>> lm.score("<UNK>") == lm.score("aliens")

    True



Here's how you get the score for a word given some preceding context.

For example we want to know what is the chance that "b" is preceded by "a".



    >>> lm.score("b", ["a"])

    0.5



To avoid underflow when working with many small score values it makes sense to

take their logarithm.

For convenience this can be done with the `logscore` method.



    >>> lm.logscore("a")

    -2.700439718141092



Building on this method, we can also evaluate our model's cross-entropy and

perplexity with respect to sequences of ngrams.



    >>> test = [('a', 'b'), ('c', 'd')]

    >>> lm.entropy(test)

    1.292481250360578

    >>> lm.perplexity(test)

    2.449489742783178



It is advisable to preprocess your test text exactly the same way as you did

the training text.



One cool feature of ngram models is that they can be used to generate text.



    >>> lm.generate(1, random_seed=3)

    '<s>'

    >>> lm.generate(5, random_seed=3)

    ['<s>', 'a', 'b', 'c', 'd']



Provide `random_seed` if you want to consistently reproduce the same text all

other things being equal. Here we are using it to test the examples.



You can also condition your generation on some preceding text with the `context`

argument.



    >>> lm.generate(5, text_seed=['c'], random_seed=3)

    ['</s>', 'c', 'd', 'c', 'd']



Note that an ngram model is restricted in how much preceding context it can

take into account. For example, a trigram model can only condition its output

on 2 preceding words. If you pass in a 4-word context, the first two words

will be ignored.

"""

from nltk.lm.counter import NgramCounter
from nltk.lm.models import (
    MLE,
    AbsoluteDiscountingInterpolated,
    KneserNeyInterpolated,
    Laplace,
    Lidstone,
    StupidBackoff,
    WittenBellInterpolated,
)
from nltk.lm.vocabulary import Vocabulary

__all__ = [
    "Vocabulary",
    "NgramCounter",
    "MLE",
    "Lidstone",
    "Laplace",
    "WittenBellInterpolated",
    "KneserNeyInterpolated",
    "AbsoluteDiscountingInterpolated",
    "StupidBackoff",
]