Spaces:
Sleeping
Sleeping
| .. Copyright (C) 2001-2023 NLTK Project | |
| .. For license information, see LICENSE.TXT | |
| ================ | |
| Corpus Readers | |
| ================ | |
| The `nltk.corpus` package defines a collection of *corpus reader* | |
| classes, which can be used to access the contents of a diverse set of | |
| corpora. The list of available corpora is given at: | |
| https://www.nltk.org/nltk_data/ | |
| Each corpus reader class is specialized to handle a specific | |
| corpus format. In addition, the `nltk.corpus` package automatically | |
| creates a set of corpus reader instances that can be used to access | |
| the corpora in the NLTK data package. | |
| Section `Corpus Reader Objects`_ ("Corpus Reader Objects") describes | |
| the corpus reader instances that can be used to read the corpora in | |
| the NLTK data package. Section `Corpus Reader Classes`_ ("Corpus | |
| Reader Classes") describes the corpus reader classes themselves, and | |
| discusses the issues involved in creating new corpus reader objects | |
| and new corpus reader classes. Section `Regression Tests`_ | |
| ("Regression Tests") contains regression tests for the corpus readers | |
| and associated functions and classes. | |
| .. contents:: **Table of Contents** | |
| :depth: 4 | |
| :backlinks: none | |
| --------------------- | |
| Corpus Reader Objects | |
| --------------------- | |
| Overview | |
| ======== | |
| NLTK includes a diverse set of corpora which can be | |
| read using the ``nltk.corpus`` package. Each corpus is accessed by | |
| means of a "corpus reader" object from ``nltk.corpus``: | |
| >>> import nltk.corpus | |
| >>> # The Brown corpus: | |
| >>> print(str(nltk.corpus.brown).replace('\\\\','/')) | |
| <CategorizedTaggedCorpusReader in '.../corpora/brown'...> | |
| >>> # The Penn Treebank Corpus: | |
| >>> print(str(nltk.corpus.treebank).replace('\\\\','/')) | |
| <BracketParseCorpusReader in '.../corpora/treebank/combined'...> | |
| >>> # The Name Genders Corpus: | |
| >>> print(str(nltk.corpus.names).replace('\\\\','/')) | |
| <WordListCorpusReader in '.../corpora/names'...> | |
| >>> # The Inaugural Address Corpus: | |
| >>> print(str(nltk.corpus.inaugural).replace('\\\\','/')) | |
| <PlaintextCorpusReader in '.../corpora/inaugural'...> | |
| Most corpora consist of a set of files, each containing a document (or | |
| other pieces of text). A list of identifiers for these files is | |
| accessed via the ``fileids()`` method of the corpus reader: | |
| >>> nltk.corpus.treebank.fileids() | |
| ['wsj_0001.mrg', 'wsj_0002.mrg', 'wsj_0003.mrg', 'wsj_0004.mrg', ...] | |
| >>> nltk.corpus.inaugural.fileids() | |
| ['1789-Washington.txt', '1793-Washington.txt', '1797-Adams.txt', ...] | |
| Each corpus reader provides a variety of methods to read data from the | |
| corpus, depending on the format of the corpus. For example, plaintext | |
| corpora support methods to read the corpus as raw text, a list of | |
| words, a list of sentences, or a list of paragraphs. | |
| >>> from nltk.corpus import inaugural | |
| >>> inaugural.raw('1789-Washington.txt') | |
| 'Fellow-Citizens of the Senate ...' | |
| >>> inaugural.words('1789-Washington.txt') | |
| ['Fellow', '-', 'Citizens', 'of', 'the', ...] | |
| >>> inaugural.sents('1789-Washington.txt') | |
| [['Fellow', '-', 'Citizens'...], ['Among', 'the', 'vicissitudes'...]...] | |
| >>> inaugural.paras('1789-Washington.txt') | |
| [[['Fellow', '-', 'Citizens'...]], | |
| [['Among', 'the', 'vicissitudes'...], | |
| ['On', 'the', 'one', 'hand', ',', 'I'...]...]...] | |
| Each of these reader methods may be given a single document's item | |
| name or a list of document item names. When given a list of document | |
| item names, the reader methods will concatenate together the contents | |
| of the individual documents. | |
| >>> l1 = len(inaugural.words('1789-Washington.txt')) | |
| >>> l2 = len(inaugural.words('1793-Washington.txt')) | |
| >>> l3 = len(inaugural.words(['1789-Washington.txt', '1793-Washington.txt'])) | |
| >>> print('%s+%s == %s' % (l1, l2, l3)) | |
| 1538+147 == 1685 | |
| If the reader methods are called without any arguments, they will | |
| typically load all documents in the corpus. | |
| >>> len(inaugural.words()) | |
| 152901 | |
| If a corpus contains a README file, it can be accessed with a ``readme()`` method: | |
| >>> inaugural.readme()[:32] | |
| 'C-Span Inaugural Address Corpus\n' | |
| Plaintext Corpora | |
| ================= | |
| Here are the first few words from each of NLTK's plaintext corpora: | |
| >>> nltk.corpus.abc.words() | |
| ['PM', 'denies', 'knowledge', 'of', 'AWB', ...] | |
| >>> nltk.corpus.genesis.words() | |
| ['In', 'the', 'beginning', 'God', 'created', ...] | |
| >>> nltk.corpus.gutenberg.words(fileids='austen-emma.txt') | |
| ['[', 'Emma', 'by', 'Jane', 'Austen', '1816', ...] | |
| >>> nltk.corpus.inaugural.words() | |
| ['Fellow', '-', 'Citizens', 'of', 'the', ...] | |
| >>> nltk.corpus.state_union.words() | |
| ['PRESIDENT', 'HARRY', 'S', '.', 'TRUMAN', "'", ...] | |
| >>> nltk.corpus.webtext.words() | |
| ['Cookie', 'Manager', ':', '"', 'Don', "'", 't', ...] | |
| Tagged Corpora | |
| ============== | |
| In addition to the plaintext corpora, NLTK's data package also | |
| contains a wide variety of annotated corpora. For example, the Brown | |
| Corpus is annotated with part-of-speech tags, and defines additional | |
| methods ``tagged_*()`` which words as `(word,tag)` tuples, rather | |
| than just bare word strings. | |
| >>> from nltk.corpus import brown | |
| >>> print(brown.words()) | |
| ['The', 'Fulton', 'County', 'Grand', 'Jury', ...] | |
| >>> print(brown.tagged_words()) | |
| [('The', 'AT'), ('Fulton', 'NP-TL'), ...] | |
| >>> print(brown.sents()) | |
| [['The', 'Fulton', 'County'...], ['The', 'jury', 'further'...], ...] | |
| >>> print(brown.tagged_sents()) | |
| [[('The', 'AT'), ('Fulton', 'NP-TL')...], | |
| [('The', 'AT'), ('jury', 'NN'), ('further', 'RBR')...]...] | |
| >>> print(brown.paras(categories='reviews')) | |
| [[['It', 'is', 'not', 'news', 'that', 'Nathan', 'Milstein'...], | |
| ['Certainly', 'not', 'in', 'Orchestra', 'Hall', 'where'...]], | |
| [['There', 'was', 'about', 'that', 'song', 'something', ...], | |
| ['Not', 'the', 'noblest', 'performance', 'we', 'have', ...], ...], ...] | |
| >>> print(brown.tagged_paras(categories='reviews')) | |
| [[[('It', 'PPS'), ('is', 'BEZ'), ('not', '*'), ...], | |
| [('Certainly', 'RB'), ('not', '*'), ('in', 'IN'), ...]], | |
| [[('There', 'EX'), ('was', 'BEDZ'), ('about', 'IN'), ...], | |
| [('Not', '*'), ('the', 'AT'), ('noblest', 'JJT'), ...], ...], ...] | |
| Similarly, the Indian Language POS-Tagged Corpus includes samples of | |
| Indian text annotated with part-of-speech tags: | |
| >>> from nltk.corpus import indian | |
| >>> print(indian.words()) # doctest: +SKIP | |
| ['\xe0\xa6\xae\xe0\xa6\xb9\xe0\xa6\xbf\...', | |
| '\xe0\xa6\xb8\xe0\xa6\xa8\xe0\xa7\x8d\xe0...', ...] | |
| >>> print(indian.tagged_words()) # doctest: +SKIP | |
| [('\xe0\xa6\xae\xe0\xa6\xb9\xe0\xa6\xbf...', 'NN'), | |
| ('\xe0\xa6\xb8\xe0\xa6\xa8\xe0\xa7\x8d\xe0...', 'NN'), ...] | |
| Several tagged corpora support access to a simplified, universal tagset, e.g. where all nouns | |
| tags are collapsed to a single category ``NOUN``: | |
| >>> print(brown.tagged_sents(tagset='universal')) | |
| [[('The', 'DET'), ('Fulton', 'NOUN'), ('County', 'NOUN'), ('Grand', 'ADJ'), ('Jury', 'NOUN'), ...], | |
| [('The', 'DET'), ('jury', 'NOUN'), ('further', 'ADV'), ('said', 'VERB'), ('in', 'ADP'), ...]...] | |
| >>> from nltk.corpus import conll2000, switchboard | |
| >>> print(conll2000.tagged_words(tagset='universal')) | |
| [('Confidence', 'NOUN'), ('in', 'ADP'), ...] | |
| Use ``nltk.app.pos_concordance()`` to access a GUI for searching tagged corpora. | |
| Chunked Corpora | |
| =============== | |
| The CoNLL corpora also provide chunk structures, which are encoded as | |
| flat trees. The CoNLL 2000 Corpus includes phrasal chunks; and the | |
| CoNLL 2002 Corpus includes named entity chunks. | |
| >>> from nltk.corpus import conll2000, conll2002 | |
| >>> print(conll2000.sents()) | |
| [['Confidence', 'in', 'the', 'pound', 'is', 'widely', ...], | |
| ['Chancellor', 'of', 'the', 'Exchequer', ...], ...] | |
| >>> for tree in conll2000.chunked_sents()[:2]: | |
| ... print(tree) | |
| (S | |
| (NP Confidence/NN) | |
| (PP in/IN) | |
| (NP the/DT pound/NN) | |
| (VP is/VBZ widely/RB expected/VBN to/TO take/VB) | |
| (NP another/DT sharp/JJ dive/NN) | |
| if/IN | |
| ...) | |
| (S | |
| Chancellor/NNP | |
| (PP of/IN) | |
| (NP the/DT Exchequer/NNP) | |
| ...) | |
| >>> print(conll2002.sents()) | |
| [['Sao', 'Paulo', '(', 'Brasil', ')', ',', ...], ['-'], ...] | |
| >>> for tree in conll2002.chunked_sents()[:2]: | |
| ... print(tree) | |
| (S | |
| (LOC Sao/NC Paulo/VMI) | |
| (/Fpa | |
| (LOC Brasil/NC) | |
| )/Fpt | |
| ...) | |
| (S -/Fg) | |
| .. note:: Since the CONLL corpora do not contain paragraph break | |
| information, these readers do not support the ``para()`` method.) | |
| .. warning:: if you call the conll corpora reader methods without any | |
| arguments, they will return the contents of the entire corpus, | |
| *including* the 'test' portions of the corpus.) | |
| SemCor is a subset of the Brown corpus tagged with WordNet senses and | |
| named entities. Both kinds of lexical items include multiword units, | |
| which are encoded as chunks (senses and part-of-speech tags pertain | |
| to the entire chunk). | |
| >>> from nltk.corpus import semcor | |
| >>> semcor.words() | |
| ['The', 'Fulton', 'County', 'Grand', 'Jury', ...] | |
| >>> semcor.chunks() | |
| [['The'], ['Fulton', 'County', 'Grand', 'Jury'], ...] | |
| >>> semcor.sents() | |
| [['The', 'Fulton', 'County', 'Grand', 'Jury', 'said', ...], | |
| ['The', 'jury', 'further', 'said', ...], ...] | |
| >>> semcor.chunk_sents() | |
| [[['The'], ['Fulton', 'County', 'Grand', 'Jury'], ['said'], ... | |
| ['.']], [['The'], ['jury'], ['further'], ['said'], ... ['.']], ...] | |
| >>> list(map(str, semcor.tagged_chunks(tag='both')[:3])) | |
| ['(DT The)', "(Lemma('group.n.01.group') (NE (NNP Fulton County Grand Jury)))", "(Lemma('state.v.01.say') (VB said))"] | |
| >>> [[str(c) for c in s] for s in semcor.tagged_sents(tag='both')[:2]] | |
| [['(DT The)', "(Lemma('group.n.01.group') (NE (NNP Fulton County Grand Jury)))", ... | |
| '(None .)'], ['(DT The)', ... '(None .)']] | |
| The IEER corpus is another chunked corpus. This corpus is unusual in | |
| that each corpus item contains multiple documents. (This reflects the | |
| fact that each corpus file contains multiple documents.) The IEER | |
| corpus defines the `parsed_docs` method, which returns the documents | |
| in a given item as `IEERDocument` objects: | |
| >>> from nltk.corpus import ieer | |
| >>> ieer.fileids() | |
| ['APW_19980314', 'APW_19980424', 'APW_19980429', | |
| 'NYT_19980315', 'NYT_19980403', 'NYT_19980407'] | |
| >>> docs = ieer.parsed_docs('APW_19980314') | |
| >>> print(docs[0]) | |
| <IEERDocument APW19980314.0391: 'Kenyans protest tax hikes'> | |
| >>> print(docs[0].docno) | |
| APW19980314.0391 | |
| >>> print(docs[0].doctype) | |
| NEWS STORY | |
| >>> print(docs[0].date_time) | |
| 03/14/1998 10:36:00 | |
| >>> print(docs[0].headline) | |
| (DOCUMENT Kenyans protest tax hikes) | |
| >>> print(docs[0].text) | |
| (DOCUMENT | |
| (LOCATION NAIROBI) | |
| , | |
| (LOCATION Kenya) | |
| ( | |
| (ORGANIZATION AP) | |
| ) | |
| _ | |
| (CARDINAL Thousands) | |
| of | |
| laborers, | |
| ... | |
| on | |
| (DATE Saturday) | |
| ...) | |
| Parsed Corpora | |
| ============== | |
| The Treebank corpora provide a syntactic parse for each sentence. The | |
| NLTK data package includes a 10% sample of the Penn Treebank (in | |
| ``treebank``), as well as the Sinica Treebank (in ``sinica_treebank``). | |
| Reading the Penn Treebank (Wall Street Journal sample): | |
| >>> from nltk.corpus import treebank | |
| >>> print(treebank.fileids()) | |
| ['wsj_0001.mrg', 'wsj_0002.mrg', 'wsj_0003.mrg', 'wsj_0004.mrg', ...] | |
| >>> print(treebank.words('wsj_0003.mrg')) | |
| ['A', 'form', 'of', 'asbestos', 'once', 'used', ...] | |
| >>> print(treebank.tagged_words('wsj_0003.mrg')) | |
| [('A', 'DT'), ('form', 'NN'), ('of', 'IN'), ...] | |
| >>> print(treebank.parsed_sents('wsj_0003.mrg')[0]) | |
| (S | |
| (S-TPC-1 | |
| (NP-SBJ | |
| (NP (NP (DT A) (NN form)) (PP (IN of) (NP (NN asbestos)))) | |
| (RRC ...)...)...) | |
| ... | |
| (VP (VBD reported) (SBAR (-NONE- 0) (S (-NONE- *T*-1)))) | |
| (. .)) | |
| If you have access to a full installation of the Penn Treebank, NLTK | |
| can be configured to load it as well. Download the ``ptb`` package, | |
| and in the directory ``nltk_data/corpora/ptb`` place the ``BROWN`` | |
| and ``WSJ`` directories of the Treebank installation (symlinks work | |
| as well). Then use the ``ptb`` module instead of ``treebank``: | |
| >>> from nltk.corpus import ptb | |
| >>> print(ptb.fileids()) # doctest: +SKIP | |
| ['BROWN/CF/CF01.MRG', 'BROWN/CF/CF02.MRG', 'BROWN/CF/CF03.MRG', 'BROWN/CF/CF04.MRG', ...] | |
| >>> print(ptb.words('WSJ/00/WSJ_0003.MRG')) # doctest: +SKIP | |
| ['A', 'form', 'of', 'asbestos', 'once', 'used', '*', ...] | |
| >>> print(ptb.tagged_words('WSJ/00/WSJ_0003.MRG')) # doctest: +SKIP | |
| [('A', 'DT'), ('form', 'NN'), ('of', 'IN'), ...] | |
| ...and so forth, like ``treebank`` but with extended fileids. Categories | |
| specified in ``allcats.txt`` can be used to filter by genre; they consist | |
| of ``news`` (for WSJ articles) and names of the Brown subcategories | |
| (``fiction``, ``humor``, ``romance``, etc.): | |
| >>> ptb.categories() # doctest: +SKIP | |
| ['adventure', 'belles_lettres', 'fiction', 'humor', 'lore', 'mystery', 'news', 'romance', 'science_fiction'] | |
| >>> print(ptb.fileids('news')) # doctest: +SKIP | |
| ['WSJ/00/WSJ_0001.MRG', 'WSJ/00/WSJ_0002.MRG', 'WSJ/00/WSJ_0003.MRG', ...] | |
| >>> print(ptb.words(categories=['humor','fiction'])) # doctest: +SKIP | |
| ['Thirty-three', 'Scotty', 'did', 'not', 'go', 'back', ...] | |
| As PropBank and NomBank depend on the (WSJ portion of the) Penn Treebank, | |
| the modules ``propbank_ptb`` and ``nombank_ptb`` are provided for access | |
| to a full PTB installation. | |
| Reading the Sinica Treebank: | |
| >>> from nltk.corpus import sinica_treebank | |
| >>> print(sinica_treebank.sents()) # doctest: +SKIP | |
| [['\xe4\xb8\x80'], ['\xe5\x8f\x8b\xe6\x83\x85'], ...] | |
| >>> sinica_treebank.parsed_sents()[25] # doctest: +SKIP | |
| Tree('S', | |
| [Tree('NP', | |
| [Tree('Nba', ['\xe5\x98\x89\xe7\x8f\x8d'])]), | |
| Tree('V\xe2\x80\xa7\xe5\x9c\xb0', | |
| [Tree('VA11', ['\xe4\xb8\x8d\xe5\x81\x9c']), | |
| Tree('DE', ['\xe7\x9a\x84'])]), | |
| Tree('VA4', ['\xe5\x93\xad\xe6\xb3\xa3'])]) | |
| Reading the CoNLL 2007 Dependency Treebanks: | |
| >>> from nltk.corpus import conll2007 | |
| >>> conll2007.sents('esp.train')[0] # doctest: +SKIP | |
| ['El', 'aumento', 'del', 'índice', 'de', 'desempleo', ...] | |
| >>> conll2007.parsed_sents('esp.train')[0] # doctest: +SKIP | |
| <DependencyGraph with 38 nodes> | |
| >>> print(conll2007.parsed_sents('esp.train')[0].tree()) # doctest: +SKIP | |
| (fortaleció | |
| (aumento El (del (índice (de (desempleo estadounidense))))) | |
| hoy | |
| considerablemente | |
| (al | |
| (euro | |
| (cotizaba | |
| , | |
| que | |
| (a (15.35 las GMT)) | |
| se | |
| (en (mercado el (de divisas) (de Fráncfort))) | |
| (a 0,9452_dólares) | |
| (frente_a , (0,9349_dólares los (de (mañana esta))))))) | |
| .) | |
| Word Lists and Lexicons | |
| ======================= | |
| The NLTK data package also includes a number of lexicons and word | |
| lists. These are accessed just like text corpora. The following | |
| examples illustrate the use of the wordlist corpora: | |
| >>> from nltk.corpus import names, stopwords, words | |
| >>> words.fileids() | |
| ['en', 'en-basic'] | |
| >>> words.words('en') | |
| ['A', 'a', 'aa', 'aal', 'aalii', 'aam', 'Aani', 'aardvark', 'aardwolf', ...] | |
| >>> stopwords.fileids() # doctest: +SKIP | |
| ['arabic', 'azerbaijani', 'bengali', 'danish', 'dutch', 'english', 'finnish', 'french', ...] | |
| >>> sorted(stopwords.words('portuguese')) | |
| ['a', 'ao', 'aos', 'aquela', 'aquelas', 'aquele', 'aqueles', ...] | |
| >>> names.fileids() | |
| ['female.txt', 'male.txt'] | |
| >>> names.words('male.txt') | |
| ['Aamir', 'Aaron', 'Abbey', 'Abbie', 'Abbot', 'Abbott', ...] | |
| >>> names.words('female.txt') | |
| ['Abagael', 'Abagail', 'Abbe', 'Abbey', 'Abbi', 'Abbie', ...] | |
| The CMU Pronunciation Dictionary corpus contains pronunciation | |
| transcriptions for over 100,000 words. It can be accessed as a list | |
| of entries (where each entry consists of a word, an identifier, and a | |
| transcription) or as a dictionary from words to lists of | |
| transcriptions. Transcriptions are encoded as tuples of phoneme | |
| strings. | |
| >>> from nltk.corpus import cmudict | |
| >>> print(cmudict.entries()[653:659]) | |
| [('acetate', ['AE1', 'S', 'AH0', 'T', 'EY2', 'T']), | |
| ('acetic', ['AH0', 'S', 'EH1', 'T', 'IH0', 'K']), | |
| ('acetic', ['AH0', 'S', 'IY1', 'T', 'IH0', 'K']), | |
| ('aceto', ['AA0', 'S', 'EH1', 'T', 'OW0']), | |
| ('acetochlor', ['AA0', 'S', 'EH1', 'T', 'OW0', 'K', 'L', 'AO2', 'R']), | |
| ('acetone', ['AE1', 'S', 'AH0', 'T', 'OW2', 'N'])] | |
| >>> # Load the entire cmudict corpus into a Python dictionary: | |
| >>> transcr = cmudict.dict() | |
| >>> print([transcr[w][0] for w in 'Natural Language Tool Kit'.lower().split()]) | |
| [['N', 'AE1', 'CH', 'ER0', 'AH0', 'L'], | |
| ['L', 'AE1', 'NG', 'G', 'W', 'AH0', 'JH'], | |
| ['T', 'UW1', 'L'], | |
| ['K', 'IH1', 'T']] | |
| WordNet | |
| ======= | |
| Please see the separate WordNet howto. | |
| FrameNet | |
| ======== | |
| Please see the separate FrameNet howto. | |
| PropBank | |
| ======== | |
| Please see the separate PropBank howto. | |
| SentiWordNet | |
| ============ | |
| Please see the separate SentiWordNet howto. | |
| Categorized Corpora | |
| =================== | |
| Several corpora included with NLTK contain documents that have been categorized for | |
| topic, genre, polarity, etc. In addition to the standard corpus interface, these | |
| corpora provide access to the list of categories and the mapping between the documents | |
| and their categories (in both directions). Access the categories using the ``categories()`` | |
| method, e.g.: | |
| >>> from nltk.corpus import brown, movie_reviews, reuters | |
| >>> brown.categories() | |
| ['adventure', 'belles_lettres', 'editorial', 'fiction', 'government', 'hobbies', 'humor', | |
| 'learned', 'lore', 'mystery', 'news', 'religion', 'reviews', 'romance', 'science_fiction'] | |
| >>> movie_reviews.categories() | |
| ['neg', 'pos'] | |
| >>> reuters.categories() | |
| ['acq', 'alum', 'barley', 'bop', 'carcass', 'castor-oil', 'cocoa', | |
| 'coconut', 'coconut-oil', 'coffee', 'copper', 'copra-cake', 'corn', | |
| 'cotton', 'cotton-oil', 'cpi', 'cpu', 'crude', 'dfl', 'dlr', ...] | |
| This method has an optional argument that specifies a document or a list | |
| of documents, allowing us to map from (one or more) documents to (one or more) categories: | |
| >>> brown.categories('ca01') | |
| ['news'] | |
| >>> brown.categories(['ca01','cb01']) | |
| ['editorial', 'news'] | |
| >>> reuters.categories('training/9865') | |
| ['barley', 'corn', 'grain', 'wheat'] | |
| >>> reuters.categories(['training/9865', 'training/9880']) | |
| ['barley', 'corn', 'grain', 'money-fx', 'wheat'] | |
| We can go back the other way using the optional argument of the ``fileids()`` method: | |
| >>> reuters.fileids('barley') | |
| ['test/15618', 'test/15649', 'test/15676', 'test/15728', 'test/15871', ...] | |
| Both the ``categories()`` and ``fileids()`` methods return a sorted list containing | |
| no duplicates. | |
| In addition to mapping between categories and documents, these corpora permit | |
| direct access to their contents via the categories. Instead of accessing a subset | |
| of a corpus by specifying one or more fileids, we can identify one or more categories, e.g.: | |
| >>> brown.tagged_words(categories='news') | |
| [('The', 'AT'), ('Fulton', 'NP-TL'), ...] | |
| >>> brown.sents(categories=['editorial','reviews']) | |
| [['Assembly', 'session', 'brought', 'much', 'good'], ['The', 'General', | |
| 'Assembly', ',', 'which', 'adjourns', 'today', ',', 'has', 'performed', | |
| 'in', 'an', 'atmosphere', 'of', 'crisis', 'and', 'struggle', 'from', | |
| 'the', 'day', 'it', 'convened', '.'], ...] | |
| Note that it is an error to specify both documents and categories. | |
| In the context of a text categorization system, we can easily test if the | |
| category assigned to a document is correct as follows: | |
| >>> def classify(doc): return 'news' # Trivial classifier | |
| >>> doc = 'ca01' | |
| >>> classify(doc) in brown.categories(doc) | |
| True | |
| Other Corpora | |
| ============= | |
| comparative_sentences | |
| --------------------- | |
| A list of sentences from various sources, especially reviews and articles. Each | |
| line contains one sentence; sentences were separated by using a sentence tokenizer. | |
| Comparative sentences have been annotated with their type, entities, features and | |
| keywords. | |
| >>> from nltk.corpus import comparative_sentences | |
| >>> comparison = comparative_sentences.comparisons()[0] | |
| >>> comparison.text | |
| ['its', 'fast-forward', 'and', 'rewind', 'work', 'much', 'more', 'smoothly', | |
| 'and', 'consistently', 'than', 'those', 'of', 'other', 'models', 'i', "'ve", | |
| 'had', '.'] | |
| >>> comparison.entity_2 | |
| 'models' | |
| >>> (comparison.feature, comparison.keyword) | |
| ('rewind', 'more') | |
| >>> len(comparative_sentences.comparisons()) | |
| 853 | |
| opinion_lexicon | |
| --------------- | |
| A list of positive and negative opinion words or sentiment words for English. | |
| >>> from nltk.corpus import opinion_lexicon | |
| >>> opinion_lexicon.words()[:4] | |
| ['2-faced', '2-faces', 'abnormal', 'abolish'] | |
| The OpinionLexiconCorpusReader also provides shortcuts to retrieve positive/negative | |
| words: | |
| >>> opinion_lexicon.negative()[:4] | |
| ['2-faced', '2-faces', 'abnormal', 'abolish'] | |
| Note that words from `words()` method in opinion_lexicon are sorted by file id, | |
| not alphabetically: | |
| >>> opinion_lexicon.words()[0:10] | |
| ['2-faced', '2-faces', 'abnormal', 'abolish', 'abominable', 'abominably', | |
| 'abominate', 'abomination', 'abort', 'aborted'] | |
| >>> sorted(opinion_lexicon.words())[0:10] | |
| ['2-faced', '2-faces', 'a+', 'abnormal', 'abolish', 'abominable', 'abominably', | |
| 'abominate', 'abomination', 'abort'] | |
| ppattach | |
| -------- | |
| The Prepositional Phrase Attachment corpus is a corpus of | |
| prepositional phrase attachment decisions. Each instance in the | |
| corpus is encoded as a ``PPAttachment`` object: | |
| >>> from nltk.corpus import ppattach | |
| >>> ppattach.attachments('training') | |
| [PPAttachment(sent='0', verb='join', noun1='board', | |
| prep='as', noun2='director', attachment='V'), | |
| PPAttachment(sent='1', verb='is', noun1='chairman', | |
| prep='of', noun2='N.V.', attachment='N'), | |
| ...] | |
| >>> inst = ppattach.attachments('training')[0] | |
| >>> (inst.sent, inst.verb, inst.noun1, inst.prep, inst.noun2) | |
| ('0', 'join', 'board', 'as', 'director') | |
| >>> inst.attachment | |
| 'V' | |
| product_reviews_1 and product_reviews_2 | |
| --------------------------------------- | |
| These two datasets respectively contain annotated customer reviews of 5 and 9 | |
| products from amazon.com. | |
| >>> from nltk.corpus import product_reviews_1 | |
| >>> camera_reviews = product_reviews_1.reviews('Canon_G3.txt') | |
| >>> review = camera_reviews[0] | |
| >>> review.sents()[0] | |
| ['i', 'recently', 'purchased', 'the', 'canon', 'powershot', 'g3', 'and', 'am', | |
| 'extremely', 'satisfied', 'with', 'the', 'purchase', '.'] | |
| >>> review.features() | |
| [('canon powershot g3', '+3'), ('use', '+2'), ('picture', '+2'), | |
| ('picture quality', '+1'), ('picture quality', '+1'), ('camera', '+2'), | |
| ('use', '+2'), ('feature', '+1'), ('picture quality', '+3'), ('use', '+1'), | |
| ('option', '+1')] | |
| It is also possible to reach the same information directly from the stream: | |
| >>> product_reviews_1.features('Canon_G3.txt') | |
| [('canon powershot g3', '+3'), ('use', '+2'), ...] | |
| We can compute stats for specific product features: | |
| >>> n_reviews = len([(feat,score) for (feat,score) in product_reviews_1.features('Canon_G3.txt') if feat=='picture']) | |
| >>> tot = sum([int(score) for (feat,score) in product_reviews_1.features('Canon_G3.txt') if feat=='picture']) | |
| >>> mean = tot / n_reviews | |
| >>> print(n_reviews, tot, mean) | |
| 15 24 1.6 | |
| pros_cons | |
| --------- | |
| A list of pros/cons sentences for determining context (aspect) dependent | |
| sentiment words, which are then applied to sentiment analysis of comparative | |
| sentences. | |
| >>> from nltk.corpus import pros_cons | |
| >>> pros_cons.sents(categories='Cons') | |
| [['East', 'batteries', '!', 'On', '-', 'off', 'switch', 'too', 'easy', | |
| 'to', 'maneuver', '.'], ['Eats', '...', 'no', ',', 'GULPS', 'batteries'], | |
| ...] | |
| >>> pros_cons.words('IntegratedPros.txt') | |
| ['Easy', 'to', 'use', ',', 'economical', '!', ...] | |
| semcor | |
| ------ | |
| The Brown Corpus, annotated with WordNet senses. | |
| >>> from nltk.corpus import semcor | |
| >>> semcor.words('brown2/tagfiles/br-n12.xml') | |
| ['When', 'several', 'minutes', 'had', 'passed', ...] | |
| senseval | |
| -------- | |
| The Senseval 2 corpus is a word sense disambiguation corpus. Each | |
| item in the corpus corresponds to a single ambiguous word. For each | |
| of these words, the corpus contains a list of instances, corresponding | |
| to occurrences of that word. Each instance provides the word; a list | |
| of word senses that apply to the word occurrence; and the word's | |
| context. | |
| >>> from nltk.corpus import senseval | |
| >>> senseval.fileids() | |
| ['hard.pos', 'interest.pos', 'line.pos', 'serve.pos'] | |
| >>> senseval.instances('hard.pos') | |
| ... | |
| [SensevalInstance(word='hard-a', | |
| position=20, | |
| context=[('``', '``'), ('he', 'PRP'), ...('hard', 'JJ'), ...], | |
| senses=('HARD1',)), | |
| SensevalInstance(word='hard-a', | |
| position=10, | |
| context=[('clever', 'NNP'), ...('hard', 'JJ'), ('time', 'NN'), ...], | |
| senses=('HARD1',)), ...] | |
| The following code looks at instances of the word 'interest', and | |
| displays their local context (2 words on each side) and word sense(s): | |
| >>> for inst in senseval.instances('interest.pos')[:10]: | |
| ... p = inst.position | |
| ... left = ' '.join(w for (w,t) in inst.context[p-2:p]) | |
| ... word = ' '.join(w for (w,t) in inst.context[p:p+1]) | |
| ... right = ' '.join(w for (w,t) in inst.context[p+1:p+3]) | |
| ... senses = ' '.join(inst.senses) | |
| ... print('%20s |%10s | %-15s -> %s' % (left, word, right, senses)) | |
| declines in | interest | rates . -> interest_6 | |
| indicate declining | interest | rates because -> interest_6 | |
| in short-term | interest | rates . -> interest_6 | |
| 4 % | interest | in this -> interest_5 | |
| company with | interests | in the -> interest_5 | |
| , plus | interest | . -> interest_6 | |
| set the | interest | rate on -> interest_6 | |
| 's own | interest | , prompted -> interest_4 | |
| principal and | interest | is the -> interest_6 | |
| increase its | interest | to 70 -> interest_5 | |
| sentence_polarity | |
| ----------------- | |
| The Sentence Polarity dataset contains 5331 positive and 5331 negative processed | |
| sentences. | |
| >>> from nltk.corpus import sentence_polarity | |
| >>> sentence_polarity.sents() | |
| [['simplistic', ',', 'silly', 'and', 'tedious', '.'], ["it's", 'so', 'laddish', | |
| 'and', 'juvenile', ',', 'only', 'teenage', 'boys', 'could', 'possibly', 'find', | |
| 'it', 'funny', '.'], ...] | |
| >>> sentence_polarity.categories() | |
| ['neg', 'pos'] | |
| >>> sentence_polarity.sents()[1] | |
| ["it's", 'so', 'laddish', 'and', 'juvenile', ',', 'only', 'teenage', 'boys', | |
| 'could', 'possibly', 'find', 'it', 'funny', '.'] | |
| shakespeare | |
| ----------- | |
| The Shakespeare corpus contains a set of Shakespeare plays, formatted | |
| as XML files. These corpora are returned as ElementTree objects: | |
| >>> from nltk.corpus import shakespeare | |
| >>> from xml.etree import ElementTree | |
| >>> shakespeare.fileids() | |
| ['a_and_c.xml', 'dream.xml', 'hamlet.xml', 'j_caesar.xml', ...] | |
| >>> play = shakespeare.xml('dream.xml') | |
| >>> print(play) | |
| <Element 'PLAY' at ...> | |
| >>> print('%s: %s' % (play[0].tag, play[0].text)) | |
| TITLE: A Midsummer Night's Dream | |
| >>> personae = [persona.text for persona in | |
| ... play.findall('PERSONAE/PERSONA')] | |
| >>> print(personae) | |
| ['THESEUS, Duke of Athens.', 'EGEUS, father to Hermia.', ...] | |
| >>> # Find and print speakers not listed as personae | |
| >>> names = [persona.split(',')[0] for persona in personae] | |
| >>> speakers = set(speaker.text for speaker in | |
| ... play.findall('*/*/*/SPEAKER')) | |
| >>> print(sorted(speakers.difference(names))) | |
| ['ALL', 'COBWEB', 'DEMETRIUS', 'Fairy', 'HERNIA', 'LYSANDER', | |
| 'Lion', 'MOTH', 'MUSTARDSEED', 'Moonshine', 'PEASEBLOSSOM', | |
| 'Prologue', 'Pyramus', 'Thisbe', 'Wall'] | |
| subjectivity | |
| ------------ | |
| The Subjectivity Dataset contains 5000 subjective and 5000 objective processed | |
| sentences. | |
| >>> from nltk.corpus import subjectivity | |
| >>> subjectivity.categories() | |
| ['obj', 'subj'] | |
| >>> subjectivity.sents()[23] | |
| ['television', 'made', 'him', 'famous', ',', 'but', 'his', 'biggest', 'hits', | |
| 'happened', 'off', 'screen', '.'] | |
| >>> subjectivity.words(categories='subj') | |
| ['smart', 'and', 'alert', ',', 'thirteen', ...] | |
| toolbox | |
| ------- | |
| The Toolbox corpus distributed with NLTK contains a sample lexicon and | |
| several sample texts from the Rotokas language. The Toolbox corpus | |
| reader returns Toolbox files as XML ElementTree objects. The | |
| following example loads the Rotokas dictionary, and figures out the | |
| distribution of part-of-speech tags for reduplicated words. | |
| .. doctest: +SKIP | |
| >>> from nltk.corpus import toolbox | |
| >>> from nltk.probability import FreqDist | |
| >>> from xml.etree import ElementTree | |
| >>> import re | |
| >>> rotokas = toolbox.xml('rotokas.dic') | |
| >>> redup_pos_freqdist = FreqDist() | |
| >>> # Note: we skip over the first record, which is actually | |
| >>> # the header. | |
| >>> for record in rotokas[1:]: | |
| ... lexeme = record.find('lx').text | |
| ... if re.match(r'(.*)\1$', lexeme): | |
| ... redup_pos_freqdist[record.find('ps').text] += 1 | |
| >>> for item, count in redup_pos_freqdist.most_common(): | |
| ... print(item, count) | |
| V 41 | |
| N 14 | |
| ??? 4 | |
| This example displays some records from a Rotokas text: | |
| .. doctest: +SKIP | |
| >>> river = toolbox.xml('rotokas/river.txt', key='ref') | |
| >>> for record in river.findall('record')[:3]: | |
| ... for piece in record: | |
| ... if len(piece.text) > 60: | |
| ... print('%-6s %s...' % (piece.tag, piece.text[:57])) | |
| ... else: | |
| ... print('%-6s %s' % (piece.tag, piece.text)) | |
| ref Paragraph 1 | |
| t ``Viapau oisio ra ovaupasi ... | |
| m viapau oisio ra ovau -pa -si ... | |
| g NEG this way/like this and forget -PROG -2/3.DL... | |
| p NEG ??? CONJ V.I -SUFF.V.3 -SUFF.V... | |
| f ``No ken lus tingting wanema samting papa i bin tok,'' Na... | |
| fe ``Don't forget what Dad said,'' yelled Naomi. | |
| ref 2 | |
| t Osa Ira ora Reviti viapau uvupasiva. | |
| m osa Ira ora Reviti viapau uvu -pa -si ... | |
| g as/like name and name NEG hear/smell -PROG -2/3... | |
| p CONJ N.PN CONJ N.PN NEG V.T -SUFF.V.3 -SUF... | |
| f Tasol Ila na David no bin harim toktok. | |
| fe But Ila and David took no notice. | |
| ref 3 | |
| t Ikaupaoro rokosiva ... | |
| m ikau -pa -oro roko -si -va ... | |
| g run/hurry -PROG -SIM go down -2/3.DL.M -RP ... | |
| p V.T -SUFF.V.3 -SUFF.V.4 ADV -SUFF.V.4 -SUFF.VT.... | |
| f Tupela i bin hariap i go long wara . | |
| fe They raced to the river. | |
| timit | |
| ----- | |
| The NLTK data package includes a fragment of the TIMIT | |
| Acoustic-Phonetic Continuous Speech Corpus. This corpus is broken | |
| down into small speech samples, each of which is available as a wave | |
| file, a phonetic transcription, and a tokenized word list. | |
| >>> from nltk.corpus import timit | |
| >>> print(timit.utteranceids()) | |
| ['dr1-fvmh0/sa1', 'dr1-fvmh0/sa2', 'dr1-fvmh0/si1466', | |
| 'dr1-fvmh0/si2096', 'dr1-fvmh0/si836', 'dr1-fvmh0/sx116', | |
| 'dr1-fvmh0/sx206', 'dr1-fvmh0/sx26', 'dr1-fvmh0/sx296', ...] | |
| >>> item = timit.utteranceids()[5] | |
| >>> print(timit.phones(item)) | |
| ['h#', 'k', 'l', 'ae', 's', 'pcl', 'p', 'dh', 'ax', | |
| 's', 'kcl', 'k', 'r', 'ux', 'ix', 'nx', 'y', 'ax', | |
| 'l', 'eh', 'f', 'tcl', 't', 'hh', 'ae', 'n', 'dcl', | |
| 'd', 'h#'] | |
| >>> print(timit.words(item)) | |
| ['clasp', 'the', 'screw', 'in', 'your', 'left', 'hand'] | |
| >>> timit.play(item) # doctest: +SKIP | |
| The corpus reader can combine the word segmentation information with | |
| the phonemes to produce a single tree structure: | |
| >>> for tree in timit.phone_trees(item): | |
| ... print(tree) | |
| (S | |
| h# | |
| (clasp k l ae s pcl p) | |
| (the dh ax) | |
| (screw s kcl k r ux) | |
| (in ix nx) | |
| (your y ax) | |
| (left l eh f tcl t) | |
| (hand hh ae n dcl d) | |
| h#) | |
| The start time and stop time of each phoneme, word, and sentence are | |
| also available: | |
| >>> print(timit.phone_times(item)) | |
| [('h#', 0, 2190), ('k', 2190, 3430), ('l', 3430, 4326), ...] | |
| >>> print(timit.word_times(item)) | |
| [('clasp', 2190, 8804), ('the', 8804, 9734), ...] | |
| >>> print(timit.sent_times(item)) | |
| [('Clasp the screw in your left hand.', 0, 32154)] | |
| We can use these times to play selected pieces of a speech sample: | |
| >>> timit.play(item, 2190, 8804) # 'clasp' # doctest: +SKIP | |
| The corpus reader can also be queried for information about the | |
| speaker and sentence identifier for a given speech sample: | |
| >>> print(timit.spkrid(item)) | |
| dr1-fvmh0 | |
| >>> print(timit.sentid(item)) | |
| sx116 | |
| >>> print(timit.spkrinfo(timit.spkrid(item))) | |
| SpeakerInfo(id='VMH0', | |
| sex='F', | |
| dr='1', | |
| use='TRN', | |
| recdate='03/11/86', | |
| birthdate='01/08/60', | |
| ht='5\'05"', | |
| race='WHT', | |
| edu='BS', | |
| comments='BEST NEW ENGLAND ACCENT SO FAR') | |
| >>> # List the speech samples from the same speaker: | |
| >>> timit.utteranceids(spkrid=timit.spkrid(item)) | |
| ['dr1-fvmh0/sa1', 'dr1-fvmh0/sa2', 'dr1-fvmh0/si1466', ...] | |
| twitter_samples | |
| --------------- | |
| Twitter is well-known microblog service that allows public data to be | |
| collected via APIs. NLTK's twitter corpus currently contains a sample of 20k Tweets | |
| retrieved from the Twitter Streaming API. | |
| >>> from nltk.corpus import twitter_samples | |
| >>> twitter_samples.fileids() | |
| ['negative_tweets.json', 'positive_tweets.json', 'tweets.20150430-223406.json'] | |
| We follow standard practice in storing full Tweets as line-separated | |
| JSON. These data structures can be accessed via `tweets.docs()`. However, in general it | |
| is more practical to focus just on the text field of the Tweets, which | |
| are accessed via the `strings()` method. | |
| >>> twitter_samples.strings('tweets.20150430-223406.json')[:5] | |
| ['RT @KirkKus: Indirect cost of the UK being in the EU is estimated to be costing Britain \xa3170 billion per year! #BetterOffOut #UKIP', ...] | |
| The default tokenizer for Tweets is specialised for 'casual' text, and | |
| the `tokenized()` method returns a list of lists of tokens. | |
| >>> twitter_samples.tokenized('tweets.20150430-223406.json')[:5] | |
| [['RT', '@KirkKus', ':', 'Indirect', 'cost', 'of', 'the', 'UK', 'being', 'in', ...], | |
| ['VIDEO', ':', 'Sturgeon', 'on', 'post-election', 'deals', 'http://t.co/BTJwrpbmOY'], ...] | |
| rte | |
| --- | |
| The RTE (Recognizing Textual Entailment) corpus was derived from the | |
| RTE1, RTE2 and RTE3 datasets (dev and test data), and consists of a | |
| list of XML-formatted 'text'/'hypothesis' pairs. | |
| >>> from nltk.corpus import rte | |
| >>> print(rte.fileids()) | |
| ['rte1_dev.xml', 'rte1_test.xml', 'rte2_dev.xml', ..., 'rte3_test.xml'] | |
| >>> rtepairs = rte.pairs(['rte2_test.xml', 'rte3_test.xml']) | |
| >>> print(rtepairs) | |
| [<RTEPair: gid=2-8>, <RTEPair: gid=2-9>, <RTEPair: gid=2-15>, ...] | |
| In the gold standard test sets, each pair is labeled according to | |
| whether or not the text 'entails' the hypothesis; the | |
| entailment value is mapped to an integer 1 (True) or 0 (False). | |
| >>> rtepairs[5] | |
| <RTEPair: gid=2-23> | |
| >>> rtepairs[5].text | |
| 'His wife Strida won a seat in parliament after forging an alliance | |
| with the main anti-Syrian coalition in the recent election.' | |
| >>> rtepairs[5].hyp | |
| 'Strida elected to parliament.' | |
| >>> rtepairs[5].value | |
| 1 | |
| The RTE corpus also supports an ``xml()`` method which produces ElementTrees. | |
| >>> xmltree = rte.xml('rte3_dev.xml') | |
| >>> xmltree # doctest: +SKIP | |
| <Element entailment-corpus at ...> | |
| >>> xmltree[7].findtext('t') | |
| "Mrs. Bush's approval ratings have remained very high, above 80%, | |
| even as her husband's have recently dropped below 50%." | |
| verbnet | |
| ------- | |
| The VerbNet corpus is a lexicon that divides verbs into classes, based | |
| on their syntax-semantics linking behavior. The basic elements in the | |
| lexicon are verb lemmas, such as 'abandon' and 'accept', and verb | |
| classes, which have identifiers such as 'remove-10.1' and | |
| 'admire-31.2-1'. These class identifiers consist of a representative | |
| verb selected from the class, followed by a numerical identifier. The | |
| list of verb lemmas, and the list of class identifiers, can be | |
| retrieved with the following methods: | |
| >>> from nltk.corpus import verbnet | |
| >>> verbnet.lemmas()[20:25] | |
| ['accelerate', 'accept', 'acclaim', 'accompany', 'accrue'] | |
| >>> verbnet.classids()[:5] | |
| ['accompany-51.7', 'admire-31.2', 'admire-31.2-1', 'admit-65', 'adopt-93'] | |
| The `classids()` method may also be used to retrieve the classes that | |
| a given lemma belongs to: | |
| >>> verbnet.classids('accept') | |
| ['approve-77', 'characterize-29.2-1-1', 'obtain-13.5.2'] | |
| The `classids()` method may additionally be used to retrieve all classes | |
| within verbnet if nothing is passed: | |
| >>> verbnet.classids() | |
| ['accompany-51.7', 'admire-31.2', 'admire-31.2-1', 'admit-65', 'adopt-93', 'advise-37.9', 'advise-37.9-1', 'allow-64', 'amalgamate-22.2', 'amalgamate-22.2-1', 'amalgamate-22.2-1-1', 'amalgamate-22.2-2', 'amalgamate-22.2-2-1', 'amalgamate-22.2-3', 'amalgamate-22.2-3-1', 'amalgamate-22.2-3-1-1', 'amalgamate-22.2-3-2', 'amuse-31.1', 'animal_sounds-38', 'appeal-31.4', 'appeal-31.4-1', 'appeal-31.4-2', 'appeal-31.4-3', 'appear-48.1.1', 'appoint-29.1', 'approve-77', 'assessment-34', 'assuming_position-50', 'avoid-52', 'banish-10.2', 'battle-36.4', 'battle-36.4-1', 'begin-55.1', 'begin-55.1-1', 'being_dressed-41.3.3', 'bend-45.2', 'berry-13.7', 'bill-54.5', 'body_internal_motion-49', 'body_internal_states-40.6', 'braid-41.2.2', 'break-45.1', 'breathe-40.1.2', 'breathe-40.1.2-1', 'bring-11.3', 'bring-11.3-1', 'build-26.1', 'build-26.1-1', 'bulge-47.5.3', 'bump-18.4', 'bump-18.4-1', 'butter-9.9', 'calibratable_cos-45.6', 'calibratable_cos-45.6-1', 'calve-28', 'captain-29.8', 'captain-29.8-1', 'captain-29.8-1-1', 'care-88', 'care-88-1', 'carry-11.4', 'carry-11.4-1', 'carry-11.4-1-1', 'carve-21.2', 'carve-21.2-1', 'carve-21.2-2', 'change_bodily_state-40.8.4', 'characterize-29.2', 'characterize-29.2-1', 'characterize-29.2-1-1', 'characterize-29.2-1-2', 'chase-51.6', 'cheat-10.6', 'cheat-10.6-1', 'cheat-10.6-1-1', 'chew-39.2', 'chew-39.2-1', 'chew-39.2-2', 'chit_chat-37.6', 'clear-10.3', 'clear-10.3-1', 'cling-22.5', 'coil-9.6', 'coil-9.6-1', 'coloring-24', 'complain-37.8', 'complete-55.2', 'concealment-16', 'concealment-16-1', 'confess-37.10', 'confine-92', 'confine-92-1', 'conjecture-29.5', 'conjecture-29.5-1', 'conjecture-29.5-2', 'consider-29.9', 'consider-29.9-1', 'consider-29.9-1-1', 'consider-29.9-1-1-1', 'consider-29.9-2', 'conspire-71', 'consume-66', 'consume-66-1', 'contiguous_location-47.8', 'contiguous_location-47.8-1', 'contiguous_location-47.8-2', 'continue-55.3', 'contribute-13.2', 'contribute-13.2-1', 'contribute-13.2-1-1', 'contribute-13.2-1-1-1', 'contribute-13.2-2', 'contribute-13.2-2-1', 'convert-26.6.2', 'convert-26.6.2-1', 'cooking-45.3', 'cooperate-73', 'cooperate-73-1', 'cooperate-73-2', 'cooperate-73-3', 'cope-83', 'cope-83-1', 'cope-83-1-1', 'correlate-86', 'correspond-36.1', 'correspond-36.1-1', 'correspond-36.1-1-1', 'cost-54.2', 'crane-40.3.2', 'create-26.4', 'create-26.4-1', 'curtsey-40.3.3', 'cut-21.1', 'cut-21.1-1', 'debone-10.8', 'declare-29.4', 'declare-29.4-1', 'declare-29.4-1-1', 'declare-29.4-1-1-1', 'declare-29.4-1-1-2', 'declare-29.4-1-1-3', 'declare-29.4-2', 'dedicate-79', 'defend-85', 'destroy-44', 'devour-39.4', 'devour-39.4-1', 'devour-39.4-2', 'differ-23.4', 'dine-39.5', 'disappearance-48.2', 'disassemble-23.3', 'discover-84', 'discover-84-1', 'discover-84-1-1', 'dress-41.1.1', 'dressing_well-41.3.2', 'drive-11.5', 'drive-11.5-1', 'dub-29.3', 'dub-29.3-1', 'eat-39.1', 'eat-39.1-1', 'eat-39.1-2', 'enforce-63', 'engender-27', 'entity_specific_cos-45.5', 'entity_specific_modes_being-47.2', 'equip-13.4.2', 'equip-13.4.2-1', 'equip-13.4.2-1-1', 'escape-51.1', 'escape-51.1-1', 'escape-51.1-2', 'escape-51.1-2-1', 'exceed-90', 'exchange-13.6', 'exchange-13.6-1', 'exchange-13.6-1-1', 'exhale-40.1.3', 'exhale-40.1.3-1', 'exhale-40.1.3-2', 'exist-47.1', 'exist-47.1-1', 'exist-47.1-1-1', 'feeding-39.7', 'ferret-35.6', 'fill-9.8', 'fill-9.8-1', 'fit-54.3', 'flinch-40.5', 'floss-41.2.1', 'focus-87', 'forbid-67', 'force-59', 'force-59-1', 'free-80', 'free-80-1', 'fulfilling-13.4.1', 'fulfilling-13.4.1-1', 'fulfilling-13.4.1-2', 'funnel-9.3', 'funnel-9.3-1', 'funnel-9.3-2', 'funnel-9.3-2-1', 'future_having-13.3', 'get-13.5.1', 'get-13.5.1-1', 'give-13.1', 'give-13.1-1', 'gobble-39.3', 'gobble-39.3-1', 'gobble-39.3-2', 'gorge-39.6', 'groom-41.1.2', 'grow-26.2', 'help-72', 'help-72-1', 'herd-47.5.2', 'hiccup-40.1.1', 'hit-18.1', 'hit-18.1-1', 'hold-15.1', 'hold-15.1-1', 'hunt-35.1', 'hurt-40.8.3', 'hurt-40.8.3-1', 'hurt-40.8.3-1-1', 'hurt-40.8.3-2', 'illustrate-25.3', 'image_impression-25.1', 'indicate-78', 'indicate-78-1', 'indicate-78-1-1', 'inquire-37.1.2', 'instr_communication-37.4', 'investigate-35.4', 'judgement-33', 'keep-15.2', 'knead-26.5', 'learn-14', 'learn-14-1', 'learn-14-2', 'learn-14-2-1', 'leave-51.2', 'leave-51.2-1', 'lecture-37.11', 'lecture-37.11-1', 'lecture-37.11-1-1', 'lecture-37.11-2', 'light_emission-43.1', 'limit-76', 'linger-53.1', 'linger-53.1-1', 'lodge-46', 'long-32.2', 'long-32.2-1', 'long-32.2-2', 'manner_speaking-37.3', 'marry-36.2', 'marvel-31.3', 'marvel-31.3-1', 'marvel-31.3-2', 'marvel-31.3-3', 'marvel-31.3-4', 'marvel-31.3-5', 'marvel-31.3-6', 'marvel-31.3-7', 'marvel-31.3-8', 'marvel-31.3-9', 'masquerade-29.6', 'masquerade-29.6-1', 'masquerade-29.6-2', 'matter-91', 'meander-47.7', 'meet-36.3', 'meet-36.3-1', 'meet-36.3-2', 'mine-10.9', 'mix-22.1', 'mix-22.1-1', 'mix-22.1-1-1', 'mix-22.1-2', 'mix-22.1-2-1', 'modes_of_being_with_motion-47.3', 'murder-42.1', 'murder-42.1-1', 'neglect-75', 'neglect-75-1', 'neglect-75-1-1', 'neglect-75-2', 'nonvehicle-51.4.2', 'nonverbal_expression-40.2', 'obtain-13.5.2', 'obtain-13.5.2-1', 'occurrence-48.3', 'order-60', 'order-60-1', 'orphan-29.7', 'other_cos-45.4', 'pain-40.8.1', 'pay-68', 'peer-30.3', 'pelt-17.2', 'performance-26.7', 'performance-26.7-1', 'performance-26.7-1-1', 'performance-26.7-2', 'performance-26.7-2-1', 'pit-10.7', 'pocket-9.10', 'pocket-9.10-1', 'poison-42.2', 'poke-19', 'pour-9.5', 'preparing-26.3', 'preparing-26.3-1', 'preparing-26.3-2', 'price-54.4', 'push-12', 'push-12-1', 'push-12-1-1', 'put-9.1', 'put-9.1-1', 'put-9.1-2', 'put_direction-9.4', 'put_spatial-9.2', 'put_spatial-9.2-1', 'reach-51.8', 'reflexive_appearance-48.1.2', 'refrain-69', 'register-54.1', 'rely-70', 'remove-10.1', 'risk-94', 'risk-94-1', 'roll-51.3.1', 'rummage-35.5', 'run-51.3.2', 'rush-53.2', 'say-37.7', 'say-37.7-1', 'say-37.7-1-1', 'say-37.7-2', 'scribble-25.2', 'search-35.2', 'see-30.1', 'see-30.1-1', 'see-30.1-1-1', 'send-11.1', 'send-11.1-1', 'separate-23.1', 'separate-23.1-1', 'separate-23.1-2', 'settle-89', 'shake-22.3', 'shake-22.3-1', 'shake-22.3-1-1', 'shake-22.3-2', 'shake-22.3-2-1', 'sight-30.2', 'simple_dressing-41.3.1', 'slide-11.2', 'slide-11.2-1-1', 'smell_emission-43.3', 'snooze-40.4', 'sound_emission-43.2', 'sound_existence-47.4', 'spank-18.3', 'spatial_configuration-47.6', 'split-23.2', 'spray-9.7', 'spray-9.7-1', 'spray-9.7-1-1', 'spray-9.7-2', 'stalk-35.3', 'steal-10.5', 'stimulus_subject-30.4', 'stop-55.4', 'stop-55.4-1', 'substance_emission-43.4', 'succeed-74', 'succeed-74-1', 'succeed-74-1-1', 'succeed-74-2', 'suffocate-40.7', 'suspect-81', 'swarm-47.5.1', 'swarm-47.5.1-1', 'swarm-47.5.1-2', 'swarm-47.5.1-2-1', 'swat-18.2', 'talk-37.5', 'tape-22.4', 'tape-22.4-1', 'tell-37.2', 'throw-17.1', 'throw-17.1-1', 'throw-17.1-1-1', 'tingle-40.8.2', 'touch-20', 'touch-20-1', 'transcribe-25.4', 'transfer_mesg-37.1.1', 'transfer_mesg-37.1.1-1', 'transfer_mesg-37.1.1-1-1', 'try-61', 'turn-26.6.1', 'turn-26.6.1-1', 'urge-58', 'vehicle-51.4.1', 'vehicle-51.4.1-1', 'waltz-51.5', 'want-32.1', 'want-32.1-1', 'want-32.1-1-1', 'weather-57', 'weekend-56', 'wink-40.3.1', 'wink-40.3.1-1', 'wipe_instr-10.4.2', 'wipe_instr-10.4.2-1', 'wipe_manner-10.4.1', 'wipe_manner-10.4.1-1', 'wish-62', 'withdraw-82', 'withdraw-82-1', 'withdraw-82-2', 'withdraw-82-3'] | |
| The primary object in the lexicon is a class record, which is stored | |
| as an ElementTree xml object. The class record for a given class | |
| identifier is returned by the `vnclass()` method: | |
| >>> verbnet.vnclass('remove-10.1') | |
| <Element 'VNCLASS' at ...> | |
| The `vnclass()` method also accepts "short" identifiers, such as '10.1': | |
| >>> verbnet.vnclass('10.1') | |
| <Element 'VNCLASS' at ...> | |
| See the Verbnet documentation, or the Verbnet files, for information | |
| about the structure of this xml. As an example, we can retrieve a | |
| list of thematic roles for a given Verbnet class: | |
| >>> vn_31_2 = verbnet.vnclass('admire-31.2') | |
| >>> for themrole in vn_31_2.findall('THEMROLES/THEMROLE'): | |
| ... print(themrole.attrib['type'], end=' ') | |
| ... for selrestr in themrole.findall('SELRESTRS/SELRESTR'): | |
| ... print('[%(Value)s%(type)s]' % selrestr.attrib, end=' ') | |
| ... print() | |
| Theme | |
| Experiencer [+animate] | |
| Predicate | |
| The Verbnet corpus also provides a variety of pretty printing | |
| functions that can be used to display the xml contents in a more | |
| concise form. The simplest such method is `pprint()`: | |
| >>> print(verbnet.pprint('57')) | |
| weather-57 | |
| Subclasses: (none) | |
| Members: blow clear drizzle fog freeze gust hail howl lightning mist | |
| mizzle pelt pour precipitate rain roar shower sleet snow spit spot | |
| sprinkle storm swelter teem thaw thunder | |
| Thematic roles: | |
| * Theme[+concrete +force] | |
| Frames: | |
| Intransitive (Expletive Subject) | |
| Example: It's raining. | |
| Syntax: LEX[it] LEX[[+be]] VERB | |
| Semantics: | |
| * weather(during(E), Weather_type, ?Theme) | |
| NP (Expletive Subject, Theme Object) | |
| Example: It's raining cats and dogs. | |
| Syntax: LEX[it] LEX[[+be]] VERB NP[Theme] | |
| Semantics: | |
| * weather(during(E), Weather_type, Theme) | |
| PP (Expletive Subject, Theme-PP) | |
| Example: It was pelting with rain. | |
| Syntax: LEX[it[+be]] VERB PREP[with] NP[Theme] | |
| Semantics: | |
| * weather(during(E), Weather_type, Theme) | |
| Verbnet gives us frames that link the syntax and semantics using an example. | |
| These frames are part of the corpus and we can use `frames()` to get a frame | |
| for a given verbnet class. | |
| >>> frame = verbnet.frames('57') | |
| >>> frame == [{'example': "It's raining.", 'description': {'primary': 'Intransitive', 'secondary': 'Expletive Subject'}, 'syntax': [{'pos_tag': 'LEX', 'modifiers': {'value': 'it', 'selrestrs': [], 'synrestrs': []}}, {'pos_tag': 'LEX', 'modifiers': {'value': '[+be]', 'selrestrs': [], 'synrestrs': []}}, {'pos_tag': 'VERB', 'modifiers': {'value': '', 'selrestrs': [], 'synrestrs': []}}], 'semantics': [{'predicate_value': 'weather', 'arguments': [{'type': 'Event', 'value': 'during(E)'}, {'type': 'VerbSpecific', 'value': 'Weather_type'}, {'type': 'ThemRole', 'value': '?Theme'}], 'negated': False}]}, {'example': "It's raining cats and dogs.", 'description': {'primary': 'NP', 'secondary': 'Expletive Subject, Theme Object'}, 'syntax': [{'pos_tag': 'LEX', 'modifiers': {'value': 'it', 'selrestrs': [], 'synrestrs': []}}, {'pos_tag': 'LEX', 'modifiers': {'value': '[+be]', 'selrestrs': [], 'synrestrs': []}}, {'pos_tag': 'VERB', 'modifiers': {'value': '', 'selrestrs': [], 'synrestrs': []}}, {'pos_tag': 'NP', 'modifiers': {'value': 'Theme', 'selrestrs': [], 'synrestrs': []}}], 'semantics': [{'predicate_value': 'weather', 'arguments': [{'type': 'Event', 'value': 'during(E)'}, {'type': 'VerbSpecific', 'value': 'Weather_type'}, {'type': 'ThemRole', 'value': 'Theme'}], 'negated': False}]}, {'example': 'It was pelting with rain.', 'description': {'primary': 'PP', 'secondary': 'Expletive Subject, Theme-PP'}, 'syntax': [{'pos_tag': 'LEX', 'modifiers': {'value': 'it[+be]', 'selrestrs': [], 'synrestrs': []}}, {'pos_tag': 'VERB', 'modifiers': {'value': '', 'selrestrs': [], 'synrestrs': []}}, {'pos_tag': 'PREP', 'modifiers': {'value': 'with', 'selrestrs': [], 'synrestrs': []}}, {'pos_tag': 'NP', 'modifiers': {'value': 'Theme', 'selrestrs': [], 'synrestrs': []}}], 'semantics': [{'predicate_value': 'weather', 'arguments': [{'type': 'Event', 'value': 'during(E)'}, {'type': 'VerbSpecific', 'value': 'Weather_type'}, {'type': 'ThemRole', 'value': 'Theme'}], 'negated': False}]}] | |
| True | |
| Verbnet corpus lets us access thematic roles individually using `themroles()`. | |
| >>> themroles = verbnet.themroles('57') | |
| >>> themroles == [{'modifiers': [{'type': 'concrete', 'value': '+'}, {'type': 'force', 'value': '+'}], 'type': 'Theme'}] | |
| True | |
| Verbnet classes may also have subclasses sharing similar syntactic and semantic properties | |
| while having differences with the superclass. The Verbnet corpus allows us to access these | |
| subclasses using `subclasses()`. | |
| >>> print(verbnet.subclasses('9.1')) #Testing for 9.1 since '57' does not have subclasses | |
| ['put-9.1-1', 'put-9.1-2'] | |
| nps_chat | |
| -------- | |
| The NPS Chat Corpus, Release 1.0 consists of over 10,000 posts in age-specific | |
| chat rooms, which have been anonymized, POS-tagged and dialogue-act tagged. | |
| >>> print(nltk.corpus.nps_chat.words()) | |
| ['now', 'im', 'left', 'with', 'this', 'gay', ...] | |
| >>> print(nltk.corpus.nps_chat.tagged_words()) | |
| [('now', 'RB'), ('im', 'PRP'), ('left', 'VBD'), ...] | |
| >>> print(nltk.corpus.nps_chat.tagged_posts()) | |
| [[('now', 'RB'), ('im', 'PRP'), ('left', 'VBD'), ('with', 'IN'), | |
| ('this', 'DT'), ('gay', 'JJ'), ('name', 'NN')], [(':P', 'UH')], ...] | |
| We can access the XML elements corresponding to individual posts. These elements | |
| have ``class`` and ``user`` attributes that we can access using ``p.attrib['class']`` | |
| and ``p.attrib['user']``. They also have text content, accessed using ``p.text``. | |
| >>> print(nltk.corpus.nps_chat.xml_posts()) | |
| [<Element 'Post' at 0...>, <Element 'Post' at 0...>, ...] | |
| >>> posts = nltk.corpus.nps_chat.xml_posts() | |
| >>> sorted(nltk.FreqDist(p.attrib['class'] for p in posts).keys()) | |
| ['Accept', 'Bye', 'Clarify', 'Continuer', 'Emotion', 'Emphasis', | |
| 'Greet', 'Other', 'Reject', 'Statement', 'System', 'nAnswer', | |
| 'whQuestion', 'yAnswer', 'ynQuestion'] | |
| >>> posts[0].text | |
| 'now im left with this gay name' | |
| In addition to the above methods for accessing tagged text, we can navigate | |
| the XML structure directly, as follows: | |
| >>> tokens = posts[0].findall('terminals/t') | |
| >>> [t.attrib['pos'] + "/" + t.attrib['word'] for t in tokens] | |
| ['RB/now', 'PRP/im', 'VBD/left', 'IN/with', 'DT/this', 'JJ/gay', 'NN/name'] | |
| multext_east | |
| ------------ | |
| The Multext-East Corpus consists of POS-tagged versions of George Orwell's book | |
| 1984 in 12 languages: English, Czech, Hungarian, Macedonian, Slovenian, Serbian, | |
| Slovak, Romanian, Estonian, Farsi, Bulgarian and Polish. | |
| The corpus can be accessed using the usual methods for tagged corpora. The tagset | |
| can be transformed from the Multext-East specific MSD tags to the Universal tagset | |
| using the "tagset" parameter of all functions returning tagged parts of the corpus. | |
| >>> print(nltk.corpus.multext_east.words("oana-en.xml")) | |
| ['It', 'was', 'a', 'bright', ...] | |
| >>> print(nltk.corpus.multext_east.tagged_words("oana-en.xml")) | |
| [('It', '#Pp3ns'), ('was', '#Vmis3s'), ('a', '#Di'), ...] | |
| >>> print(nltk.corpus.multext_east.tagged_sents("oana-en.xml", "universal")) | |
| [[('It', 'PRON'), ('was', 'VERB'), ('a', 'DET'), ...] | |
| --------------------- | |
| Corpus Reader Classes | |
| --------------------- | |
| NLTK's *corpus reader* classes are used to access the contents of a | |
| diverse set of corpora. Each corpus reader class is specialized to | |
| handle a specific corpus format. Examples include the | |
| `PlaintextCorpusReader`, which handles corpora that consist of a set | |
| of unannotated text files, and the `BracketParseCorpusReader`, which | |
| handles corpora that consist of files containing | |
| parenthesis-delineated parse trees. | |
| Automatically Created Corpus Reader Instances | |
| ============================================= | |
| When the `nltk.corpus` module is imported, it automatically creates a | |
| set of corpus reader instances that can be used to access the corpora | |
| in the NLTK data distribution. Here is a small sample of those | |
| corpus reader instances: | |
| >>> import nltk | |
| >>> nltk.corpus.brown | |
| <CategorizedTaggedCorpusReader ...> | |
| >>> nltk.corpus.treebank | |
| <BracketParseCorpusReader ...> | |
| >>> nltk.corpus.names | |
| <WordListCorpusReader ...> | |
| >>> nltk.corpus.genesis | |
| <PlaintextCorpusReader ...> | |
| >>> nltk.corpus.inaugural | |
| <PlaintextCorpusReader ...> | |
| This sample illustrates that different corpus reader classes are used | |
| to read different corpora; but that the same corpus reader class may | |
| be used for more than one corpus (e.g., ``genesis`` and ``inaugural``). | |
| Creating New Corpus Reader Instances | |
| ==================================== | |
| Although the `nltk.corpus` module automatically creates corpus reader | |
| instances for the corpora in the NLTK data distribution, you may | |
| sometimes need to create your own corpus reader. In particular, you | |
| would need to create your own corpus reader if you want... | |
| - To access a corpus that is not included in the NLTK data | |
| distribution. | |
| - To access a full copy of a corpus for which the NLTK data | |
| distribution only provides a sample. | |
| - To access a corpus using a customized corpus reader (e.g., with | |
| a customized tokenizer). | |
| To create a new corpus reader, you will first need to look up the | |
| signature for that corpus reader's constructor. Different corpus | |
| readers have different constructor signatures, but most of the | |
| constructor signatures have the basic form:: | |
| SomeCorpusReader(root, files, ...options...) | |
| Where ``root`` is an absolute path to the directory containing the | |
| corpus data files; ``files`` is either a list of file names (relative | |
| to ``root``) or a regexp specifying which files should be included; | |
| and ``options`` are additional reader-specific options. For example, | |
| we can create a customized corpus reader for the genesis corpus that | |
| uses a different sentence tokenizer as follows: | |
| >>> # Find the directory where the corpus lives. | |
| >>> genesis_dir = nltk.data.find('corpora/genesis') | |
| >>> # Create our custom sentence tokenizer. | |
| >>> my_sent_tokenizer = nltk.RegexpTokenizer('[^.!?]+') | |
| >>> # Create the new corpus reader object. | |
| >>> my_genesis = nltk.corpus.PlaintextCorpusReader( | |
| ... genesis_dir, r'.*\.txt', sent_tokenizer=my_sent_tokenizer) | |
| >>> # Use the new corpus reader object. | |
| >>> print(my_genesis.sents('english-kjv.txt')[0]) | |
| ['In', 'the', 'beginning', 'God', 'created', 'the', 'heaven', | |
| 'and', 'the', 'earth'] | |
| If you wish to read your own plaintext corpus, which is stored in the | |
| directory '/usr/share/some-corpus', then you can create a corpus | |
| reader for it with:: | |
| >>> my_corpus = nltk.corpus.PlaintextCorpusReader( | |
| ... '/usr/share/some-corpus', r'.*\.txt') # doctest: +SKIP | |
| For a complete list of corpus reader subclasses, see the API | |
| documentation for `nltk.corpus.reader`. | |
| Corpus Types | |
| ============ | |
| Corpora vary widely in the types of content they include. This is | |
| reflected in the fact that the base class `CorpusReader` only defines | |
| a few general-purpose methods for listing and accessing the files that | |
| make up a corpus. It is up to the subclasses to define *data access | |
| methods* that provide access to the information in the corpus. | |
| However, corpus reader subclasses should be consistent in their | |
| definitions of these data access methods wherever possible. | |
| At a high level, corpora can be divided into three basic types: | |
| - A *token corpus* contains information about specific occurrences of | |
| language use (or linguistic tokens), such as dialogues or written | |
| texts. Examples of token corpora are collections of written text | |
| and collections of speech. | |
| - A *type corpus*, or *lexicon*, contains information about a coherent | |
| set of lexical items (or linguistic types). Examples of lexicons | |
| are dictionaries and word lists. | |
| - A *language description corpus* contains information about a set of | |
| non-lexical linguistic constructs, such as grammar rules. | |
| However, many individual corpora blur the distinctions between these | |
| types. For example, corpora that are primarily lexicons may include | |
| token data in the form of example sentences; and corpora that are | |
| primarily token corpora may be accompanied by one or more word lists | |
| or other lexical data sets. | |
| Because corpora vary so widely in their information content, we have | |
| decided that it would not be wise to use separate corpus reader base | |
| classes for different corpus types. Instead, we simply try to make | |
| the corpus readers consistent wherever possible, but let them differ | |
| where the underlying data itself differs. | |
| Common Corpus Reader Methods | |
| ============================ | |
| As mentioned above, there are only a handful of methods that all | |
| corpus readers are guaranteed to implement. These methods provide | |
| access to the files that contain the corpus data. Every corpus is | |
| assumed to consist of one or more files, all located in a common root | |
| directory (or in subdirectories of that root directory). The absolute | |
| path to the root directory is stored in the ``root`` property: | |
| >>> import os | |
| >>> str(nltk.corpus.genesis.root).replace(os.path.sep,'/') | |
| '.../nltk_data/corpora/genesis' | |
| Each file within the corpus is identified by a platform-independent | |
| identifier, which is basically a path string that uses ``/`` as the | |
| path separator. I.e., this identifier can be converted to a relative | |
| path as follows: | |
| >>> some_corpus_file_id = nltk.corpus.reuters.fileids()[0] | |
| >>> import os.path | |
| >>> os.path.normpath(some_corpus_file_id).replace(os.path.sep,'/') | |
| 'test/14826' | |
| To get a list of all data files that make up a corpus, use the | |
| ``fileids()`` method. In some corpora, these files will not all contain | |
| the same type of data; for example, for the ``nltk.corpus.timit`` | |
| corpus, ``fileids()`` will return a list including text files, word | |
| segmentation files, phonetic transcription files, sound files, and | |
| metadata files. For corpora with diverse file types, the ``fileids()`` | |
| method will often take one or more optional arguments, which can be | |
| used to get a list of the files with a specific file type: | |
| >>> nltk.corpus.timit.fileids() | |
| ['dr1-fvmh0/sa1.phn', 'dr1-fvmh0/sa1.txt', 'dr1-fvmh0/sa1.wav', ...] | |
| >>> nltk.corpus.timit.fileids('phn') | |
| ['dr1-fvmh0/sa1.phn', 'dr1-fvmh0/sa2.phn', 'dr1-fvmh0/si1466.phn', ...] | |
| In some corpora, the files are divided into distinct categories. For | |
| these corpora, the ``fileids()`` method takes an optional argument, | |
| which can be used to get a list of the files within a specific category: | |
| >>> nltk.corpus.brown.fileids('hobbies') | |
| ['ce01', 'ce02', 'ce03', 'ce04', 'ce05', 'ce06', 'ce07', ...] | |
| The ``abspath()`` method can be used to find the absolute path to a | |
| corpus file, given its file identifier: | |
| >>> str(nltk.corpus.brown.abspath('ce06')).replace(os.path.sep,'/') | |
| '.../corpora/brown/ce06' | |
| The ``abspaths()`` method can be used to find the absolute paths for | |
| one corpus file, a list of corpus files, or (if no fileids are specified), | |
| all corpus files. | |
| This method is mainly useful as a helper method when defining corpus | |
| data access methods, since data access methods can usually be called | |
| with a string argument (to get a view for a specific file), with a | |
| list argument (to get a view for a specific list of files), or with no | |
| argument (to get a view for the whole corpus). | |
| Data Access Methods | |
| =================== | |
| Individual corpus reader subclasses typically extend this basic set of | |
| file-access methods with one or more *data access methods*, which provide | |
| easy access to the data contained in the corpus. The signatures for | |
| data access methods often have the basic form:: | |
| corpus_reader.some_data access(fileids=None, ...options...) | |
| Where ``fileids`` can be a single file identifier string (to get a view | |
| for a specific file); a list of file identifier strings (to get a view | |
| for a specific list of files); or None (to get a view for the entire | |
| corpus). Some of the common data access methods, and their return | |
| types, are: | |
| - I{corpus}.words(): list of str | |
| - I{corpus}.sents(): list of (list of str) | |
| - I{corpus}.paras(): list of (list of (list of str)) | |
| - I{corpus}.tagged_words(): list of (str,str) tuple | |
| - I{corpus}.tagged_sents(): list of (list of (str,str)) | |
| - I{corpus}.tagged_paras(): list of (list of (list of (str,str))) | |
| - I{corpus}.chunked_sents(): list of (Tree w/ (str,str) leaves) | |
| - I{corpus}.parsed_sents(): list of (Tree with str leaves) | |
| - I{corpus}.parsed_paras(): list of (list of (Tree with str leaves)) | |
| - I{corpus}.xml(): A single xml ElementTree | |
| - I{corpus}.raw(): str (unprocessed corpus contents) | |
| For example, the `words()` method is supported by many different | |
| corpora, and returns a flat list of word strings: | |
| >>> nltk.corpus.brown.words() | |
| ['The', 'Fulton', 'County', 'Grand', 'Jury', ...] | |
| >>> nltk.corpus.treebank.words() | |
| ['Pierre', 'Vinken', ',', '61', 'years', 'old', ...] | |
| >>> nltk.corpus.conll2002.words() | |
| ['Sao', 'Paulo', '(', 'Brasil', ')', ',', '23', ...] | |
| >>> nltk.corpus.genesis.words() | |
| ['In', 'the', 'beginning', 'God', 'created', ...] | |
| On the other hand, the `tagged_words()` method is only supported by | |
| corpora that include part-of-speech annotations: | |
| >>> nltk.corpus.brown.tagged_words() | |
| [('The', 'AT'), ('Fulton', 'NP-TL'), ...] | |
| >>> nltk.corpus.treebank.tagged_words() | |
| [('Pierre', 'NNP'), ('Vinken', 'NNP'), ...] | |
| >>> nltk.corpus.conll2002.tagged_words() | |
| [('Sao', 'NC'), ('Paulo', 'VMI'), ('(', 'Fpa'), ...] | |
| >>> nltk.corpus.genesis.tagged_words() | |
| Traceback (most recent call last): | |
| ... | |
| AttributeError: 'PlaintextCorpusReader' object has no attribute 'tagged_words' | |
| Although most corpus readers use file identifiers to index their | |
| content, some corpora use different identifiers instead. For example, | |
| the data access methods for the ``timit`` corpus uses *utterance | |
| identifiers* to select which corpus items should be returned: | |
| >>> nltk.corpus.timit.utteranceids() | |
| ['dr1-fvmh0/sa1', 'dr1-fvmh0/sa2', 'dr1-fvmh0/si1466', ...] | |
| >>> nltk.corpus.timit.words('dr1-fvmh0/sa2') | |
| ["don't", 'ask', 'me', 'to', 'carry', 'an', 'oily', 'rag', 'like', 'that'] | |
| Attempting to call ``timit``\ 's data access methods with a file | |
| identifier will result in an exception: | |
| >>> nltk.corpus.timit.fileids() | |
| ['dr1-fvmh0/sa1.phn', 'dr1-fvmh0/sa1.txt', 'dr1-fvmh0/sa1.wav', ...] | |
| >>> nltk.corpus.timit.words('dr1-fvmh0/sa1.txt') # doctest: +SKIP | |
| Traceback (most recent call last): | |
| ... | |
| IOError: No such file or directory: '.../dr1-fvmh0/sa1.txt.wrd' | |
| As another example, the ``propbank`` corpus defines the ``roleset()`` | |
| method, which expects a roleset identifier, not a file identifier: | |
| >>> roleset = nltk.corpus.propbank.roleset('eat.01') | |
| >>> from xml.etree import ElementTree as ET | |
| >>> print(ET.tostring(roleset).decode('utf8')) | |
| <roleset id="eat.01" name="consume" vncls="39.1"> | |
| <roles> | |
| <role descr="consumer, eater" n="0">...</role>... | |
| </roles>... | |
| </roleset>... | |
| Stream Backed Corpus Views | |
| ========================== | |
| An important feature of NLTK's corpus readers is that many of them | |
| access the underlying data files using "corpus views." A *corpus | |
| view* is an object that acts like a simple data structure (such as a | |
| list), but does not store the data elements in memory; instead, data | |
| elements are read from the underlying data files on an as-needed | |
| basis. | |
| By only loading items from the file on an as-needed basis, corpus | |
| views maintain both memory efficiency and responsiveness. The memory | |
| efficiency of corpus readers is important because some corpora contain | |
| very large amounts of data, and storing the entire data set in memory | |
| could overwhelm many machines. The responsiveness is important when | |
| experimenting with corpora in interactive sessions and in in-class | |
| demonstrations. | |
| The most common corpus view is the `StreamBackedCorpusView`, which | |
| acts as a read-only list of tokens. Two additional corpus view | |
| classes, `ConcatenatedCorpusView` and `LazySubsequence`, make it | |
| possible to create concatenations and take slices of | |
| `StreamBackedCorpusView` objects without actually storing the | |
| resulting list-like object's elements in memory. | |
| In the future, we may add additional corpus views that act like other | |
| basic data structures, such as dictionaries. | |
| Writing New Corpus Readers | |
| ========================== | |
| In order to add support for new corpus formats, it is necessary to | |
| define new corpus reader classes. For many corpus formats, writing | |
| new corpus readers is relatively straight-forward. In this section, | |
| we'll describe what's involved in creating a new corpus reader. If | |
| you do create a new corpus reader, we encourage you to contribute it | |
| back to the NLTK project. | |
| Don't Reinvent the Wheel | |
| ------------------------ | |
| Before you start writing a new corpus reader, you should check to be | |
| sure that the desired format can't be read using an existing corpus | |
| reader with appropriate constructor arguments. For example, although | |
| the `TaggedCorpusReader` assumes that words and tags are separated by | |
| ``/`` characters by default, an alternative tag-separation character | |
| can be specified via the ``sep`` constructor argument. You should | |
| also check whether the new corpus format can be handled by subclassing | |
| an existing corpus reader, and tweaking a few methods or variables. | |
| Design | |
| ------ | |
| If you decide to write a new corpus reader from scratch, then you | |
| should first decide which data access methods you want the reader to | |
| provide, and what their signatures should be. You should look at | |
| existing corpus readers that process corpora with similar data | |
| contents, and try to be consistent with those corpus readers whenever | |
| possible. | |
| You should also consider what sets of identifiers are appropriate for | |
| the corpus format. Where it's practical, file identifiers should be | |
| used. However, for some corpora, it may make sense to use additional | |
| sets of identifiers. Each set of identifiers should have a distinct | |
| name (e.g., fileids, utteranceids, rolesets); and you should be consistent | |
| in using that name to refer to that identifier. Do not use parameter | |
| names like ``id``, which leave it unclear what type of identifier is | |
| required. | |
| Once you've decided what data access methods and identifiers are | |
| appropriate for your corpus, you should decide if there are any | |
| customizable parameters that you'd like the corpus reader to handle. | |
| These parameters make it possible to use a single corpus reader to | |
| handle a wider variety of corpora. The ``sep`` argument for | |
| `TaggedCorpusReader`, mentioned above, is an example of a customizable | |
| corpus reader parameter. | |
| Implementation | |
| -------------- | |
| Constructor | |
| ~~~~~~~~~~~ | |
| If your corpus reader implements any customizable parameters, then | |
| you'll need to override the constructor. Typically, the new | |
| constructor will first call its base class's constructor, and then | |
| store the customizable parameters. For example, the | |
| `ConllChunkCorpusReader`\ 's constructor is defined as follows: | |
| >>> def __init__(self, root, fileids, chunk_types, encoding='utf8', | |
| ... tagset=None, separator=None): | |
| ... ConllCorpusReader.__init__( | |
| ... self, root, fileids, ('words', 'pos', 'chunk'), | |
| ... chunk_types=chunk_types, encoding=encoding, | |
| ... tagset=tagset, separator=separator) | |
| If your corpus reader does not implement any customization parameters, | |
| then you can often just inherit the base class's constructor. | |
| Data Access Methods | |
| ~~~~~~~~~~~~~~~~~~~ | |
| The most common type of data access method takes an argument | |
| identifying which files to access, and returns a view covering those | |
| files. This argument may be a single file identifier string (to get a | |
| view for a specific file); a list of file identifier strings (to get a | |
| view for a specific list of files); or None (to get a view for the | |
| entire corpus). The method's implementation converts this argument to | |
| a list of path names using the `abspaths()` method, which handles all | |
| three value types (string, list, and None): | |
| >>> print(str(nltk.corpus.brown.abspaths()).replace('\\\\','/')) | |
| [FileSystemPathPointer('.../corpora/brown/ca01'), | |
| FileSystemPathPointer('.../corpora/brown/ca02'), ...] | |
| >>> print(str(nltk.corpus.brown.abspaths('ce06')).replace('\\\\','/')) | |
| [FileSystemPathPointer('.../corpora/brown/ce06')] | |
| >>> print(str(nltk.corpus.brown.abspaths(['ce06', 'ce07'])).replace('\\\\','/')) | |
| [FileSystemPathPointer('.../corpora/brown/ce06'), | |
| FileSystemPathPointer('.../corpora/brown/ce07')] | |
| An example of this type of method is the `words()` method, defined by | |
| the `PlaintextCorpusReader` as follows: | |
| >>> def words(self, fileids=None): | |
| ... return concat([self.CorpusView(fileid, self._read_word_block) | |
| ... for fileid in self.abspaths(fileids)]) | |
| This method first uses `abspaths()` to convert ``fileids`` to a list of | |
| absolute paths. It then creates a corpus view for each file, using | |
| the `PlaintextCorpusReader._read_word_block()` method to read elements | |
| from the data file (see the discussion of corpus views below). | |
| Finally, it combines these corpus views using the | |
| `nltk.corpus.reader.util.concat()` function. | |
| When writing a corpus reader for a corpus that is never expected to be | |
| very large, it can sometimes be appropriate to read the files | |
| directly, rather than using a corpus view. For example, the | |
| `WordListCorpusView` class defines its `words()` method as follows: | |
| >>> def words(self, fileids=None): | |
| ... return concat([[w for w in open(fileid).read().split('\n') if w] | |
| ... for fileid in self.abspaths(fileids)]) | |
| (This is usually more appropriate for lexicons than for token corpora.) | |
| If the type of data returned by a data access method is one for which | |
| NLTK has a conventional representation (e.g., words, tagged words, and | |
| parse trees), then you should use that representation. Otherwise, you | |
| may find it necessary to define your own representation. For data | |
| structures that are relatively corpus-specific, it's usually best to | |
| define new classes for these elements. For example, the ``propbank`` | |
| corpus defines the `PropbankInstance` class to store the semantic role | |
| labeling instances described by the corpus; and the ``ppattach`` | |
| corpus defines the `PPAttachment` class to store the prepositional | |
| attachment instances described by the corpus. | |
| Corpus Views | |
| ~~~~~~~~~~~~ | |
| .. (Much of the content for this section is taken from the | |
| StreamBackedCorpusView docstring.) | |
| The heart of a `StreamBackedCorpusView` is its *block reader* | |
| function, which reads zero or more tokens from a stream, and returns | |
| them as a list. A very simple example of a block reader is: | |
| >>> def simple_block_reader(stream): | |
| ... return stream.readline().split() | |
| This simple block reader reads a single line at a time, and returns a | |
| single token (consisting of a string) for each whitespace-separated | |
| substring on the line. A `StreamBackedCorpusView` built from this | |
| block reader will act like a read-only list of all the | |
| whitespace-separated tokens in an underlying file. | |
| When deciding how to define the block reader for a given corpus, | |
| careful consideration should be given to the size of blocks handled by | |
| the block reader. Smaller block sizes will increase the memory | |
| requirements of the corpus view's internal data structures (by 2 | |
| integers per block). On the other hand, larger block sizes may | |
| decrease performance for random access to the corpus. (But note that | |
| larger block sizes will *not* decrease performance for iteration.) | |
| Internally, the `StreamBackedCorpusView` class maintains a partial | |
| mapping from token index to file position, with one entry per block. | |
| When a token with a given index *i* is requested, the corpus view | |
| constructs it as follows: | |
| 1. First, it searches the toknum/filepos mapping for the token index | |
| closest to (but less than or equal to) *i*. | |
| 2. Then, starting at the file position corresponding to that index, it | |
| reads one block at a time using the block reader until it reaches | |
| the requested token. | |
| The toknum/filepos mapping is created lazily: it is initially empty, | |
| but every time a new block is read, the block's initial token is added | |
| to the mapping. (Thus, the toknum/filepos map has one entry per | |
| block.) | |
| You can create your own corpus view in one of two ways: | |
| 1. Call the `StreamBackedCorpusView` constructor, and provide your | |
| block reader function via the ``block_reader`` argument. | |
| 2. Subclass `StreamBackedCorpusView`, and override the | |
| `read_block()` method. | |
| The first option is usually easier, but the second option can allow | |
| you to write a single `read_block` method whose behavior can be | |
| customized by different parameters to the subclass's constructor. For | |
| an example of this design pattern, see the `TaggedCorpusView` class, | |
| which is used by `TaggedCorpusView`. | |
| ---------------- | |
| Regression Tests | |
| ---------------- | |
| The following helper functions are used to create and then delete | |
| testing corpora that are stored in temporary directories. These | |
| testing corpora are used to make sure the readers work correctly. | |
| >>> import tempfile, os.path, textwrap | |
| >>> def make_testcorpus(ext='', **fileids): | |
| ... root = tempfile.mkdtemp() | |
| ... for fileid, contents in fileids.items(): | |
| ... fileid += ext | |
| ... f = open(os.path.join(root, fileid), 'w') | |
| ... f.write(textwrap.dedent(contents)) | |
| ... f.close() | |
| ... return root | |
| >>> def del_testcorpus(root): | |
| ... for fileid in os.listdir(root): | |
| ... os.remove(os.path.join(root, fileid)) | |
| ... os.rmdir(root) | |
| Plaintext Corpus Reader | |
| ======================= | |
| The plaintext corpus reader is used to access corpora that consist of | |
| unprocessed plaintext data. It assumes that paragraph breaks are | |
| indicated by blank lines. Sentences and words can be tokenized using | |
| the default tokenizers, or by custom tokenizers specified as | |
| parameters to the constructor. | |
| >>> root = make_testcorpus(ext='.txt', | |
| ... a="""\ | |
| ... This is the first sentence. Here is another | |
| ... sentence! And here's a third sentence. | |
| ... | |
| ... This is the second paragraph. Tokenization is currently | |
| ... fairly simple, so the period in Mr. gets tokenized. | |
| ... """, | |
| ... b="""This is the second file.""") | |
| >>> from nltk.corpus.reader.plaintext import PlaintextCorpusReader | |
| The list of documents can be specified explicitly, or implicitly (using a | |
| regexp). The ``ext`` argument specifies a file extension. | |
| >>> corpus = PlaintextCorpusReader(root, ['a.txt', 'b.txt']) | |
| >>> corpus.fileids() | |
| ['a.txt', 'b.txt'] | |
| >>> corpus = PlaintextCorpusReader(root, r'.*\.txt') | |
| >>> corpus.fileids() | |
| ['a.txt', 'b.txt'] | |
| The directory containing the corpus is corpus.root: | |
| >>> str(corpus.root) == str(root) | |
| True | |
| We can get a list of words, or the raw string: | |
| >>> corpus.words() | |
| ['This', 'is', 'the', 'first', 'sentence', '.', ...] | |
| >>> corpus.raw()[:40] | |
| 'This is the first sentence. Here is ano' | |
| Check that reading individual documents works, and reading all documents at | |
| once works: | |
| >>> len(corpus.words()), [len(corpus.words(d)) for d in corpus.fileids()] | |
| (46, [40, 6]) | |
| >>> corpus.words('a.txt') | |
| ['This', 'is', 'the', 'first', 'sentence', '.', ...] | |
| >>> corpus.words('b.txt') | |
| ['This', 'is', 'the', 'second', 'file', '.'] | |
| >>> corpus.words()[:4], corpus.words()[-4:] | |
| (['This', 'is', 'the', 'first'], ['the', 'second', 'file', '.']) | |
| We're done with the test corpus: | |
| >>> del_testcorpus(root) | |
| Test the plaintext corpora that come with nltk: | |
| >>> from nltk.corpus import abc, genesis, inaugural | |
| >>> from nltk.corpus import state_union, webtext | |
| >>> for corpus in (abc, genesis, inaugural, state_union, | |
| ... webtext): | |
| ... print(str(corpus).replace('\\\\','/')) | |
| ... print(' ', repr(corpus.fileids())[:60]) | |
| ... print(' ', repr(corpus.words()[:10])[:60]) | |
| <PlaintextCorpusReader in '.../nltk_data/corpora/ab...'> | |
| ['rural.txt', 'science.txt'] | |
| ['PM', 'denies', 'knowledge', 'of', 'AWB', ... | |
| <PlaintextCorpusReader in '.../nltk_data/corpora/genesi...'> | |
| ['english-kjv.txt', 'english-web.txt', 'finnish.txt', ... | |
| ['In', 'the', 'beginning', 'God', 'created', 'the', ... | |
| <PlaintextCorpusReader in '.../nltk_data/corpora/inaugura...'> | |
| ['1789-Washington.txt', '1793-Washington.txt', ... | |
| ['Fellow', '-', 'Citizens', 'of', 'the', 'Senate', ... | |
| <PlaintextCorpusReader in '.../nltk_data/corpora/state_unio...'> | |
| ['1945-Truman.txt', '1946-Truman.txt', ... | |
| ['PRESIDENT', 'HARRY', 'S', '.', 'TRUMAN', "'", ... | |
| <PlaintextCorpusReader in '.../nltk_data/corpora/webtex...'> | |
| ['firefox.txt', 'grail.txt', 'overheard.txt', ... | |
| ['Cookie', 'Manager', ':', '"', 'Don', "'", 't', ... | |
| Tagged Corpus Reader | |
| ==================== | |
| The Tagged Corpus reader can give us words, sentences, and paragraphs, | |
| each tagged or untagged. All of the read methods can take one item | |
| (in which case they return the contents of that file) or a list of | |
| documents (in which case they concatenate the contents of those files). | |
| By default, they apply to all documents in the corpus. | |
| >>> root = make_testcorpus( | |
| ... a="""\ | |
| ... This/det is/verb the/det first/adj sentence/noun ./punc | |
| ... Here/det is/verb another/adj sentence/noun ./punc | |
| ... Note/verb that/comp you/pron can/verb use/verb \ | |
| ... any/noun tag/noun set/noun | |
| ... | |
| ... This/det is/verb the/det second/adj paragraph/noun ./punc | |
| ... word/n without/adj a/det tag/noun :/: hello ./punc | |
| ... """, | |
| ... b="""\ | |
| ... This/det is/verb the/det second/adj file/noun ./punc | |
| ... """) | |
| >>> from nltk.corpus.reader.tagged import TaggedCorpusReader | |
| >>> corpus = TaggedCorpusReader(root, list('ab')) | |
| >>> corpus.fileids() | |
| ['a', 'b'] | |
| >>> str(corpus.root) == str(root) | |
| True | |
| >>> corpus.words() | |
| ['This', 'is', 'the', 'first', 'sentence', '.', ...] | |
| >>> corpus.sents() | |
| [['This', 'is', 'the', 'first', ...], ['Here', 'is', 'another'...], ...] | |
| >>> corpus.paras() | |
| [[['This', ...], ['Here', ...], ...], [['This', ...], ...], ...] | |
| >>> corpus.tagged_words() | |
| [('This', 'DET'), ('is', 'VERB'), ('the', 'DET'), ...] | |
| >>> corpus.tagged_sents() | |
| [[('This', 'DET'), ('is', 'VERB'), ...], [('Here', 'DET'), ...], ...] | |
| >>> corpus.tagged_paras() | |
| [[[('This', 'DET'), ...], ...], [[('This', 'DET'), ...], ...], ...] | |
| >>> corpus.raw()[:40] | |
| 'This/det is/verb the/det first/adj sente' | |
| >>> len(corpus.words()), [len(corpus.words(d)) for d in corpus.fileids()] | |
| (38, [32, 6]) | |
| >>> len(corpus.sents()), [len(corpus.sents(d)) for d in corpus.fileids()] | |
| (6, [5, 1]) | |
| >>> len(corpus.paras()), [len(corpus.paras(d)) for d in corpus.fileids()] | |
| (3, [2, 1]) | |
| >>> print(corpus.words('a')) | |
| ['This', 'is', 'the', 'first', 'sentence', '.', ...] | |
| >>> print(corpus.words('b')) | |
| ['This', 'is', 'the', 'second', 'file', '.'] | |
| >>> del_testcorpus(root) | |
| The Brown Corpus uses the tagged corpus reader: | |
| >>> from nltk.corpus import brown | |
| >>> brown.fileids() | |
| ['ca01', 'ca02', 'ca03', 'ca04', 'ca05', 'ca06', 'ca07', ...] | |
| >>> brown.categories() | |
| ['adventure', 'belles_lettres', 'editorial', 'fiction', 'government', 'hobbies', 'humor', | |
| 'learned', 'lore', 'mystery', 'news', 'religion', 'reviews', 'romance', 'science_fiction'] | |
| >>> print(repr(brown.root).replace('\\\\','/')) | |
| FileSystemPathPointer('.../corpora/brown') | |
| >>> brown.words() | |
| ['The', 'Fulton', 'County', 'Grand', 'Jury', ...] | |
| >>> brown.sents() | |
| [['The', 'Fulton', 'County', 'Grand', ...], ...] | |
| >>> brown.paras() | |
| [[['The', 'Fulton', 'County', ...]], [['The', 'jury', ...]], ...] | |
| >>> brown.tagged_words() | |
| [('The', 'AT'), ('Fulton', 'NP-TL'), ...] | |
| >>> brown.tagged_sents() | |
| [[('The', 'AT'), ('Fulton', 'NP-TL'), ('County', 'NN-TL'), ...], ...] | |
| >>> brown.tagged_paras() | |
| [[[('The', 'AT'), ...]], [[('The', 'AT'), ...]], ...] | |
| Verbnet Corpus Reader | |
| ===================== | |
| Make sure we're picking up the right number of elements: | |
| >>> from nltk.corpus import verbnet | |
| >>> len(verbnet.lemmas()) | |
| 3621 | |
| >>> len(verbnet.wordnetids()) | |
| 4953 | |
| >>> len(verbnet.classids()) | |
| 429 | |
| Selecting classids based on various selectors: | |
| >>> verbnet.classids(lemma='take') | |
| ['bring-11.3', 'characterize-29.2', 'convert-26.6.2', 'cost-54.2', | |
| 'fit-54.3', 'performance-26.7-2', 'steal-10.5'] | |
| >>> verbnet.classids(wordnetid='lead%2:38:01') | |
| ['accompany-51.7'] | |
| >>> verbnet.classids(fileid='approve-77.xml') | |
| ['approve-77'] | |
| >>> verbnet.classids(classid='admire-31.2') # subclasses | |
| ['admire-31.2-1'] | |
| vnclass() accepts filenames, long ids, and short ids: | |
| >>> a = ElementTree.tostring(verbnet.vnclass('admire-31.2.xml')) | |
| >>> b = ElementTree.tostring(verbnet.vnclass('admire-31.2')) | |
| >>> c = ElementTree.tostring(verbnet.vnclass('31.2')) | |
| >>> a == b == c | |
| True | |
| fileids() can be used to get files based on verbnet class ids: | |
| >>> verbnet.fileids('admire-31.2') | |
| ['admire-31.2.xml'] | |
| >>> verbnet.fileids(['admire-31.2', 'obtain-13.5.2']) | |
| ['admire-31.2.xml', 'obtain-13.5.2.xml'] | |
| >>> verbnet.fileids('badidentifier') | |
| Traceback (most recent call last): | |
| . . . | |
| ValueError: vnclass identifier 'badidentifier' not found | |
| longid() and shortid() can be used to convert identifiers: | |
| >>> verbnet.longid('31.2') | |
| 'admire-31.2' | |
| >>> verbnet.longid('admire-31.2') | |
| 'admire-31.2' | |
| >>> verbnet.shortid('31.2') | |
| '31.2' | |
| >>> verbnet.shortid('admire-31.2') | |
| '31.2' | |
| >>> verbnet.longid('badidentifier') | |
| Traceback (most recent call last): | |
| . . . | |
| ValueError: vnclass identifier 'badidentifier' not found | |
| >>> verbnet.shortid('badidentifier') | |
| Traceback (most recent call last): | |
| . . . | |
| ValueError: vnclass identifier 'badidentifier' not found | |
| Corpus View Regression Tests | |
| ============================ | |
| Select some corpus files to play with: | |
| >>> import nltk.data | |
| >>> # A very short file (160 chars): | |
| >>> f1 = nltk.data.find('corpora/inaugural/README') | |
| >>> # A relatively short file (791 chars): | |
| >>> f2 = nltk.data.find('corpora/inaugural/1793-Washington.txt') | |
| >>> # A longer file (32k chars): | |
| >>> f3 = nltk.data.find('corpora/inaugural/1909-Taft.txt') | |
| >>> fileids = [f1, f2, f3] | |
| Concatenation | |
| ------------- | |
| Check that concatenation works as intended. | |
| >>> from nltk.corpus.reader.util import * | |
| >>> c1 = StreamBackedCorpusView(f1, read_whitespace_block, encoding='utf-8') | |
| >>> c2 = StreamBackedCorpusView(f2, read_whitespace_block, encoding='utf-8') | |
| >>> c3 = StreamBackedCorpusView(f3, read_whitespace_block, encoding='utf-8') | |
| >>> c123 = c1+c2+c3 | |
| >>> print(c123) | |
| ['C-Span', 'Inaugural', 'Address', 'Corpus', 'US', ...] | |
| >>> l1 = f1.open(encoding='utf-8').read().split() | |
| >>> l2 = f2.open(encoding='utf-8').read().split() | |
| >>> l3 = f3.open(encoding='utf-8').read().split() | |
| >>> l123 = l1+l2+l3 | |
| >>> list(c123) == l123 | |
| True | |
| >>> (c1+c2+c3)[100] == l123[100] | |
| True | |
| Slicing | |
| ------- | |
| First, do some tests with fairly small slices. These will all | |
| generate tuple values. | |
| >>> from nltk.util import LazySubsequence | |
| >>> c1 = StreamBackedCorpusView(f1, read_whitespace_block, encoding='utf-8') | |
| >>> l1 = f1.open(encoding='utf-8').read().split() | |
| >>> print(len(c1)) | |
| 21 | |
| >>> len(c1) < LazySubsequence.MIN_SIZE | |
| True | |
| Choose a list of indices, based on the length, that covers the | |
| important corner cases: | |
| >>> indices = [-60, -30, -22, -21, -20, -1, | |
| ... 0, 1, 10, 20, 21, 22, 30, 60] | |
| Test slicing with explicit start & stop value: | |
| >>> for s in indices: | |
| ... for e in indices: | |
| ... assert list(c1[s:e]) == l1[s:e] | |
| Test slicing with stop=None: | |
| >>> for s in indices: | |
| ... assert list(c1[s:]) == l1[s:] | |
| Test slicing with start=None: | |
| >>> for e in indices: | |
| ... assert list(c1[:e]) == l1[:e] | |
| Test slicing with start=stop=None: | |
| >>> list(c1[:]) == list(l1[:]) | |
| True | |
| Next, we'll do some tests with much longer slices. These will | |
| generate LazySubsequence objects. | |
| >>> c3 = StreamBackedCorpusView(f3, read_whitespace_block, encoding='utf-8') | |
| >>> l3 = f3.open(encoding='utf-8').read().split() | |
| >>> print(len(c3)) | |
| 5430 | |
| >>> len(c3) > LazySubsequence.MIN_SIZE*2 | |
| True | |
| Choose a list of indices, based on the length, that covers the | |
| important corner cases: | |
| >>> indices = [-12000, -6000, -5431, -5430, -5429, -3000, -200, -1, | |
| ... 0, 1, 200, 3000, 5000, 5429, 5430, 5431, 6000, 12000] | |
| Test slicing with explicit start & stop value: | |
| >>> for s in indices: | |
| ... for e in indices: | |
| ... assert list(c3[s:e]) == l3[s:e] | |
| Test slicing with stop=None: | |
| >>> for s in indices: | |
| ... assert list(c3[s:]) == l3[s:] | |
| Test slicing with start=None: | |
| >>> for e in indices: | |
| ... assert list(c3[:e]) == l3[:e] | |
| Test slicing with start=stop=None: | |
| >>> list(c3[:]) == list(l3[:]) | |
| True | |
| Multiple Iterators | |
| ------------------ | |
| If multiple iterators are created for the same corpus view, their | |
| iteration can be interleaved: | |
| >>> c3 = StreamBackedCorpusView(f3, read_whitespace_block) | |
| >>> iterators = [c3.iterate_from(n) for n in [0,15,30,45]] | |
| >>> for i in range(15): | |
| ... for iterator in iterators: | |
| ... print('%-15s' % next(iterator), end=' ') | |
| ... print() | |
| My a duties in | |
| fellow heavy of a | |
| citizens: weight the proper | |
| Anyone of office sense | |
| who responsibility. upon of | |
| has If which the | |
| taken not, he obligation | |
| the he is which | |
| oath has about the | |
| I no to oath | |
| have conception enter, imposes. | |
| just of or The | |
| taken the he office | |
| must powers is of | |
| feel and lacking an | |
| SeekableUnicodeStreamReader | |
| =========================== | |
| The file-like objects provided by the ``codecs`` module unfortunately | |
| suffer from a bug that prevents them from working correctly with | |
| corpus view objects. In particular, although the expose ``seek()`` | |
| and ``tell()`` methods, those methods do not exhibit the expected | |
| behavior, because they are not synchronized with the internal buffers | |
| that are kept by the file-like objects. For example, the ``tell()`` | |
| method will return the file position at the end of the buffers (whose | |
| contents have not yet been returned by the stream); and therefore this | |
| file position can not be used to return to the 'current' location in | |
| the stream (since ``seek()`` has no way to reconstruct the buffers). | |
| To get around these problems, we define a new class, | |
| `SeekableUnicodeStreamReader`, to act as a file-like interface to | |
| files containing encoded unicode data. This class is loosely based on | |
| the ``codecs.StreamReader`` class. To construct a new reader, we call | |
| the constructor with an underlying stream and an encoding name: | |
| >>> from io import StringIO, BytesIO | |
| >>> from nltk.data import SeekableUnicodeStreamReader | |
| >>> stream = BytesIO(b"""\ | |
| ... This is a test file. | |
| ... It is encoded in ascii. | |
| ... """.decode('ascii').encode('ascii')) | |
| >>> reader = SeekableUnicodeStreamReader(stream, 'ascii') | |
| `SeekableUnicodeStreamReader`\ s support all of the normal operations | |
| supplied by a read-only stream. Note that all of the read operations | |
| return ``unicode`` objects (not ``str`` objects). | |
| >>> reader.read() # read the entire file. | |
| 'This is a test file.\nIt is encoded in ascii.\n' | |
| >>> reader.seek(0) # rewind to the start. | |
| >>> reader.read(5) # read at most 5 bytes. | |
| 'This ' | |
| >>> reader.readline() # read to the end of the line. | |
| 'is a test file.\n' | |
| >>> reader.seek(0) # rewind to the start. | |
| >>> for line in reader: | |
| ... print(repr(line)) # iterate over lines | |
| 'This is a test file.\n' | |
| 'It is encoded in ascii.\n' | |
| >>> reader.seek(0) # rewind to the start. | |
| >>> reader.readlines() # read a list of line strings | |
| ['This is a test file.\n', 'It is encoded in ascii.\n'] | |
| >>> reader.close() | |
| Size argument to ``read()`` | |
| --------------------------- | |
| The ``size`` argument to ``read()`` specifies the maximum number of | |
| *bytes* to read, not the maximum number of *characters*. Thus, for | |
| encodings that use multiple bytes per character, it may return fewer | |
| characters than the ``size`` argument: | |
| >>> stream = BytesIO(b"""\ | |
| ... This is a test file. | |
| ... It is encoded in utf-16. | |
| ... """.decode('ascii').encode('utf-16')) | |
| >>> reader = SeekableUnicodeStreamReader(stream, 'utf-16') | |
| >>> reader.read(10) | |
| 'This ' | |
| If a read block ends in the middle of the byte string encoding a | |
| single character, then that byte string is stored in an internal | |
| buffer, and re-used on the next call to ``read()``. However, if the | |
| size argument is too small to read even a single character, even | |
| though at least one character is available, then the ``read()`` method | |
| will read additional bytes until it can return a single character. | |
| This ensures that the ``read()`` method does not return an empty | |
| string, which could be mistaken for indicating the end of the file. | |
| >>> reader.seek(0) # rewind to the start. | |
| >>> reader.read(1) # we actually need to read 4 bytes | |
| 'T' | |
| >>> int(reader.tell()) | |
| 4 | |
| The ``readline()`` method may read more than a single line of text, in | |
| which case it stores the text that it does not return in a buffer. If | |
| this buffer is not empty, then its contents will be included in the | |
| value returned by the next call to ``read()``, regardless of the | |
| ``size`` argument, since they are available without reading any new | |
| bytes from the stream: | |
| >>> reader.seek(0) # rewind to the start. | |
| >>> reader.readline() # stores extra text in a buffer | |
| 'This is a test file.\n' | |
| >>> print(reader.linebuffer) # examine the buffer contents | |
| ['It is encoded i'] | |
| >>> reader.read(0) # returns the contents of the buffer | |
| 'It is encoded i' | |
| >>> print(reader.linebuffer) # examine the buffer contents | |
| None | |
| Seek and Tell | |
| ------------- | |
| In addition to these basic read operations, | |
| `SeekableUnicodeStreamReader` also supports the ``seek()`` and | |
| ``tell()`` operations. However, some care must still be taken when | |
| using these operations. In particular, the only file offsets that | |
| should be passed to ``seek()`` are ``0`` and any offset that has been | |
| returned by ``tell``. | |
| >>> stream = BytesIO(b"""\ | |
| ... This is a test file. | |
| ... It is encoded in utf-16. | |
| ... """.decode('ascii').encode('utf-16')) | |
| >>> reader = SeekableUnicodeStreamReader(stream, 'utf-16') | |
| >>> reader.read(20) | |
| 'This is a ' | |
| >>> pos = reader.tell(); print(pos) | |
| 22 | |
| >>> reader.read(20) | |
| 'test file.' | |
| >>> reader.seek(pos) # rewind to the position from tell. | |
| >>> reader.read(20) | |
| 'test file.' | |
| The ``seek()`` and ``tell()`` methods work property even when | |
| ``readline()`` is used. | |
| >>> stream = BytesIO(b"""\ | |
| ... This is a test file. | |
| ... It is encoded in utf-16. | |
| ... """.decode('ascii').encode('utf-16')) | |
| >>> reader = SeekableUnicodeStreamReader(stream, 'utf-16') | |
| >>> reader.readline() | |
| 'This is a test file.\n' | |
| >>> pos = reader.tell(); print(pos) | |
| 44 | |
| >>> reader.readline() | |
| 'It is encoded in utf-16.\n' | |
| >>> reader.seek(pos) # rewind to the position from tell. | |
| >>> reader.readline() | |
| 'It is encoded in utf-16.\n' | |
| Squashed Bugs | |
| ============= | |
| svn 5276 fixed a bug in the comment-stripping behavior of | |
| parse_sexpr_block. | |
| >>> from io import StringIO | |
| >>> from nltk.corpus.reader.util import read_sexpr_block | |
| >>> f = StringIO(b""" | |
| ... (a b c) | |
| ... # This line is a comment. | |
| ... (d e f\ng h)""".decode('ascii')) | |
| >>> print(read_sexpr_block(f, block_size=38, comment_char='#')) | |
| ['(a b c)'] | |
| >>> print(read_sexpr_block(f, block_size=38, comment_char='#')) | |
| ['(d e f\ng h)'] | |
| svn 5277 fixed a bug in parse_sexpr_block, which would cause it to | |
| enter an infinite loop if a file ended mid-sexpr, or ended with a | |
| token that was not followed by whitespace. A related bug caused | |
| an infinite loop if the corpus ended in an unmatched close paren -- | |
| this was fixed in svn 5279 | |
| >>> f = StringIO(b""" | |
| ... This file ends mid-sexpr | |
| ... (hello (world""".decode('ascii')) | |
| >>> for i in range(3): print(read_sexpr_block(f)) | |
| ['This', 'file', 'ends', 'mid-sexpr'] | |
| ['(hello (world'] | |
| [] | |
| >>> f = StringIO(b"This file has no trailing whitespace.".decode('ascii')) | |
| >>> for i in range(3): print(read_sexpr_block(f)) | |
| ['This', 'file', 'has', 'no', 'trailing'] | |
| ['whitespace.'] | |
| [] | |
| >>> # Bug fixed in 5279: | |
| >>> f = StringIO(b"a b c)".decode('ascii')) | |
| >>> for i in range(3): print(read_sexpr_block(f)) | |
| ['a', 'b'] | |
| ['c)'] | |
| [] | |
| svn 5624 & 5265 fixed a bug in ConcatenatedCorpusView, which caused it | |
| to return the wrong items when indexed starting at any index beyond | |
| the first file. | |
| >>> import nltk | |
| >>> sents = nltk.corpus.brown.sents() | |
| >>> print(sents[6000]) | |
| ['Cholesterol', 'and', 'thyroid'] | |
| >>> print(sents[6000]) | |
| ['Cholesterol', 'and', 'thyroid'] | |
| svn 5728 fixed a bug in Categorized*CorpusReader, which caused them | |
| to return words from *all* files when just one file was specified. | |
| >>> from nltk.corpus import reuters | |
| >>> reuters.words('training/13085') | |
| ['SNYDER', '&', 'lt', ';', 'SOI', '>', 'MAKES', ...] | |
| >>> reuters.words('training/5082') | |
| ['SHEPPARD', 'RESOURCES', 'TO', 'MERGE', 'WITH', ...] | |
| svn 7227 fixed a bug in the qc corpus reader, which prevented | |
| access to its tuples() method | |
| >>> from nltk.corpus import qc | |
| >>> qc.tuples('test.txt') | |
| [('NUM:dist', 'How far is it from Denver to Aspen ?'), ('LOC:city', 'What county is Modesto , California in ?'), ...] | |
| Ensure that KEYWORD from `comparative_sents.py` no longer contains a ReDoS vulnerability. | |
| >>> import re | |
| >>> import time | |
| >>> from nltk.corpus.reader.comparative_sents import KEYWORD | |
| >>> sizes = { | |
| ... "short": 4000, | |
| ... "long": 40000 | |
| ... } | |
| >>> exec_times = { | |
| ... "short": [], | |
| ... "long": [], | |
| ... } | |
| >>> for size_name, size in sizes.items(): | |
| ... for j in range(9): | |
| ... start_t = time.perf_counter() | |
| ... payload = "( " + "(" * size | |
| ... output = KEYWORD.findall(payload) | |
| ... exec_times[size_name].append(time.perf_counter() - start_t) | |
| ... exec_times[size_name] = sorted(exec_times[size_name])[4] # Get the median | |
| Ideally, the execution time of such a regular expression is linear | |
| in the length of the input. As such, we would expect exec_times["long"] | |
| to be roughly 10 times as big as exec_times["short"]. | |
| With the ReDoS in place, it took roughly 80 times as long. | |
| For now, we accept values below 30 (times as long), due to the potential | |
| for variance. This ensures that the ReDoS has certainly been reduced, | |
| if not removed. | |
| >>> exec_times["long"] / exec_times["short"] < 30 # doctest: +SKIP | |
| True | |