Spaces:
Sleeping
Sleeping
File size: 2,060 Bytes
d916065 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 |
.. Copyright (C) 2001-2023 NLTK Project
.. For license information, see LICENSE.TXT
Crubadan Corpus Reader
======================
Crubadan is an NLTK corpus reader for ngram files provided
by the Crubadan project. It supports several languages.
>>> from nltk.corpus import crubadan
>>> crubadan.langs()
['abk', 'abn',..., 'zpa', 'zul']
----------------------------------------
Language code mapping and helper methods
----------------------------------------
The web crawler that generates the 3-gram frequencies works at the
level of "writing systems" rather than languages. Writing systems
are assigned internal 2-3 letter codes that require mapping to the
standard ISO 639-3 codes. For more information, please refer to
the README in nltk_data/crubadan folder after installing it.
To translate ISO 639-3 codes to "Crubadan Code":
>>> crubadan.iso_to_crubadan('eng')
'en'
>>> crubadan.iso_to_crubadan('fra')
'fr'
>>> crubadan.iso_to_crubadan('aaa')
In reverse, print ISO 639-3 code if we have the Crubadan Code:
>>> crubadan.crubadan_to_iso('en')
'eng'
>>> crubadan.crubadan_to_iso('fr')
'fra'
>>> crubadan.crubadan_to_iso('aa')
---------------------------
Accessing ngram frequencies
---------------------------
On initialization the reader will create a dictionary of every
language supported by the Crubadan project, mapping the ISO 639-3
language code to its corresponding ngram frequency.
You can access individual language FreqDist and the ngrams within them as follows:
>>> english_fd = crubadan.lang_freq('eng')
>>> english_fd['the']
728135
Above accesses the FreqDist of English and returns the frequency of the ngram 'the'.
A ngram that isn't found within the language will return 0:
>>> english_fd['sometest']
0
A language that isn't supported will raise an exception:
>>> crubadan.lang_freq('elvish')
Traceback (most recent call last):
...
RuntimeError: Unsupported language.
|