Update README.md

README.md

@@ -16,8 +16,7 @@ I undertook the following preprocessing steps:
  - Language identification with [langid](https://github.com/saffsd/langid.py)
  - Compute the ratio of Latin vocabulary in each sentence (against the digital-born vocab of the corpus)
  - Retain only sentences with a Latin vocabulary ratio of > 85%.
- - Retaining only lines containing letters of the Latin alphabet, numerals, and certain punctuation (--> `grep -P '^[A-z0-9ÄÖÜäöüÆ挜ᵫĀāūōŌ.,;:?!\- Ęę]+$' la.nolorem.tok.txt`
- - deduplication of the corpus
+ - Exclude all lines containing '^' --> hints at the presence of OCR errors.
 
 The result is a corpus of ~390 million tokens.