kl3m-004-correction-001

kl3m-004-correction-001 is a small, ~500M parameter language model designed to assist in the correction of common typing, spelling, OCR, and format issues in English text, especially in the financial and legal domains.

Notably, this model has been trained on the alea-institute/kl3m-004-char-8k-cased tokenizer, which is a BPE tokenizer trained with a 3-character maximum token constraint. This character-level tokenization enables the model to effectively handle character-level corrections and transformations.

Model Details

Architecture: LlamaForCausalLM
Size: 478.2M parameters
Hidden Size: 1024
Layers: 32
Attention Heads: 16
Key-Value Heads: 16
Intermediate Size: 4096
Max Position Embeddings: 512
Context Window: 512 tokens (no ROPE)
Language(s): Primarily English
Tokenizer: kl3m-004-char-8k-cased BPE tokenizer (8,192 tokens, between 1-3 characters each)
Training Objective: Next token prediction with special tokens for correction
Developed by: ALEA Institute
License: CC-BY 4.0
Hardware Requirements: Runs real-time in fp32 on CPU or consumer NV/AMD GPUs

Use Cases

kl3m-004-correction-001 is particularly effective for:

Correcting common typing or spelling errors
Correcting common OCR errors
Correcting common formatting errors
Text normalization
Character-level transformations

Key Features

Character-Level Processing: Utilizes a character-based tokenizer (1-3 chars per token) for fine-grained text manipulation
Clean Training Data: Built on what was originally referred to as the Kelvin Legal DataPack, ensuring all training data is ethically sourced and legally permissible
Low Toxicity: Empirically lower toxicity and bias
Enterprise Focus: Specifically designed for legal, regulatory, and financial workflows
Efficient Deployment: Optimized for real-time inference on consumer hardware

Usage

Simply prompt the model with the original text, followed by the <|sep|> token, and wait for stop token (<|end|>) generation. You can use pipeline to handle this for you.

Deterministic Correction

In many situations, deterministic correction (i.e., most probable logit sequence) is fine.

from transformers import pipeline
p = pipeline('text-generation', 'alea-institute/kl3m-004-correction-001', device='cpu')

text = "Tne Uni+ed 5tates is nct responsib|e for 5uch pr0duction"

correction = p(text + "<|sep|>", max_new_tokens=512, return_full_text=False)[0]['generated_text']

# Output: The United States is not responsible for such production

Sampled with Frequency Weighting

In other situations, it can be useful to generate multiple corrections with a sampler and evaluate the distribution. For example:

using a string or token-based distance metric to score or rank corrections
showing multiple suggestions to a user with frequency-weighted order

from transformers import pipeline
from collections import Counter

p = pipeline('text-generation', 'alea-institute/kl3m-004-correction-001', device='cuda')

text = "Tne Uni+ed 5tates is nct responsib|e for 5uch pr0duction"

corrections = Counter(
  [
    g['generated_text']
    for g in p(
      text + "<|sep|>",
      max_new_tokens=512,
      return_full_text=False,
      temperature=0.5,
      # top_p, top_k, custom sampler, etc.
      do_sample=True,
      num_return_sequences=10
    )
  ]
).most_common(3)

# Output: [('The United States is not responsible for such production', 7), ('the United States is not responsible for such production', 3)]

Generation Parameters

The model supports various parameters to control the generation process:

temperature: Controls randomness (lower = more deterministic)
top_p: Nucleus sampling parameter (lower = more focused)
top_k: Limits vocabulary selection to top k tokens
max_new_tokens: Maximum number of tokens to generate
do_sample: Whether to use sampling vs. greedy decoding
num_return_sequences: Number of different sequences to generate

Training

This model was originally trained for 3 days on 1xRTX3090, and a larger ~3B parameter MoE version is pending release.

Training Data

This model was trained on a dataset generated with the KL3M data collection and the alea-data-generator library, which can create realistic synthetic samples using traditional (non-generative) techniques.

The source code to retrieve and process this dataset is available here: https://github.com/alea-institute/kl3m-data

Some pre-tokenized subsets of the KL3M data collection are available on Hugging Face: https://huggingface.co/datasets?sort=most_rows&search=kl3m-data

Complete, raw data is available upon request at this time via S3 under a Requester Pays model. We are actively working on a zero-cost distribution model as soon as we can obtain additional support.

Special Tokens

kl3m-004-correction-001 uses the following special tokens from the kl3m-004-char-8k-cased tokenizer:

<|start|>: Token ID 0
<|end|>: Token ID 1
<|pad|>: Token ID 2
<|unk|>: Token ID 3
<|sep|>: Token ID 4 (used as a separator between the input text and corrected output)
<|cls|>: Token ID 5
<|mask|>: Token ID 6

Intended Usage

This model is intended for use in:

Text preprocessing pipelines for legal and financial documents
OCR post-processing for document digitization
Autocorrection features in text editors
Data cleaning workflows
Quality assurance for document management systems

Limitations

Limited to a 512 token context window
Primarily focused on English language corrections
May not handle complex semantic errors or context-dependent corrections
Not designed for translation or content generation
Limited to character-level and simple syntax corrections

Source

https://github.com/alea-institute/kl3m-model-research

References

KL3M Tokenizers: A Family of Domain-Specific and Character-Level Tokenizers for Legal, Financial, and Preprocessing Applications
Additional tokenizer, dataset, and model publications are pending.

Citation

@misc{kl3m-004-correction-001,
  author = {ALEA Institute},
  title = {kl3m-004-correction-001: A Character-Level Text Correction Model},
  year = {2024},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/alea-institute/kl3m-004-correction-001}}
}

@article{bommarito2025kl3m,
  title={KL3M Tokenizers: A Family of Domain-Specific and Character-Level Tokenizers for Legal, Financial, and Preprocessing Applications},
  author={Bommarito, Michael J and Katz, Daniel Martin and Bommarito, Jillian},
  journal={arXiv preprint arXiv:2503.17247},
  year={2025}
}

License

Model weights are released under the CC-BY 4.0 License.

Contact

The KL3M model family is maintained by the ALEA Institute. For technical support, collaboration opportunities, or general inquiries:

GitHub: https://github.com/alea-institute/kl3m-model-research
Email: [email protected]
Website: https://aleainstitute.ai