kl3m-doc-small-uncased-001

kl3m-doc-small-uncased-001 is a domain-specific masked language model (MLM) based on the RoBERTa architecture, specifically designed for legal and financial document analysis. With approximately 236M parameters, it provides a balanced model for specialized NLP tasks in both fill-mask prediction and feature extraction for document embeddings. This uncased variant is particularly useful for case-insensitive applications, maintaining strong performance while disregarding capitalization differences.

Model Details

Architecture: RoBERTa
Size: 236M parameters
Hidden Size: 1024
Layers: 8
Attention Heads: 8
Intermediate Size: 4096
Max Position Embeddings: 512
Max Sequence Length: 509
Tokenizer: alea-institute/kl3m-004-128k-uncased
Vector Dimension: 1024 (hidden_size)
Pooling Strategy: CLS token or mean pooling

Use Cases

This model is particularly useful for:

Document classification in legal and financial domains
Entity recognition for specialized terminology
Understanding legal citations and references
Filling in missing terms in legal documents
Feature extraction for downstream legal analysis tasks
Document similarity and retrieval tasks where capitalization is not significant
Applications requiring a balance between performance and model size

The uncased nature of this model makes it efficient for scenarios where case distinctions are not important to the task, while its small size offers a good balance between performance and computational requirements.

Standard Test Examples

Using our standardized test examples for comparing embedding models:

Fill-Mask Results

Contract Clause Heading:
"<|cls|> 8. representations and<|mask|>. each party hereby represents and warrants to the other party as of the date hereof as follows: <|sep|>"

Top 5 predictions:
1. warranties (0.940)
2. warranty (0.016)
3. warranties (0.015)
4. covenants (0.005)
5. warrants (0.004)
Defined Term Example:
"<|cls|> \"effective<|mask|>\" means the date on which all conditions precedent set forth in article v are satisfied or waived by the administrative agent. <|sep|>"

Top 5 predictions:
1. date (0.989)
2. time (0.005)
3. deadline (0.003)
4. day (0.001)
5. dates (0.0005)
Regulation Example:
"<|cls|> all transactions shall comply with the requirements set forth in the truth in<|mask|> act and its implementing regulation z. <|sep|>"

Top 5 predictions:
1. lending (0.609)
2. information (0.075)
3. practices (0.049)
4. claims (0.020)
5. lending (0.019)

Document Similarity Results

Using the standardized document examples for embeddings:

Document Pair	Cosine Similarity (CLS token)	Cosine Similarity (Mean pooling)
Court Complaint vs. Consumer Terms	0.429	0.440
Court Complaint vs. Credit Agreement	0.435	0.572
Consumer Terms vs. Credit Agreement	0.562	0.498

The small-sized model shows moderate document similarity performance, with better differentiation between document types than some larger models in the family. The model demonstrates balanced performance using both CLS token and mean pooling strategies.

Usage

Masked Language Modeling

You can use this model for masked language modeling with the simple pipeline approach:

from transformers import pipeline

# Load the fill-mask pipeline with the model
fill_mask = pipeline('fill-mask', model="alea-institute/kl3m-doc-small-uncased-001")

# Example: Contract clause heading
# Note the mask token placement - directly adjacent to "and" without space
text = "<|cls|> 8. representations and<|mask|>. each party hereby represents and warrants to the other party as of the date hereof as follows: <|sep|>"
results = fill_mask(text)

# Display predictions
print("Top predictions:")
for result in results:
    print(f"- {result['token_str']} (score: {result['score']:.3f})")

# Output:
# Top predictions:
# - warranties (score: 0.940)
# - warranty (score: 0.016)
# - warranties (score: 0.015)
# - covenants (score: 0.005)
# - warrants (score: 0.004)

Feature Extraction for Embeddings

from transformers import pipeline
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

# Load the feature-extraction pipeline
extractor = pipeline('feature-extraction', model="alea-institute/kl3m-doc-small-uncased-001", return_tensors=True)

# Example legal documents (truncated for brevity)
texts = [
    # Court Complaint
    "<|cls|> in the united states district court for the eastern district of pennsylvania\n\njohn doe,\nplaintiff,\n\nvs.\n\nacme corporation,\ndefendant. <|sep|>",
    
    # Consumer Terms
    "<|cls|> terms and conditions\n\nlast updated: april 10, 2025\n\nthese terms and conditions govern your access to and use of the service. <|sep|>",
    
    # Credit Agreement
    "<|cls|> credit agreement\n\ndated as of april 10, 2025\n\namong\n\nacme borrower inc.,\nas the borrower,\n\nand bank of finance,\nas administrative agent. <|sep|>"
]

# Generate embeddings for each document
embeddings = []
for text in texts:
    # Get features for the text
    features = extractor(text)
    
    # Extract the CLS token embedding (first token)
    # Convert to numpy if needed
    features_array = features[0].numpy() if hasattr(features[0], 'numpy') else features[0]
    cls_embedding = features_array[0]  # Shape: [hidden_size]
    
    embeddings.append(cls_embedding)

# Calculate cosine similarity between documents
similarity_matrix = cosine_similarity(embeddings)
print("\nDocument similarity matrix:")
print(similarity_matrix)

For more advanced usage with mean pooling:

# Mean pooling - taking average of all token embeddings
def mean_pooling(features):
    # Convert to numpy if needed
    features_array = features[0].numpy() if hasattr(features[0], 'numpy') else features[0]
    return np.mean(features_array, axis=0)

# Generate mean-pooled embeddings
mean_embeddings = [mean_pooling(extractor(text)) for text in texts]

# Calculate similarity with mean pooling
mean_similarity = cosine_similarity(mean_embeddings)
print("\nMean pooling similarity matrix:")
print(mean_similarity)

Training

The model was trained on a diverse corpus of legal and financial documents, ensuring high-quality performance in these domains. It leverages the KL3M tokenizer which provides 9-17% more efficient tokenization for domain-specific content than cl100k_base or the LLaMA/Mistral tokenizers.

Training included both masked language modeling (MLM) objectives and attention to dense document representation for retrieval and classification tasks. The model was trained on lowercase text to improve efficiency while maintaining strong performance.

Intended Usage

This model is intended for both:

Masked Language Modeling: Filling in missing words/terms in legal and financial documents
Document Embedding: Generating fixed-length vector representations for document similarity and classification

It is particularly well-suited for applications requiring a good balance between model size and performance, and where case-sensitivity is not important.

Special Tokens

This model includes the following special tokens:

CLS token: <|cls|> (ID: 5) - Used for the beginning of input text
MASK token: <|mask|> (ID: 6) - Used to mark tokens for prediction
SEP token: <|sep|> (ID: 4) - Used for the end of input text
PAD token: <|pad|> (ID: 2) - Used for padding sequences to a uniform length
BOS token: <|start|> (ID: 0) - Beginning of sequence
EOS token: <|end|> (ID: 1) - End of sequence
UNK token: <|unk|> (ID: 3) - Unknown token

Important usage notes:

When using the MASK token for predictions, be aware that this model uses a space-prefixed BPE tokenizer. The <|mask|> token should be placed IMMEDIATELY after the previous token with NO space, because most tokens in this tokenizer have an initial space encoded within them. For example: "word<|mask|>" rather than "word <|mask|>".

This space-aware placement is crucial for getting accurate predictions.

Limitations

While providing a good balance between size and performance, this model has some limitations:

Moderate parameter count (236M) compared to larger language models
Uncased nature means it cannot distinguish between different capitalizations
Primarily focused on English legal and financial texts
Best suited for domain-specific rather than general-purpose tasks
Maximum sequence length of 512 tokens may require chunking for lengthy documents
Requires domain expertise to interpret results effectively

References

Citation

If you use this model in your research, please cite:

@misc{kl3m-doc-small-uncased-001,
  author = {ALEA Institute},
  title = {kl3m-doc-small-uncased-001: A Domain-Specific Uncased Language Model for Legal and Financial Text Analysis},
  year = {2025},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/alea-institute/kl3m-doc-small-uncased-001}}
}

@article{bommarito2025kl3m,
  title={KL3M Tokenizers: A Family of Domain-Specific and Character-Level Tokenizers for Legal, Financial, and Preprocessing Applications},
  author={Bommarito, Michael J and Katz, Daniel Martin and Bommarito, Jillian},
  journal={arXiv preprint arXiv:2503.17247},
  year={2025}
}

@misc{bommarito2025kl3mdata,
  title={The KL3M Data Project: Copyright-Clean Training Resources for Large Language Models},
  author={Bommarito II, Michael J. and Bommarito, Jillian and Katz, Daniel Martin},
  year={2025},
  eprint={2504.07854},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}

License

This model is licensed under CC-BY 4.0.

Contact

The KL3M model family is maintained by the ALEA Institute. For technical support, collaboration opportunities, or general inquiries:

Email: [email protected]
Website: https://aleainstitute.ai
GitHub: https://github.com/alea-institute/kl3m-model-research

alea-institute
/

kl3m-doc-small-uncased-001