Issue with Loading the Model
Dear Sirs,
I want to use this model for chunking purposes.
I have pulled the project and trying to load the model but I keep getting this error: "ValueError: The checkpoint you are trying to load has model type xlm-token but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date."
Initially I tried python 3.13 but didn't work.
I tried python 3.11 in order to use pytorch also didn't work.
I made sure to have the latest transformers too.
Also, I cant find the model paper since I keep getting Error 404 (https://huggingface.co/segment-any-text/sat-12l-sm/blob/main/arxiv.org/abs/2406.16678)
Same for model documentation: (https://huggingface.co/docs/transformers/main/en/model_doc/xlm-token#transformers.SubwordXLMForTokenClassification)
Even without pulling the project, if I just use
Use a pipeline as a high-level helper
from transformers import pipeline
pipe = pipeline("token-classification", model="segment-any-text/sat-12l-sm")
OR
Load model directly
from transformers import AutoModelForTokenClassification
model = AutoModelForTokenClassification.from_pretrained("segment-any-text/sat-12l-sm")
Both do not work and give the same error
It worked by using sat_sm = SaT("sat-12l-sm") but not model = AutoModelForTokenClassification.from_pretrained("segment-any-text/sat-12l-sm")
Hi! Indeed, models are supposed to be used as documented here: https://github.com/segment-any-text/wtpsplit?tab=readme-ov-file#usage