This is an adapted tokenizer from GPT2 that can recognize tokens to do with Python coding. It is part of the huggingfaceNLP course exercise. It uses the method train_new_from_iterator()

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Examples

Text Generation

This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Skier8402
/

code-search-net-tokenizer

Dataset used to train Skier8402/code-search-net-tokenizer