Search Packages

ginza-transformers

Use custom tokenizers in spacy-transformers

rs-bpe

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

llm-magnet

the small distributed language model toolkit. fine-tune state-of-the-art LLMs anywhere, rapidly.

tokenizerchanger

Library for manipulating the existing tokenizer.

tftokenizers

Use Huggingface Transformer and Tokenizers as Tensorflow Reusable SavedModels

precious-nlp

A tokenizer-free NLP library with T-FREE, CANINE, and byte-level approaches