A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust
V-gram builder library
Auto summarization from BPE tokenization
A byte pair encoding (BPE) tokenizer for chess portable game notation (PGN)
Text Tokenizer in C++
