Search Packages

rs-bpe

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

tinybpe

This is an ultra-fast, lightweight and clean code implementation of the Byte Pair Encoding (BPE) algorithm.

hf-tokenizer-visualizer

Visualize HuggingFace Byte-Pair Encoding (BPE) Tokenizer encoding process

tokenizerz

BPE tokenizer for LLMs in Pure Zig

rbpe

R-BPE: Improving BPE-Tokenizers with Token Reuse

amharic-tokenizer

Amharic tokenizer with BPE-like merges over decomposed fidel (Cython)

tokenize2

Official Repository of Tokenize2 Tokenizers by TNSA