Sudachi in Rust 🦀 and new generation of SudachiPy
A fast, modular and light-weight BPE tokenizer for NLP research and prototyping.
