Search Packages

pyonmttok

Fast and customizable text tokenization library with BPE and SentencePiece support

nfelib

nfelib - bindings Python para e ler e gerar XML de NF-e, NFS-e nacional, CT-e, MDF-e, BP-e

subword-nmt

Unsupervised Word Segmentation for Neural Machine Translation and Text Generation

rs-bpe

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

youtokentome

Unsupervised text tokenizer focused on computational efficiency

bpe

Byte Pair Encoding for Python!

tinybpe

This is an ultra-fast, lightweight and clean code implementation of the Byte Pair Encoding (BPE) algorithm.

bytetok

A fast, modular and light-weight BPE tokenizer for NLP research and prototyping.

kitoken

Fast and versatile tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization

tokenization-scorer

Simple-to-use scoring function for arbitrarily tokenized texts.

bpetokenizer

(py package) train your own tokenizer based on BPE algorithm for the LLMs (supports the regex pattern and special tokens)

runtoken

A blazing-fast BPE tokenizer for LLMs. Drop-in tiktoken replacement, 20-80x faster.

bpe-summarizer

Auto summarization from BPE tokenization

nlcodec

Natural Language EnCoder-Decoder: word, char, bpe etc

amharic-tokenizer

Amharic tokenizer with BPE-like merges over decomposed fidel (Cython)

pgn-tokenizer

A byte pair encoding (BPE) tokenizer for chess portable game notation (PGN)

tokenize2

Official Repository of Tokenize2 Tokenizers by TNSA

geobpe

Protein Structure Tokenization via Geometric Byte Pair Encoding (GeoBPE)

nfelib-xsdata

nfelib - bindings Python para e ler e gerar XML de NF-e, NFS-e nacional, CT-e, MDF-e, BP-e