Tokenization Python Packages

spacy

💫 Industrial-strength Natural Language Processing (NLP) in Python

22.2M 34K 5K

sudachipy

Sudachi in Rust 🦀 and new generation of SudachiPy

1.9M 442 50

tokencost

Easy token price estimates for 400+ LLMs. TokenOps.

388K 2K 104

ff3

FPE - Format Preserving Encryption with FF3 in Python

262K 104 20

toonify

Toonify: Compact data format reducing LLM token usage by 30-60%

174K 337 24

razdel

Rule-based token, sentence segmentation for Russian language

99K 281 34

simplemma

Simple multilingual lemmatizer for Python, especially useful for speed and efficiency

94K 195 15

pyonmttok

Fast and customizable text tokenization library with BPE and SentencePiece support

62K 333 82

count-tokens

Count tokens in a text file.

45K 13 0

toon-llm

Token-Oriented Object Notation (TOON) is an LLM-optimized data serialization format implemented in Python.

28K 9 3

nlpcloud

NLP Cloud serves high performance pre-trained or custom models for NER, sentiment-analysis, classification, summarization, paraphrasing, intent classification, product description and ad generation, chatbot, grammar and spelling correction, keywords and keyphrases extraction, text generation, image generation, code generation, and more...

16K 86 8

quebra-frases

chunks strings into byte sized pieces

12K 1 3

youtokentome

Unsupervised text tokenizer focused on computational efficiency

11K 977 108

attacut

A Fast and Accurate Neural Thai Word Segmenter

8K 94 18

mayatok

MayaTok is a Byte Pair Encoding based Tokenizer.

6K 1 0

icetk

A unified tokenization tool for Images, Chinese and English.

4K 153 17

h-net-dynamic-chunking

H-Net Dynamic Chunking Modules

4K 71 2

spacy-streamlit

👑 spaCy building blocks and visualizers for Streamlit apps

4K 854 117

bitbadgespy-sdk

The most feature-rich tokenization standard ever built — TypeScript SDK for the BitBadges tokenization Cosmos SDK module

3K 0 0

ekphrasis

Text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction.

2K 675 94