Search Packages

spacy

💫 Industrial-strength Natural Language Processing (NLP) in Python

sudachipy

Sudachi in Rust 🦀 and new generation of SudachiPy

tokencost

Easy token price estimates for 400+ LLMs. TokenOps.

ff3

FPE - Format Preserving Encryption with FF3 in Python

toonify

Toonify: Compact data format reducing LLM token usage by 30-60%

razdel

Rule-based token, sentence segmentation for Russian language

simplemma

Simple multilingual lemmatizer for Python, especially useful for speed and efficiency

pyonmttok

Fast and customizable text tokenization library with BPE and SentencePiece support

count-tokens

Count tokens in a text file.

toon-llm

Token-Oriented Object Notation (TOON) is an LLM-optimized data serialization format implemented in Python.

nlpcloud

NLP Cloud serves high performance pre-trained or custom models for NER, sentiment-analysis, classification, summarization, paraphrasing, intent classification, product description and ad generation, chatbot, grammar and spelling correction, keywords and keyphrases extraction, text generation, image generation, code generation, and more...

quebra-frases

chunks strings into byte sized pieces

youtokentome

Unsupervised text tokenizer focused on computational efficiency

mayatok

MayaTok is a Byte Pair Encoding based Tokenizer.

attacut

A Fast and Accurate Neural Thai Word Segmenter

icetk

A unified tokenization tool for Images, Chinese and English.

spacy-streamlit

👑 spaCy building blocks and visualizers for Streamlit apps

bitbadgespy-sdk

The most feature-rich tokenization standard ever built — TypeScript SDK for the BitBadges tokenization Cosmos SDK module

rosette-api

Babel Street Analytics Client Library for Python

h-net-dynamic-chunking

H-Net Dynamic Chunking Modules

code-tokenize

Fast tokenization and structural analysis of any programming language

vaporetto

🛥 Vaporetto is a fast and lightweight pointwise prediction based tokenizer. This is a Python wrapper for Vaporetto.

ekphrasis

Text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction.

tivars

A Python library for interacting with TI-(e)z80 (82/83/84 series) calculator files