Search Packages

count-tokens

Count tokens in a text file.

rs-bpe

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

pyrwkv-tokenizer

A fast RWKV Tokenizer written in Rust

rwkv-tokenizer

A fast RWKV Tokenizer written in Rust

pytokencounter

A simple Python library for tokenizing text and counting tokens. While currently only supporting OpenAI LLMs, it helps with text processing and managing token limits in AI applications.

toksight

Tokenizer analysis toolkit. Compare vocabulary coverage, compression ratios, and token boundaries across GPT-4o, Llama 3, Mistral, and any HuggingFace tokenizer.

promptminify

Minify LLM prompts to save tokens — domain-aware, tiktoken-validated, zero regressions

runtoken

A blazing-fast BPE tokenizer for LLMs. Drop-in tiktoken replacement, 20-80x faster.

openai-function-tokens

Predict the exact openai token usage of functions

slash-tokens

Token Optimization for Context Engineers. 4.8 KB WASM. Sub-millisecond. Zero dependencies.

tokdu

tokdu (Token Disk Usage) is a terminal-based utility that helps you analyze and visualize token usage in your codebase. Similar to the classic du (disk usage) command, tokdu shows you how many tokens your files and directories consume, which is essential when working with Large Language Models (LLMs) that have token limits.

chatgpt-long-term-memory

The ChatGPT Long Term Memory package is a powerful tool designed to empower your projects with the ability to handle a large number of simultaneous users and external sources.

llm-tokenscope

Profile your LLM payloads. Find the waste. Cut the cost. Field-level token attribution, cost leak detection, and payload optimization for any LLM API.