Quantization Python Packages

ctranslate2

Fast inference engine for Transformer models

faster-whisper

Faster Whisper transcription with CTranslate2

bitsandbytes

Accessible large language models via k-bit quantization for PyTorch.

torchao

PyTorch native quantization and sparsity for training and inference

optimum

🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools

onnx2tf

A tool for converting ONNX files to LiteRT/TFLite/TensorFlow, PyTorch native code (nn.Module), TorchScript (.pt), state_dict (.pt), Exported Program (.pt2), and Dynamo ONNX. It also supports direct conversion from LiteRT to PyTorch.

nncf

Neural Network Compression Framework for enhanced OpenVINO™ inference

llmcompressor

Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM

optimum-quanto

A pytorch quantization backend for optimum

sageattention

[ICLR2025, ICML2025, NeurIPS2025 Spotlight] Quantized Attention achieves speedup of 2-5x compared to FlashAttention, without losing end-to-end metrics across language, image, and video models.

tensorflow-model-optimization

A toolkit to optimize ML models for deployment for Keras and TensorFlow, including quantization and pruning.

auto-gptq

An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.

auto-round

A SOTA quantization algorithm for high-accuracy low-bit LLM inference, seamlessly optimized for CPU/XPU/CUDA, with multi-datatype support and full compatibility with vLLM, SGLang, and Transformers.

navec

Compact high quality word embeddings for Russian language

quantcpp

LLM inference with 7x longer context. Pure C, zero dependencies. Lossless KV cache compression + single-header library.

gptqmodel

LLM model quantization (compression) toolkit with HW acceleration support for Nvidia, AMD, Intel GPU and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.

intel-extension-for-pytorch

A Python package for extending the official PyTorch that can easily obtain performance on Intel platform

tqdb

Embedded vector database in Rust with Python bindings — TurboQuant algorithm (arXiv:2504.19874), zero training, 2–4 bit compression, HNSW ANN search, WAL persistence

aimet-torch

AIMET is a library that provides advanced quantization and compression techniques for trained neural network models.

llamafactory

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

neural-compressor

SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime

qonnx

QONNX: Arbitrary-Precision Quantized Neural Networks in ONNX

brevitas

Brevitas: neural network quantization in PyTorch

aimet-onnx

AIMET is a library that provides advanced quantization and compression techniques for trained neural network models.