Transformers Python Packages

tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

peft

🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.

ultralytics-thop

Profile PyTorch models for FLOPs and parameters, helping to evaluate computational efficiency and memory usage.

presidio-analyzer

An open-source framework for detecting, redacting, masking, and anonymizing sensitive data (PII) across text, images, and structured data. Supports NLP, pattern matching, and customizable pipelines.

presidio-anonymizer

An open-source framework for detecting, redacting, masking, and anonymizing sensitive data (PII) across text, images, and structured data. Supports NLP, pattern matching, and customizable pipelines.

mleap

MLeap: Deploy ML Pipelines to Production

optimum

🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools

openvino

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

x-transformers

A concise but complete full-attention transformer with a set of promising experimental features from various papers

spark-nlp

State of the Art Natural Language Processing

chronos-forecasting

Chronos: Pretrained Models for Time Series Forecasting

ipex-llm

Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, DeepSeek, Mixtral, Gemma, Phi, MiniCPM, Qwen-VL, MiniCPM-V, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM, DeepSpeed, Axolotl, etc.

haystack-ai

Open-source AI orchestration framework for building context-engineered, production-ready LLM applications. Design modular pipelines and agent workflows with explicit control over retrieval, routing, memory, and generation. Built for scalable agents, RAG, multimodal applications, semantic search, and conversational systems.

segmentation-models-pytorch

Semantic segmentation models with 500+ pretrained convolutional and transformer-based backbones.

curated-transformers

🤖 A PyTorch library of curated Transformer models and their composable components

nncf

Neural Network Compression Framework for enhanced OpenVINO™ inference

bertopic

Leveraging BERT and c-TF-IDF to create easily interpretable topics.

llm-guard

The Security Toolkit for LLM Interactions

megatron-core

Ongoing research training transformer models at scale

needle-python

Needle simplifies building RAG pipelines.

argostranslate

Open-source offline translation library written in Python

swanlab

⚡️SwanLab - an open-source, modern-design AI training tracking and visualization tool. Supports Cloud / Self-hosted use. Integrated with PyTorch / Transformers / verl / LLaMA Factory / ms-swift / Ultralytics / MMEngine / Keras etc.

openvino-dev

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

transformer-smaller-training-vocab

Temporary remove unused tokens during training to save ram and speed.