Evaluation Python Packages

langsmith

LangSmith Client SDK Implementations

mlflow-skinny

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

mlflow

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

simpleeval

Simple Safe Sandboxed Extensible Expression Evaluator for Python

mlflow-tracing

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

evaluate

🤗 Evaluate: A library for easily evaluating machine learning models and datasets.

opik

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

ragas

Supercharge Your LLM Application Evaluations 🚀

faster-coco-eval

Continuation of an abandoned project fast-coco-eval

evo

Python package for the evaluation of odometry and SLAM

spreadscript

SpreadScript: Use a spreadsheet as a function.

ranx

⚡️A Blazing-Fast Python Library for Ranking Evaluation, Comparison, and Fusion 🐍

opik-optimizer

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

agenta

The open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM observability all in one place.

pytrec-eval

pytrec_eval is an Information Retrieval evaluation tool for Python, based on the popular trec_eval.

evalidate

Safe and fast evaluation of untrusted user-supplied python expressions

evalscope

A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.

sprint-toolkit

SPRINT: A Unified Toolkit for Evaluating and Demystifying Zero-shot Neural Sparse Retrieval

unitxt

🦄 Unitxt is a Python library for enterprise-grade evaluation of AI performance, offering the world's largest catalog of tools and data for end-to-end AI benchmarking

kubetorch

Distribute and run AI workloads on Kubernetes magically in Python, like PyTorch for ML infra.

runhouse

Distribute and run AI workloads on Kubernetes magically in Python, like PyTorch for ML infra.

lighteval

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

fore

The fore client package

evalica

Evalica, your favourite evaluation toolkit