Evaluation Metrics Python Packages

deepeval

The LLM Evaluation Framework

agentops

Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI

faster-coco-eval

Continuation of an abandoned project fast-coco-eval

unbabel-comet

A Neural Framework for MT Evaluation

ranx

⚡️A Blazing-Fast Python Library for Ranking Evaluation, Comparison, and Fusion 🐍

nervaluate

Full named-entity (i.e., not tag/token) evaluation metrics based on SemEval’13

unitxt

🦄 Unitxt is a Python library for enterprise-grade evaluation of AI performance, offering the world's largest catalog of tools and data for end-to-end AI benchmarking

cd-fvd

[CVPR 2024] On the Content Bias in Fréchet Video Distance

lighteval

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

fast-bss-eval

A fast implementation of bss_eval metrics for blind source separation

rliable

[NeurIPS'21 Outstanding Paper] Library for reliable evaluation on RL and ML benchmarks, even with only a handful of seeds.

permetrics

Artificial intelligence (AI, ML, DL) performance metrics implemented in Python

pyrouge

A Python wrapper for the ROUGE summarization evaluation package

codebleu

Pip compatible CodeBLEU metric implementation available for linux/macos/win

pynlpl

PyNLPl, pronounced as 'pineapple', is a Python library for Natural Language Processing. It contains various modules useful for common, and less common, NLP tasks. PyNLPl can be used for basic tasks such as the extraction of n-grams and frequency lists, and to build simple language model. There are also more complex data types and algorithms. Moreover, there are parsers for file formats common in NLP (e.g. FoLiA/Giza/Moses/ARPA/Timbl/CQL). There are also clients to interface with various NLP specific servers. PyNLPl most notably features a very extensive library for working with FoLiA XML (Format for Linguistic Annotation).

ir-metrics

The most common information retrieval (IR) metrics

prdc

Code base for the precision, recall, density, and coverage metrics for generative models. ICML 2020.

octis

OCTIS: Comparing Topic Models is Simple! A python package to optimize and evaluate topic models (accepted at EACL2021 demo track)

classeval

Evaluation of supervised predictions for two-class and multi-class classifiers

nf1

A novel approach to evaluate community detection algorithms on ground truth

pytolemaic

Toolbox for analysis of model's quality and model's description. For further details see

survivaleval

The most comprehensive Python package for evaluating survival analysis models.

k-eval

Simple context-aware evaluation framework for AI agents using MCP.

ecp-runtime

ECP is a standardized interface for orchestrating, auditing, and enforcing authority limits in AI Agent evaluations. It moves evaluation from "brittle Python scripts" to a deterministic infrastructure protocol