Evaluation Framework Python Packages

deepeval

The LLM Evaluation Framework

lm-eval

A framework for few-shot evaluation of language models.

lighteval

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

scandeval

The robust European language model benchmark.

irspack

Train, evaluate, and optimize implicit feedback-based recommender systems.

euroeval

The robust European language model benchmark.

letta-evals

Evaluation kit for testing stateful agents

agentlab

AgentLab: An open-source framework for developing, testing, and benchmarking web agents on diverse tasks, designed for scalability and reproducibility.

kiln-ai

Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.

aiverify-moonshot

Moonshot - A simple and modular tool to evaluate and red-team any LLM application.

pyrddlgym

A toolkit for auto-generation of OpenAI Gym environments from RDDL description files.

kaiko-eva

Evaluation framework for oncology foundation models (FMs)

zenoml

AI Data Management & Evaluation Platform

kiln-server

Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.

trustlens

Open-source Python library for evaluating ML model reliability beyond accuracy — with calibration, failure, and fairness diagnostics for informed deployment decisions.

lapixdl

Python package with Deep Learning utilities for Computer Vision

k-eval

Simple context-aware evaluation framework for AI agents using MCP.

ceval

CEval is a Python package for evaluating the quality of counterfactual explanations produced by any post-hoc XAI (Explainable AI) method. It computes 14 established metrics with a single call and works with diverse model architecture.

beyondbench

[ICLR 2026 Accepted paper] BeyondBench: Contamination-Resistant Evaluation of Reasoning in Language Models

judicator

Who evaluates the evaluator? Judicator audits LLM-as-a-Judge systems for 7 documented bias types. Zero config. Works with any LLM.

fieldtest

LLM evaluation framework — define what correct, well-formed, and safe means before you measure

quica

quica is a tool to run inter coder agreement pipelines in an easy and effective ways. Multiple measures are run and results are collected in a single table than can be easily exported in Latex

gval

Flexible, portable, and efficient geospatial evaluations for a variety of data.

repsys-framework

Framework for Interactive Evaluation of Recommender Systems