Model Evaluation Python Packages

gauntlet-cli

Behavioral reliability under pressure. Test how LLMs behave when things get hard.

debiai-gui

Bias detection and contextual evaluation tool for your AI projects

starwhale-bootstrap

an MLOps/LLMOps platform

pyaerocom

Python tools for climate and air quality model evaluation

fovux-mcp

Local-first YOLO workbench for edge-AI computer vision: MCP server + VS Code Studio for dataset inspection, training, evaluation, export, and RTSP inference.

starwhale

an MLOps/LLMOps platform

modeldiffx

Model behavioral diffing - compare LLM outputs across versions, detect regressions.

trustlens

Open-source Python library for evaluating ML model reliability beyond accuracy — with calibration, failure, and fairness diagnostics for informed deployment decisions.

ecp-runtime

ECP is a standardized interface for orchestrating, auditing, and enforcing authority limits in AI Agent evaluations. It moves evaluation from "brittle Python scripts" to a deterministic infrastructure protocol

ecp-sdk

ECP is a standardized interface for orchestrating, auditing, and enforcing authority limits in AI Agent evaluations. It moves evaluation from "brittle Python scripts" to a deterministic infrastructure protocol

judicator

Who evaluates the evaluator? Judicator audits LLM-as-a-Judge systems for 7 documented bias types. Zero config. Works with any LLM.

insurance-cv

Temporal and distributional cross-validation, and feature screening, for insurance pricing models

easymlselector

A model selection process for Machine Learning tasks on subset of training sample

machlearn

A Simple Yet Powerful Machine Learning Python Library

titus2

Titus 2 : Portable Format for Analytics (PFA) implementation for Python 3.4+

evalcards

Librería Python para generar reportes de evaluación (clasificación, regresión, forecasting) con métricas y gráficos listos en Markdown, JSON y pronto HTML.

llm-sentry

Unified AI Reliability Platform. One install, 12 diagnostic engines. Zero-dependency LLM pipeline monitoring.

metriculous

Measure and visualize machine learning model performance without the usual boilerplate.

scorecardbundle

A High-level Scorecard Modeling API | 评分卡建模尽在于此

utils-axn-2237

🧰 Essential EDA and Data Cleaning Helpers for Any DataFrame This collection of functions is designed to accelerate exploratory data analysis (EDA), quickly surface data quality issues, and offer high-level insights into the structure and content of your dataset.

data-science-snippets

A modular set of data science utilities for EDA, cleaning, and more.

skrobot

skrobot is a Python module for designing, running and tracking Machine Learning experiments / tasks. It is built on top of scikit-learn framework.

luna-ml

Luna ML - ML Leaderboard for your team with automatic model evaluation

diagnost

A diagnostics toolkit to help data scientists trust their models, not just train them.