Search Packages

gauntlet-cli

Behavioral reliability under pressure. Test how LLMs behave when things get hard.

uqlm

UQLM: Uncertainty Quantification for Language Models, is a Python package for UQ-based LLM hallucination detection

eval-ai-library

Comprehensive AI Model Evaluation Framework with support for multiple LLM providers

agent-action-guard

🛡️ Safe AI Agents through Action Classifier

ragscore

⚡️ The "1-Minute RAG Audit" — Generate QA datasets & evaluate RAG systems in Colab, Jupyter, or CLI. Privacy-first, async, visual reports.

ai-stability

Measure LLM output consistency from the command line.

promptstats

Statistical analysis methods for comparing prompt and model performance in LLM evaluations.

evalstats

Statistical analysis methods for comparing prompt and model performance in LLM evaluations.

judicator

Who evaluates the evaluator? Judicator audits LLM-as-a-Judge systems for 7 documented bias types. Zero config. Works with any LLM.

chatbot-auditor

Quality auditor for AI chatbots. Analyzes your conversation logs to show where the bot is underperforming.

workflowbench

Lightweight benchmark harness for AI-driven business workflows

gaico

A Python library providing evaluation metrics to compare generated texts from LLMs, often against reference texts. Features streamlined workflows for model comparison and visualization.

aisert

Assert-style validation library for AI outputs - ensure your LLMs behave exactly as expected.

rail-score

DEPRECATED — use rail-score-sdk instead. This package redirects to rail-score-sdk.

factly-eval

CLI tool to evaluate LLM factuality on MMLU benchmark.

aspire-ai

Adversarial Student-Professor Internalized Reasoning Engine