Llm As A Judge Python Packages

agenta

The open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM observability all in one place.

llm-council-core

Multi-LLM council system with peer review and synthesis

ragscore

⚡️ The "1-Minute RAG Audit" — Generate QA datasets & evaluate RAG systems in Colab, Jupyter, or CLI. Privacy-first, async, visual reports.

verdict

Inference-time scaling for LLMs-as-a-judge.

dingo-python

Dingo: A Comprehensive AI Data, Model and Application Quality Evaluation Tool

scorable

The Python SDK for API of Scorable

judicator

Who evaluates the evaluator? Judicator audits LLM-as-a-Judge systems for 7 documented bias types. Zero config. Works with any LLM.

mcp-as-a-judge

MCP as a Judge is a behavioral MCP that strengthens AI coding assistants by requiring explicit LLM evaluations

root-signals

Scorable SDK

openevalkit

Production-grade Python framework for evaluating LLM and agentic systems with traditional scorers, LLM judges (OpenAI, Anthropic, Ollama, 100+ models via LiteLLM), ensemble aggregation, and smart caching for cost-effective testing.

veritail

Ecommerce search relevance evaluation tool

vllm-judge

A tiny, lightweight library for LLM-as-a-Judge evaluations on vLLM-hosted models.

pytest-llm-rubric

Pytest plugin for semantic PASS/FAIL checks using LLM-as-a-Judge

docling-sdg

A set of tools to create synthetically-generated data from documents

xfinder

[ICLR 2025] xFinder: Large Language Models as Automated Evaluators for Reliable Evaluation

llm-summary

Use an LLM to summarize paragraphs

artemis-agents

Production-ready multi-agent debate framework with adaptive evaluation and safety monitoring

root-signals-cli

CLI for the Root Signals API

scorable-cli

Scorable SDK

llm-antibodies

Antibodies for LLMs hallucinations (grouping LLM as a judge, NLI, reward models)

iflow-mcp-hepivax-mcp-as-a-judge

MCP as a Judge is a behavioral MCP that strengthens AI coding assistants by requiring explicit LLM evaluations

antibodies-rafaelsandroni

Antibodies for LLM hallucinations

dingo-client

Dingo: A Comprehensive AI Data, Model and Application Quality Evaluation Tool