Llm Testing Python Packages

gauntlet-cli

Behavioral reliability under pressure. Test how LLMs behave when things get hard.

langtest

Pacific AI provides a library for delivering safe & effective NLP models.

agentneo

Python SDK for Agent AI Observability, Monitoring and Evaluation Framework. Includes features like agent, llm and tools tracing, debugging multi-agentic system, self-hosted dashboard and advanced analytics with timeline and execution graph view

agentassay

Token-efficient stochastic testing for AI agents. 5-20x cost reduction. 10 framework adapters. Paper: arXiv:2603.02601

nlptest

Deliver safe & effective language models

infer-check

Correctness and reliability testing for LLM inference engines

llamator

Framework for testing vulnerabilities of GenAI systems.

ai-safety-tester

LLM security testing framework with CVE-style severity scoring and multi-model benchmarking

agentcloudkelp

YAML-first stress testing for AI agents. Write a contract, inject faults, catch behavioral drift, enforce cost budgets. No Python test code needed — just kelp.yaml and a terminal.

llm-behave

Behavioral testing for LLM applications. pytest plugin with semantic assertions, multi-turn conversation testing, and drift detection. No LLM judge needed.

api-test-ninja

API Testing Framework to automate and simplify API testing using LLM Agents and tests defined in plain English.

ccheck

MIT-licensed Framework for LLMs, RAGs, Chatbots testing. Configurable via YAML and integrable into CI pipelines for automated testing.

trajex

AI agent behavioral testing — learns what correct looks like, catches deviations automatically. Zero API keys needed.

mocktopus

🐙 Multi-armed mocks for LLM apps - Drop-in replacement for OpenAI/Anthropic APIs for deterministic testing

toolcallcheck

Deterministic Python testing for tool-using agents. Mock MCP tools, assert exact tool calls and trajectories, verify headers, and run offline in CI.

tinyqabenchmarkpp

Tiny QA Benchmark++ a micro-benchmark suite (52-item gold + on-demand multilingual synthetic packs), generator CLI, and CI-ready eval harness for ultra-fast LLM smoke-testing & regression-catching.

llm-sentry

Unified AI Reliability Platform. One install, 12 diagnostic engines. Zero-dependency LLM pipeline monitoring.

pyllmtest

🚀 Comprehensive testing framework for LLM applications with semantic assertions, multi-provider support, RAG testing, and prompt optimization. Test AI the right way!

agent-assembly-line

The simple way to build and embed AI agents into any software stack. Code-native, modular, and LLM-agnostic.

syncreus-eval

Evaluate your LLM apps with one function call. Hallucination detection, RAG scoring, and agent evals for OpenAI, Anthropic, and more. 14 evaluators, pytest plugin, composite trust scores.

llmtest-framework

pytest for LLM apps - Test for grounding failures, prompt injection, safety violations, and regressions

misalign

A Python library testing LLMs with prompts

ragaliq

LLM & RAG evaluation testing framework✨