Ai Testing Python Packages

langwatch-scenario

Agentic testing for agentic codebases

giskard

🐢 Open-Source Evaluation & Testing library for LLM Agents

prompture

Prompture is an API-first library for requesting structured JSON output from LLMs (or any structure), validating it against a schema, and running comparative tests between models.

aat-devqa

AI-powered automated E2E testing. Just enter a URL — AI generates and runs test scenarios.

langtest

Pacific AI provides a library for delivering safe & effective NLP models.

tailtester

tailtest — the test + security validator that lives inside Claude Code. Never blocks, never lies.

nlptest

Deliver safe & effective language models

tenro

Open-source simulation harness for testing AI agents. Simulate LLM and tool calls to test edge cases, failure paths, and agent logic without live API calls.

intentguard

A Python library for verifying code properties using natural language assertions.

chatbot-tracer

An automated approach for exploring and testing conversational agents using large language models. TRACER discovers chatbot functionalities, generates user profiles, and creates comprehensive test suites for conversational AI systems.

sharingan-autotest

Autonomous testing agent for Claude Code. Discovers, tests, diagnoses, and fixes your web app.

py-toolguard

The "Cloudflare for AI Agents". 7-layer security interceptor, real-time observability dashboard, and automated reliability testing for MCP and AI tool chains. Prevent hallucinations, prompt injection, and destructive tool calls.

maia-test-framework

A pytest-based framework for testing multi AI agents system. It provides a flexible and extensible platform for creating and running complex multi-agent simulations and capturing the results.

agentrial

Statistical evaluation framework for AI agents - pytest for agent trajectories

llm-behave

Behavioral testing for LLM applications. pytest plugin with semantic assertions, multi-turn conversation testing, and drift detection. No LLM judge needed.

alignmenter

Persona-aligned evaluation toolkit for auditing conversational AI authenticity, safety, and stability.

ccheck

MIT-licensed Framework for LLMs, RAGs, Chatbots testing. Configurable via YAML and integrable into CI pipelines for automated testing.

aetherlab

Official Python SDK for AetherLab's AI Guardrails and Compliance Platform

llm-sentry

Unified AI Reliability Platform. One install, 12 diagnostic engines. Zero-dependency LLM pipeline monitoring.

pyllmtest

🚀 Comprehensive testing framework for LLM applications with semantic assertions, multi-provider support, RAG testing, and prompt optimization. Test AI the right way!

llmtest-framework

pytest for LLM apps - Test for grounding failures, prompt injection, safety violations, and regressions

llmtestr

A new package that helps developers integration-test AI and LLM applications by validating structured outputs. It takes a user's test scenario or prompt as input, sends it to an LLM, and uses pattern

forge-nc

Local-first AI coding assistant with adversarial model testing, transparent context management, and cryptographic audit trails

housemonkey

Chaos testing for AI apps. 18 extreme personas attack your AI to find edge cases before users do. OWASP LLM Top 10 coverage.