Benchmarking Python Packages

locust

Write scalable load tests in plain Python 🚗💨

pytest-benchmark

pytest fixture for benchmarking code

pytest-codspeed

A pytest plugin to create benchmarks

pyperf

Toolkit to run Python benchmarks

asv-runner

Core pure python benchmarking for asv

openml

OpenML's Python API for a World of Data and More 💫

openjij

OpenJij : Framework for the Ising model and QUBO.

pytest-django-queries

Generate performance reports from your django database performance tests.

fev

Forecast evaluation library

holobench

A package for benchmarking the characteristics of arbitrary functions

rliable

[NeurIPS'21 Outstanding Paper] Library for reliable evaluation on RL and ML benchmarks, even with only a handful of seeds.

salesforce-merlion

Merlion: A Machine Learning Framework for Time Series Intelligence

reframe-hpc

A powerful Python framework for writing and running portable regression tests and benchmarks for HPC systems.

cmind

Collective Knowledge (CK), Collective Mind (CM/CMX) and MLPerf automations: community-driven projects to facilitate collaborative and reproducible research and to learn how to run AI, ML, and other emerging workloads more efficiently and cost-effectively across diverse models, datasets, software, and hardware using MLPerf methodology and benchmarks

arthur-observability-sdk

Make AI work for Everyone - Monitoring and governing for your AI/ML

benchexec

BenchExec: A Framework for Reliable Benchmarking and Resource Measurement

isat-tuner

Inference Stack Auto-Tuner — one command to find the fastest ONNX inference config for any model on any GPU. Bayesian optimization, Pareto analysis, thermal-aware benchmarking, multi-provider (ROCm/CUDA/TensorRT/OpenVINO).

opendsstar

OpenDsStar is an open-source implementation of the DS-Star agent that replaces file-based workflows with a flexible, tool-centric architecture. It supports incremental execution, reuses intermediate results, and makes complex multi-step agents more modular, efficient, and extensible.

autorag

AutoRAG: An Open-Source Framework for Retrieval-Augmented Generation (RAG) Evaluation & Optimization with AutoML-Style Automation

yuclid

Orchestrating experiments on N-dimensional parameter spaces with custom metric collection + sliced-space visualization

dagzoo

Synthetic tabular data generator for causal modeling

easy-tpp

EasyTPP: Towards Open Benchmarking Temporal Point Processes

mqt-bench

MQT Bench - An MQT Tool for Benchmarking Quantum Software Tools

pythae

Unifying Variational Autoencoder (VAE) implementations in Pytorch (NeurIPS 2022)