Benchmark Python Packages

swebench

SWE-bench: Can Language Models Resolve Real-world Github Issues?

pytest-benchmark

pytest fixture for benchmarking code

mteb

MTEB: Massive Text Embedding Benchmark

pytest-harvest

Store data created during your `pytest` tests execution, and retrieve it at the end of the session, e.g. for applicative benchmarking purposes.

asv

Airspeed Velocity: A simple Python benchmarking tool with web-based reporting

motmetrics

:bar_chart: Benchmark multiple object trackers (MOT) in Python

evo

Python package for the evaluation of odometry and SLAM

google-benchmark

A microbenchmark support library

pyperformance

Python Performance Benchmark Suite

picows

Ultra-fast websocket client and server for asyncio

membrowse

Track and analyze binary size and memory footprint in embedded firmware

medmnist

[pip install medmnist] 18x Standardized Datasets for 2D and 3D Biomedical Image Classification

beir

A Heterogeneous Benchmark for Information Retrieval. Easy to use, evaluate your models across 15+ diverse IR datasets.

mmpose

OpenMMLab Pose Estimation Toolbox and Benchmark.

pytest-django-queries

Generate performance reports from your django database performance tests.

agentdojo

A Dynamic Environment to Evaluate Attacks and Defenses for LLM Agents.

evalplus

Rigourous evaluation of LLM-synthesized code - NeurIPS 2023 & COLM 2024

holobench

A package for benchmarking the characteristics of arbitrary functions

optunahub

Python library to use and implement packages in OptunaHub

lmms-eval

One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks

optimum-benchmark

🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of Optimum's hardware optimizations & quantization schemes.

apebench

[Neurips 2024] A benchmark suite for autoregressive neural emulation of PDEs. (≥46 PDEs in 1D, 2D, 3D; Differentiable Physics; Unrolled Training; Rollout Metrics)

logparser3

A machine learning toolkit for log parsing [ICSE'19, DSN'16]

gauntlet-cli

Behavioral reliability under pressure. Test how LLMs behave when things get hard.