Parallel Processing Python Packages

fastcore

Python supercharged for the fastai library

p-tqdm

Parallel processing with progress bars

accelerator

The Accelerator is a tool for fast and reproducible processing of eBay-scale datasets on a single computer.

pygmo

A Python platform to perform parallel computations of optimisation tasks (global and local) via the asynchronous generalized island model.

fastyaml-rs

Parse YAML at Rust speed. Full 1.2.2 spec, built-in linter, parallel processing. Native bindings for Python & Node.js.

envpool

C++-based high-performance parallel environment execution engine (vectorized env) for general RL environments.

dotflow

🎲 Dotflow turns an idea into flow! — Lightweight Python library for execution pipelines

glide

Easy ETL

structai

StructAI offers a robust toolkit for LLM interaction—such as structured outputs, context management, and parallel execution.

hypergol

An opinionated multithreaded Data Science framework

tfce-mediation

Fast regression and mediation analysis of vertex or voxel MRI data with TFCE

pyrallel-lib

Yet another easy-to-use python3 parallel library for humans.

ixtract

Self-tuning data extraction engine for Python that automatically optimizes parallelism and explains performance—no manual tuning required.

exp

python experiment management toolset

octochains

A lightweight, zero-dependency Python framework for parallel, isolated, collaborative AI reasoning.

parallelqueue

A package for simulating parallel queueing systems in Python.

tmi-viewer

tmi_viewer

pyexpool

Python Multi-Process Execution Pool: concurrent asynchronous execution pool with custom resource constraints (memory, timeouts, affinity, CPU cores and caching), load balancing and profiling capabilities of the external apps on NUMA architecture

mpfd

A decorator that makes a function into a parallel processor using multithreading.

mpunittest

Multi processing for the Python unittest library

abopus

Fast Python tool to convert audiobooks to Opus format with parallel processing. Optimized for voice content with automatic metadata/cover preservation, smart stereo handling, and significant file size reduction while maintaining quality.

multithread-parallel-processing

Library support parallel processing with multi-thread

easy-runner

⭐ A lightweight tool for efficiently managing and executing parallel experiments.

pprodigal

Python3 wrapper for parallelized gene prediction using Prodigal