Parallel Computing Python Packages

joblib

Computing with Python functions.

ctranslate2

Fast inference engine for Transformer models

swifter

A package which efficiently applies any function to a pandas dataframe or series in the fastest available manner

mpi4py

Python bindings for MPI

pqdm

Comfortable parallel TQDM using concurrent.futures

pyopencl

OpenCL integration for Python, plus shiny features

imops

Efficient parallelizable algorithms for multidimensional arrays to speed up your data pipelines

arm-mango

Parallel Hyperparameter Tuning in Python

adaptive

:chart_with_upwards_trend: Adaptive: parallel active learning of mathematical functions

bodo

High Performance Data Processing in Python

schwimmbad

A common interface to processing pools.

pygmo

A Python platform to perform parallel computations of optimisation tasks (global and local) via the asynchronous generalized island model.

pydisort

Modern Python & C++ API for the radiative transfer solver DISORT. Parallelized with PyTorch tensors; compile-free with: pip install pydisort.

cuda-cccl

CUDA Core Compute Libraries

rmath-py

Drop-in numerical accelerator for Python, powered by Rust. High-performance arrays, linear algebra, and statistics — outside the GIL.

adaptive-scheduler

Run many functions (adaptively) on many cores (>10k-100k) using mpi4py.futures, ipyparallel, loky, or dask-mpi. :tada:

pmmc

Python bindings for Mesh-based Monte Carlo (MMC) photon transport simulator

pipefunc

Lightweight fast function pipeline (DAG) creation in pure Python for scientific (HPC) workflows 🕸️🧪

ultranest

Fit and compare complex models reliably and rapidly. Advanced nested sampling.

tensorpac

Phase-Amplitude Coupling under Python

psyclone

PSyclone is a source-to-source Fortran compiler designed to programmatically optimise, parallelise and instrument HPC applications via user-provided transformation scripts.

optimagic

optimagic is a Python package for numerical optimization. It is a unified interface to optimizers from SciPy, NlOpt and other packages. optimagic's minimize function works just like SciPy's, so you don't have to adjust your code. You simply get more optimizers for free. On top you get diagnostic tools, parallel numerical derivatives and more.

hyperactive

A unified interface for optimization algorithms and experiments

mtkahypar

Mt-KaHyPar (Multi-Threaded Karlsruhe Hypergraph Partitioner) is a shared-memory multilevel graph and hypergraph partitioner equipped with parallel implementations of techniques used in the best sequential partitioning algorithms. Mt-KaHyPar can partition extremely large hypergraphs very fast and with high quality.