Distributed Systems Python Packages

xgboost

Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C++ and more. Runs on single machine, Hadoop, Spark, Dask, Flink and DataFlow

fugue

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.

fugue-sql-antlr

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.

faststream

FastStream is a powerful and easy-to-use asynchronous Python framework for building asynchronous services interacting with event streams such as Apache Kafka, RabbitMQ, NATS, MQTT and Redis.

daft

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

faust-streaming

Python Stream Processing. A Faust fork

xgboost-cpu

Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C++ and more. Runs on single machine, Hadoop, Spark, Dask, Flink and DataFlow

typeid-python

Python implementation of TypeIDs: type-safe, K-sortable, and globally unique identifiers inspired by Stripe IDs

pyro5

Pyro 5 - Python remote objects

torchft-nightly

Fault tolerance for PyTorch (HSDP, LocalSGD, DiLoCo, Streaming DiLoCo)

rucio-clients

Rucio - Scientific Data Management

eventsourcing

A library for event sourcing in Python.

safe-init

Safe Init is a Python library that enhances AWS Lambda functions with advanced error handling, logging, monitoring, and resilience features, providing comprehensive observability and reliability for serverless applications.

vineyard

vineyard (v6d): an in-memory immutable data manager. (Project under CNCF, TAG-Storage)

sherlock

Easy distributed locks for Python with a choice of backends.

pegasus-wms-api

Pegasus Workflow Management System - Automate, recover, and debug scientific computations.

pegasus-wms-common

Pegasus Workflow Management System - Automate, recover, and debug scientific computations.

google-vizier

Python-based research interface for blackbox and hyperparameter optimization, based on the internal Google Vizier Service.

streaq

Fast, async, fully-typed distributed task queue via Redis streams

vineyard-bdist

vineyard (v6d): an in-memory immutable data manager. (Project under CNCF, TAG-Storage)

redis-dict

Python dictionary with Redis as backend, built for large datasets. Simplifies Redis operations for large-scale and distributed systems. Supports various data types, namespacing, pipelining, and expiration.

faust

Python Stream Processing

vineyard-ml

vineyard (v6d): an in-memory immutable data manager. (Project under CNCF, TAG-Storage)

fugue-sql-antlr-cpp

Fugue SQL Antlr C++ Parser