Pipelines Python Packages

meltano

Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.

torchx

TorchX is a universal job launcher for PyTorch applications. TorchX is designed to have fast iteration time for training/research and support for E2E production ML pipelines when you're ready.

pipestat

Pipeline results reporting package

zenml

ZenML 🙏: One AI Platform from Pipelines to Agents. https://zenml.io.

torchx-nightly

TorchX is a universal job launcher for PyTorch applications. TorchX is designed to have fast iteration time for training/research and support for E2E production ML pipelines when you're ready.

mage-ai

🧙 Build, run, and manage data pipelines for integrating and transforming data.

polyaxon

MLOps Tools For Managing & Orchestrating The Machine Learning LifeCycle

kfp-notebook

Elyra extends JupyterLab with an AI centric approach.

vents

Open source connections catalog, integrations, tools, alerting, and notifications library.

pypyr

pypyr task-runner cli & api for automation pipelines. Automate anything by combining commands, different scripts in different languages & applications into one pipeline process.

taipy

Turns Data and AI algorithms into production-ready web applications in no time.

mabel

😊 mabel is a platform for authoring data processing systems.

metaxy

Pluggable sample-level metadata versioning for incremental multimodal pipelines.

zenml-nightly

ZenML 🙏: One AI Platform from Pipelines to Agents. https://zenml.io.

tenzir

Tenzir is the data pipeline engine for security teams.

lsst-pex-config

Configuration interface and history-tracking for LSST Data Management.

tibanna

Tibanna helps you run your genomic pipelines on Amazon cloud (AWS). It is used by the 4DN DCIC (4D Nucleome Data Coordination and Integration Center) to process data. Tibanna supports CWL/WDL (w/ docker), Snakemake (w/ conda) and custom Docker/shell command.

lsst-pipe-base

LSST Data Management: base classes for data processing tasks

data-expectations

Are your data meeting your expectations?

pytreeclass

Visualize, create, and operate on pytrees in the most intuitive way possible.

lsst-ctrl-mpexec

Execution framework for PipelineTask

ploomber

The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️

taipy-gui

Turns Data and AI algorithms into production-ready web applications in no time.

lsst-ctrl-bps

A PipelineTask execution framework for multi-node processing for the LSST Batch Production Service (BPS).