Data Processing Python Packages

pandera

A light-weight, flexible, and expressive statistical data testing library

lithops

A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀

pysparkling

A pure Python implementation of Apache Spark's RDD and DStream interfaces.

nvidia-nvimgcodec-cu12

A nvImageCodec library of GPU- and CPU- accelerated codecs featuring a unified interface

nvidia-dali-cuda120

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

datachain

Data Memory: the operational data context layer for AI agents - typed, versioned datasets over images, video, docs and tables

dolma

Data and tools for generating and inspecting OLMo pre-training data.

kubetorch

Distribute and run AI workloads on Kubernetes magically in Python, like PyTorch for ML infra.

bytewax

Python Stream Processing

runhouse

Distribute and run AI workloads on Kubernetes magically in Python, like PyTorch for ML infra.

bonobo

Extract Transform Load for Python 3.5+

untruncate-json

Python library to repair truncated json. Translated directly from the typescript original version

cratedb-toolkit

CrateDB Toolkit, an SDK for CrateDB and CrateDB Cloud.

pathway

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

itertable

⇔ IterTable is a Pythonic API for iterating through tabular data formats, including CSV, XLSX, XML, and JSON.

nvidia-nvimgcodec-cu11

A nvImageCodec library of GPU- and CPU- accelerated codecs featuring a unified interface

nvidia-nvimgcodec-cu13

A nvImageCodec library of GPU- and CPU- accelerated codecs featuring a unified interface

pipe21

Simple functional pipes for python

haupt

Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon

nvidia-dali-nightly-cuda120

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

glide

Easy ETL

spectroview

SPECTROview : A Tool for Spectroscopic Data Processing and Visualization.

deepcsv

Automatically processes data files in directories, converts array-like strings to NumPy arrays, detects and fixes data type issues, and saves results as optimized Parquet files and MORE!

cdp-backend

Data storage utilities and processing pipelines used by CDP instances.