Dataloader Python Packages

dynabatch

PyTorch/Hugging Face batching utility that sorts variable-length text by difficulty, then dynamically increases batch size on easier samples using a pre-trained VRAM predictor to improve GPU utilization and throughput while reducing OOM risk with fallback handling.

conecta

SQL to Arrow made easy.

datago

A natively parallel dataloader for Python, written in Rust. Serving data at GB/s speeds, while covering aspect ratio bucketing, crop and resize for image ML workloads.

jax-dataloader

Pytorch-like dataloaders for JAX.

cellmap-data

Utility for loading CellMap data for machine learning training, utilizing PyTorch, TensorStore, Xarray, and PyDantic.

scdataloader

a dataloader to work with large single cell datasets from lamindb

continuum

A clean and simple data loading library for Continual Learning

lavender-data

Load & manage evolving datasets efficiently

ukv

Python bindings for Unum's UStore.

fewsoldataloader

This repo hosts the PyTorch dataloader for FewSOL dataset | ICRA 2023

torchvtk

PyTorch volume toolkit. Efficient data loading, dataset conversions, visualization tools

bruker

A modular toolkit for Bruker MRI raw-data handling.

datatap

Focus on Algorithm Design, Not on Data Wrangling

torch-dataloader-utils

Efficient Data Loader Utils for loading data from structured sources into Pytorch

torch-waymo

PyTorch dataloader for Waymo Open Dataset

vdataset

Load video datasets to PyTorch DataLoader. (Custom Video Data set for PyTorch DataLoader)

ffcv-pl

manage fast data loading with ffcv and pytorch lightning

lrp-pf-auc

Explain Neural Networks using Layer-Wise Relevance Propagation and evaluate the explanations using Pixel-Flipping and Area Under the Curve.

audiotree

Audio data loading and augmentations in JAX

hydrax

A zero-copy multiprocess dataloader for JAX.

light-dataloader

Faster DataLoader for datasets that are fully loaded into memory as tensors.

dloader

A Python implementation of the DataLoader pattern for data fetching with automatic batching.

rpcdataloader

A variant of the PyTorch Dataloader using remote workers.

peaksql

Dynamic machine learning database for genomics.