Data Preparation Python Packages

skrub

Machine learning with dataframes

optimuspyspark

:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

jupyterlab-amphi

visual data prep powered by python

convmerge

Merge heterogeneous chat/text sources into a single LLM training format (JSONL)

datarec-lib

Compatibility wrapper for the renamed DataRec package.

hdsemg-select

hdsemg-select package

pyoptimus

:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

amphi-scheduler

Amphi Scheduler (JupyterLab extension + Python backend)

datarec

A Python Library for Standardized and Reproducible Data Management in Recommender Systems

tracebloc-ingestor

tracebloc data pipeline for training/test dataset setup

refinedoc

Python library for post-extraction refinement of text that may be derived from PDF extraction.

label-maker

Data Preparation for Satellite Machine Learning

dptools

Data Preprocessing Tools

forestiler

Create Image Tiles From Large Input Rasters According to a Classified Mask Vector File

prosto

Prosto is a data processing toolkit radically changing how data is processed by heavily relying on functions and operations with functions - an alternative to map-reduce and join-groupby

dataclr

A Python library for feature selection in tabular datasets

machine-learning-data-pipeline

Pipeline module for parallel real-time data processing for machine learning models development and production purposes.

ml-express

A Python library for day to day data analysis and machine learning.

invisible-rabbit

Scalable data pre processing and curation toolkit for LLMs

spltr

A simple PyTorch-based data loader and splitter

daxpy

A pre-machine-learning model package

invisible-unicorn

Scalable data pre processing and curation toolkit for LLMs

lava-ray

Scalable data pre processing and curation toolkit for LLMs