Data Preprocessing Python Packages

skrub

Machine learning with dataframes

pyhelpers

PyHelpers: An open-source toolkit for facilitating Python users' data manipulation tasks

desbordante

Desbordante is a high-performance data profiler that is capable of discovering many different patterns in data using various algorithms. It also allows to run data cleaning scenarios using these algorithms. Desbordante has a console version and an easy-to-use web application.

melpy

A NumPy-based deep learning library for building neural networks. It features an automatic differentiation engine and supports training architectures like LSTM, CNN, and FNN.

mzutils

Mohan Zhang's toolkit

split-markdown4gpt

A Python tool for splitting large Markdown files into smaller sections based on a specified token limit. This is particularly useful for processing large Markdown files with GPT models, as it allows the models to handle the data in manageable chunks.

lucifer-ml

Semi-Auto Machine Learning Library by d4rk-lucif3r

loren-frank-data-processing

Python tools for reading in data from Loren Frank's lab

nutsml

Flow-based data pre-processing for deep learning

hdsemg-select

hdsemg-select package

data-preprocessors

🛠️An easy to use tool for Data Preprocessing specially for Text Preprocessing

clearbox-preprocessor

A fast and felxible data preprocessor based on polars.

duplipy

DupliPy is a quick and easy-to-use package that can handle text formatting and data augmentation tasks for NLP in Python, with added support for image augmentation.

ptrail

PTRAIL is a state-of-the art parallel computation library for Mobility Data Preprocessing and feature extraction.

data-purifier

A Python library for Automated Exploratory Data Analysis, Automated Data Cleaning, and Automated Data Preprocessing For Machine Learning and Natural Language Processing Applications in Python.

atlantic

Atlantic: Automated Data Preprocessing Framework for Machine Learning

deocr

A high-performance highly-customizable reverse OCR tool that renders text or huggingface-compatible datasets to images. Dimension, DPI, CSS configurable!

nonechucks

Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!

phaeton

A high-performance preprocessing and ETL engine python library for sanitizing raw data streams, accelerated by Rust.

protclust

protclust is a Python library for protein sequence analysis that integrates MMseqs2 for fast clustering and provides tools for creating robust machine learning datasets. It offers cluster-aware data splitting to prevent sequence similarity bias in model evaluation, along with comprehensive protein embedding capabilities for feature generation.

dptools

Data Preprocessing Tools

sparklanes

A lightweight data processing framework for Apache Spark

vision-converter

This project consist of a library and a CLI for converting datasets between annotation formats.

loclean

High-performance, local-first semantic data cleaning library