Preprocessing Python Packages

unstructured

Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.

jaconv

Pure-Python Japanese character interconverter for Hiragana, Katakana, Hankaku, and Zenkaku

nnaudio

Audio processing by using pytorch 1D convolution network

semhash

Fast Multimodal Semantic Deduplication & Filtering

neologdn

Japanese text normalizer for mecab-neologd

pyprep

PyPREP: A Python implementation of the Preprocessing Pipeline (PREP) for EEG data

autots

Automated Time Series Forecasting

indic-num2words

Python library for converting numbers to words for all Indian Languages.

contextualspellcheck

✔️Contextual word checker for better suggestions (not actively maintained)

df2onehot

Convert a unstructured array into a stuctured dataframe.

nums-from-string

Extract numbers from a string

pyhealth

A Deep Learning Python Toolkit for Healthcare Applications.

autoreject

Automated rejection and repair of bad trials/sensors in M/EEG

kditransform

Kernel density integral transformation: feature preprocessing and univariate clustering (TMLR, 2023)

nvtabular

NVTabular is a feature engineering and preprocessing library for tabular data designed to quickly and easily manipulate terabyte scale datasets used to train deep learning based recommender systems.

eegprep

EEGPrep is an automated preprocessing tool for human EEG data built on a benchmarked EEGLAB pipeline

smashed

SMASHED is a toolkit designed to apply transformations to samples in datasets, such as fields extraction, tokenization, prompting, batching, and more. Supports datasets from Huggingface, torchdata iterables, or simple lists of dictionaries.

logprep

log data pre processing, generation and shipping in python

jionlp

中文 NLP 预处理、解析工具包，准确、高效、易用 A Chinese NLP Preprocessing & Parsing Package www.jionlp.com

maldiamrkit

Comprehensive toolkit for MALDI-TOF mass spectrometry data preprocessing for antimicrobial resistance (AMR) prediction purposes

cmip6-preprocessing

Analysis ready CMIP6 data in python the easy way with pangeo tools.

sktutor

sktutor helps your machines learn

preocr

Fast document classification and OCR detection. Analyzes any file type to determine if OCR is needed, saving time and money on unnecessary processing.

seqtools

A python library to manipulate and transform indexable data (lists, arrays, ...)