Pyarrow Python Packages

narwhals

Lightweight and extensible compatibility layer between dataframe libraries!

pyarrow-stubs

Type annotations for pyarrow

ibis-framework

the portable Python dataframe library

petastorm

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

fast-xml-flattener

Fast XML flattening library with Python bindings

keyedstablehash

Stable, keyed hashing for Python objects and columnar data. Think `stablehash`, but with SipHash-like keyed PRF semantics so hashes are deterministic for a given key and resistant to adversarial inputs.

daffy

Lightweight DataFrame validation decorators for Pandas, Polars, Modin, and PyArrow. No custom types required.

timedatamodel

A lightweight data model for time series data with pandas, numpy, and polars support

fsspec-utils

Enhanced utilities and extensions for fsspec filesystems with multi-format I/O support

datagrunt

Datagrunt is a Python library designed to simplify the way you work with CSV files. It provides a streamlined approach to reading, processing, and transforming your data into various formats, making data manipulation efficient and intuitive.

db2ixf

db2ixf is a python package with a CLI that simplifies the parsing and processing of IBM Integration eXchange Format (IXF) files.

schema2pyarrow

Converts AsyncApi and JsonSchema to PyArrow schema

exness-data-preprocess

Professional forex tick data preprocessing with unified DuckDB storage, Phase7 OHLC schema, and sub-15ms query performance

pqviewer

View Apache Parquet Files In Your Terminal

sql2arrow

This is a Python library that provides convenient and high-performance methods to parse INSERT SQL statements into Arrow arrays.

pdf2dataset

Converts a whole subdirectory with a big (or small) volume of PDF documents to a dataset (pandas DataFrame) with error tracking and choice of features

dremio-arrow

Dremio Arrow Flight Client

pymetagen

Metadata Generator

swemaps

Maps of Sweden in GeoParquet

pysummaries

Generate beautiful summary tables from pandas, polars or pyarrow dataframes

xdlake

A loose implementation of the deltalake protocol, written in Python on top of pyarrow, focused on extensibility, customizability, and distributed data.

turntable-spoonbill

the portable Python dataframe library

biosets

A bioinformatics extension of 🤗 Datasets library, built for ML applications on biological and omics data, offering easy integration of metadata and low-code data management tools.

rxls

Reading both XLSX and XLSB files, fast and memory-safe, with Python, into PyArrow