Pyspark Python Packages

narwhals

Lightweight and extensible compatibility layer between dataframe libraries!

chispa

PySpark test helper methods with beautiful error messages

graphframes

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

datacompy

Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!

synapseml

Simple and Distributed Machine Learning

ibis-framework

the portable Python dataframe library

hdijupyterutils

Jupyter magics and kernels for working with remote Spark clusters

autovizwidget

Jupyter magics and kernels for working with remote Spark clusters

graphframes-py

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

pyspark-hnsw

Java library for approximate nearest neighbors search using Hierarchical Navigable Small World graphs

spark-nlp

State of the Art Natural Language Processing

koheesio

Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.

quinn

pyspark methods to enhance developer productivity 📣 👯 🎉

pyspark-stubs

Apache (Py)Spark type annotations (stub files).

pyspark-test

Testing library for pyspark, inspired from pandas testing module but for pyspark, to help users write unit tests.

sparkorm

ORM for Apache Spark and DataFrames schema manager

petastorm

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

pbspark

protobuf pyspark conversion

dbldatagen

Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POCs, and other uses in Databricks environments including in Delta Live Tables pipelines

tdigest

t-Digest data structure in Python. Useful for percentiles and quantiles, including distributed enviroments like PySpark

cuallee

Possibly the fastest DataFrame-agnostic quality check library in town.

copybook

Python copybook parser

pyspark-extension

A library that provides useful extensions to Apache Spark and PySpark.

h2o-pysparkling-3-1

Sparkling Water provides H2O functionality inside Spark cluster