Hadoop Python Packages

luigi

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

yarn-api-client

Python client for Hadoop® YARN API

h2o

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

skein

A tool and library for easily deploying applications on Apache YARN

dvc-hdfs

HDFS/WebHDFS plugin for dvc

hadoop-fs-wrapper

Python Wrappers for Hadoop FileSystem

pyhdfs

Python HDFS client

spooq

Spooq is a PySpark based helper library for ETL data ingestion pipeline in Data Lakes.

h2o-client

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

hadoop-yarn-rest-api

This is Python Library for YARN REST api

h2o-mlflow-flavor

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

madoop

A light weight MapReduce framework for education.

yarn-dev-tools

Various scripts to automate and ease Apache Hadoop YARN development.

hadeploy

An Hadoop Application Deployment tool

splitlog

Utility to split aggregated logs from Apache Hadoop Yarn applications into a folder hierarchy

streamsx-hdfs

HDFS integration for IBM Streams

cobra-policytool

Manage Apache Atlas and Ranger configuration for your Hadoop environment.

tf-yarn

Train TensorFlow models on YARN in just a few lines of code!

lakekeeper

A configurable PySpark package to identify fragmented external tables and perform safe in-place compaction

clusterdock

clusterdock is a framework for creating Docker-based container clusters

knit

Deprecated, please use https://github.com/jcrist/skein or https://github.com/dask/dask-yarn instead

jumpy

Numpy and nd4j interop

hadopy

Easy parallel map-reduce command line tool

python3-lzo-indexer

Python library for indexing block offsets within LZO compressed files