Hdfs Python Packages

smart-open

Utils for streaming large files (S3, HDFS, gzip, bz2...)

tiledb

Python interface to the TileDB storage engine

skein

A tool and library for easily deploying applications on Apache YARN

dvc-hdfs

HDFS/WebHDFS plugin for dvc

megfile

Megvii FILE Library - Working with Files in Python same as the standard library

wradlib

weather radar data processing - python package

pyhdfs

Python HDFS client

snakebite

A pure python HDFS client

cluster-pack

A library on top of either pex or conda-pack to make your Python code easily available on a cluster

pyquery-polars

PyQuery is a local-first data operating system built on lazy execution that processes 100GB+ files while you doomscroll. No cap. 🧢

hadeploy

An Hadoop Application Deployment tool

webhdfspy

Python wrapper to access Hadoop HDFS REST API

streamsx-hdfs

HDFS integration for IBM Streams

alphareader

A reader for large files with custom delimiters and encodings

lakekeeper

A configurable PySpark package to identify fragmented external tables and perform safe in-place compaction

pyquery-core

PyQuery is a local-first data operating system built on lazy execution that processes 100GB+ files while you doomscroll. No cap. 🧢

pydistcp

pydistcp: python WebHDFS inter/intra-cluster data copy tool.

schemaindex

SchemaIndex is designed for data scientists to index and search metadata more efficiently.

test-cephadm

Ceph is a distributed object, block, and file storage platform

dfspy

Distributed File System written in Python

kraken-pyds

Kraken - A distributed data transfer tool.

sqoopit

A simple package to let you Sqoop into HDFS/Hive/HBase with python

srcd-smart-open

Utils for streaming large files (S3, HDFS, gzip, bz2...) - temporary source{d} fork

tanit

Kraken - A distributed data transfer tool.