Data Engineering Pipeline Python Packages

quickstart-vdk

One framework to develop, deploy and operate data workflows with Python and SQL.

vdk-core

One framework to develop, deploy and operate data workflows with Python and SQL.

wxdata

A Python package of end-to-end weather data clients & raw data clients with VPN/PROXY support, data processors that decode variable keys from GRIB format into a plain-language format & various tools for assisting Python automated workflows, querying meteorological datasets and filling gaps in meteorological data.

vdk-jupyterlab-extension

One framework to develop, deploy and operate data workflows with Python and SQL.

vdk-control-cli

VDK Control CLI allows user to create, delete, manage and their Data Jobs in Kubernetes runtime.

xontrib-pipeliner

Let your pipe lines flow thru the Python code in xonsh.

marshmallow-pyspark

PySpark data serializer

perustats

Tools to download and process public datasets from Peru (INEI & BCRP).

vdk-impala

One framework to develop, deploy and operate data workflows with Python and SQL.

vdk-trino

Versatile Data Kit SDK plugin provides support for trino database and trino transformation templates.

vdk-heartbeat

One framework to develop, deploy and operate data workflows with Python and SQL.

vdk-server

Versatile Data Kit SDK plugin that facilitates the installation of a local Control Service.

vdk-test-utils

One framework to develop, deploy and operate data workflows with Python and SQL.

vdk-plugin-control-cli

Versatile Data Kit SDK plugin exposing CLI commands for managing the lifecycle of a Data Jobs.

tiny-blocks

Tiny Block Operations for Data Pipelines

airflow-provider-vdk

Airflow provider for Versatile Data Kit.

vdk-ingest-http

Versatile Data Kit SDK ingestion plugin to ingest data via http requests.

vdk-lineage-model

One framework to develop, deploy and operate data workflows with Python and SQL.

vdk-kerberos-auth

One framework to develop, deploy and operate data workflows with Python and SQL.

vdk-control-api-auth

Versatile Data Kit plugin library provides support for authentication.

vdk-postgres

Versatile Data Kit SDK plugin provides support for PostgreSQL database and postgres transformation templates.

pybro-cli

yo, it's ya boy, pybro! 😎 | a personal collection of python hacks for 24.04 debian

vdk-csv

Versatile Data Kit SDK CSV plugin to ingest, export, or manipulate csv files.

vdk-oracle

Support for VDK Managed Oracle connection