Deduplication Python Packages

recordlinkage

A powerful and modular toolkit for record linkage and duplicate detection in Python

4.6M 1K 153

splink

Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends

740K 2K 234

borgbackup

Deduplicating archiver with compression and authenticated encryption.

76K 13K 843

fastcdc

FastCDC implementation in Python https://pypi.org/project/fastcdc/

72K 64 18

semhash

Fast Multimodal Semantic Deduplication & Filtering

52K 919 56

nomenklatura

Framework and command-line tools for integrating FollowTheMoney data streams from multiple sources

38K 239 43

removedup

Remove duplicates from parallel corpora

6K 7 1

pyfastcdc

A high-performance FastCDC 2020 implementation written in Python + Cython

6K 2 0

goldenmatch

🟡 Golden Suite — polyglot data-quality + entity-resolution toolkit. GoldenCheck profiles → GoldenFlow standardizes → GoldenMatch dedupes → GoldenPipe orchestrates. Zero-config defaults, 97% F1, MCP server per package + one master, multi-arch container images, drop-in Airflow DAGs.

5K 36 5

zingg

Scalable identity resolution, entity resolution, data mastering and deduplication using ML

5K 1K 165

cashet

A Python memoization cache with Redis, async support, and an HTTP server.Cache Python function results like git objects. Content-addressable, pipeline-friendly, and CLI-inspectable. Run once, reuse forever.

5K 0 0

crossfire-rules

Regex rule overlap analyzer for DLP, secret scanning, SAST, and IDS tools

4K 0 0

dbretina

DBRetina Python Package

4K 1 3

mailmap-checker

Pre-commit hook that checks and maintains .mailmap completeness

3K 2 0

redis-message-queue

Reliable Python message queuing with Redis and built-in deduplication. Publish once, process once, recover from crashes - across any number of producers and consumers.

3K 5 1

replicat

Configurable and lightweight backup utility with deduplication and encryption.

3K 5 0

pyjedai

An open-source library that leverages Python’s data science ecosystem to build powerful end-to-end Entity Resolution workflows.

2K 93 13

bankstatementparser

Parse bank statements across CAMT, PAIN.001, CSV, OFX/QFX, MT940, and PDFs (digital + scanned) into unified Transaction models. Deterministic ISO 20022 parsers, LLM fallback for PDFs, vision for scans, balance verification, categorization, and interactive review mode.

2K 19 5