Deduplication Python Packages

recordlinkage

A powerful and modular toolkit for record linkage and duplicate detection in Python

4.6M 1K 153

splink

Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends

718K 2K 234

borgbackup

Deduplicating archiver with compression and authenticated encryption.

72K 13K 843

fastcdc

FastCDC implementation in Python https://pypi.org/project/fastcdc/

72K 64 18

semhash

Fast Multimodal Semantic Deduplication & Filtering

53K 919 56

nomenklatura

Framework and command-line tools for integrating FollowTheMoney data streams from multiple sources

35K 239 43

removedup

Remove duplicates from parallel corpora

6K 7 1

pyfastcdc

A high-performance FastCDC 2020 implementation written in Python + Cython

6K 2 0

goldenmatch

🟡 Golden Suite — polyglot data-quality + entity-resolution toolkit. GoldenCheck profiles → GoldenFlow standardizes → GoldenMatch dedupes → GoldenPipe orchestrates. Zero-config defaults, 97% F1, MCP server per package + one master, multi-arch container images, drop-in Airflow DAGs.

5K 36 5

zingg

Scalable identity resolution, entity resolution, data mastering and deduplication using ML

4K 1K 165

cashet

A Python memoization cache with Redis, async support, and an HTTP server.Cache Python function results like git objects. Content-addressable, pipeline-friendly, and CLI-inspectable. Run once, reuse forever.

4K 0 0

dbretina

DBRetina Python Package

4K 1 3

crossfire-rules

Regex rule overlap analyzer for DLP, secret scanning, SAST, and IDS tools

4K 0 0

mailmap-checker

Pre-commit hook that checks and maintains .mailmap completeness

3K 2 0

redis-message-queue

Reliable Python message queuing with Redis and built-in deduplication. Publish once, process once, recover from crashes - across any number of producers and consumers.

3K 5 1

replicat

Configurable and lightweight backup utility with deduplication and encryption.

3K 5 0

pyjedai

An open-source library that leverages Python’s data science ecosystem to build powerful end-to-end Entity Resolution workflows.

2K 93 13

deduplipy

Python package for deduplication/entity resolution using active learning

2K 82 8

bankstatementparser

Parse bank statements across CAMT, PAIN.001, CSV, OFX/QFX, MT940, and PDFs (digital + scanned) into unified Transaction models. Deterministic ISO 20022 parsers, LLM fallback for PDFs, vision for scans, balance verification, categorization, and interactive review mode.

2K 19 5