Duplicate Detection Python Packages

nomic

Nomic Developer API SDK

51K 2K 197

videohash

Near Duplicate Video Detection (Perceptual Video Hashing) - Get a 64-bit comparable hash-value for any video.

12K 371 62

fast-suffix-array

O(n) suffix array construction (SA-IS) with LCP arrays, BWT, FM-index, and pattern search. Rust-powered Python bindings.

4K 0 0

pyjedai

An open-source library that leverages Python’s data science ecosystem to build powerful end-to-end Entity Resolution workflows.

2K 93 13

nmavail

Async Python CLI tool to check name availability across domains, GitHub, NPM, PyPI, Crates.io, and system packages. Zero-config tool for startups and developers."

2K 0 0

simages

Find similar images in a dataset

848 23 3

duplicate-finder

This Python packages identifies duplicate files in a folder of interest.

741 24 5

pydupes

A duplicate file finder like rdfind/fdupes et al that may be faster in environments with millions of files and terabytes of data or over high latency filesystems (e.g. NFS).

495 4 2

deplicate

Advanced Duplicate File Finder for Python. Nothing is impossible to solve.

446 79 17

clonehunter

CloneHunter finds duplicate code across mixed-language repositories. It runs a semantic retrieval pipeline with model inference and indexing to catch harder, non-trivial duplicate patterns.

289 2 0

thebear

Bear - the decluttering deduplicator

272 4 1

demystify-digipres

Engine for analysis of Siegfried export files and DROID CSV. The tool has three purposes, break the export into its components and store them within a SQLite database; create additional columns to augment the output where useful; and query the SQLite database, outputting results in a readable form useful for analysis by researchers and archivists within digital preservation departments in memory institutions. The tool will find duplicates, unidentified files, blacklisted objects, character encoding issues, and more.

267 33 5