PyPI Stats
  • Insights
  • PyPI
  • GitHub
  • Search
  • Compare
  • Advisories
  • Ecosystem
  • About
Home

Search Packages

Find Python packages by name, description, GitHub topic, or filter by metrics
andreburgaud
robotspy

Alternative robots parser module for Python

338K 22 0
nzrsky
fast-robotstxt

Fast and modern robots.txt parser and matcher (C++20). Fork of Google's library with zero-copy parsing (~30% faster) and RFC 9309 compliance fixes.

3K 5 0
jwmorley73
jwm-robotstxt

Provides python access to Googles parser for robot.txt files as used by their GoogleBot webscraper.

2K 1 0
KnuckleheadsClub
rbdt

rbdt is a python library (written in rust) for parsing robots.txt files for large scale batch processing.

372 0 0
onuratakan
site-seo-scanner

The site seo scanner.

100 1 0
chigwell
forgejo-bot-guard

This package helps administrators of Forgejo (or Git hosting platforms) generate clear, actionable anti-crawling measures by analyzing user-provided text inputs. When given a description of suspicious

77 1 0
    • Data from PyPI, GitHub, ClickHouse, and BigQuery