Search Packages

protego

A pure-Python robots.txt parser with support for modern conventions.

ultimate-sitemap-parser

Ultimate Website Sitemap Parser

advertools

advertools - online marketing productivity and analysis tools

datasette-block-robots

Datasette plugin that blocks robots and crawlers using robots.txt

fast-robotstxt

Fast and modern robots.txt parser and matcher (C++20). Fork of Google's library with zero-copy parsing (~30% faster) and RFC 9309 compliance fixes.

jwm-robotstxt

Provides python access to Googles parser for robot.txt files as used by their GoogleBot webscraper.

texting-robots-py

Python binding for Texting Robots

yirabot

YiraBot: Simplifying Web Scraping for All. A user-friendly tool for developers and enthusiasts, offering command-line ease and Python integration. Ideal for research, SEO, and data collection.

django-cs-robots

A django app to change robots.txt from the admin panel without using the database.

greenflare

Open-Source Python Based SEO Web Crawler

rbdt

rbdt is a python library (written in rust) for parsing robots.txt files for large scale batch processing.

sitemap-harvester

Crawl sitemap of a given website and export metadata of its pages recursively into CSV format.

weboptout

Opt-Out tool to check Copyright reservations in a way that even machines can understand.

pyrobotstxt

pyrobotstxt: Python Package for robots.txt Files

context-linter

LLM readiness linter for websites. Audits robots.txt, llms.txt, Schema.org, and content density on a 0-100 scale. Includes MCP server. Published on PyPI: pip install context-cli.

sitemap-grabber

Grab and recursively parse website sitemaps, robots.txt, and other related files.

astel

An asyncronous web crawling library for Python.

aeo-cli

Agentic Engine Optimization CLI — audit URLs for AI crawler readiness