Search Packages

courlan

Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters

scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

newspaper3k

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

scrapling

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

crawlee

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

scrapy-selenium

Scrapy middleware to handle javascript pages using selenium

abx-dl

⬇️ A simple all-in-one CLI tool to download EVERYTHING from a URL (like youtube-dl/yt-dlp, forum-dl, gallery-dl, simpler ArchiveBox). 🎭 Uses headless Chrome to get HTML, JS, CSS, images/video/audio/subtitles, PDFs, screenshots, article text, git repos, and more...

spidermon

Scrapy Extension for monitoring spiders execution.

kreuzcrawl

High-performance web crawling engine with bindings for 11 languages

arxiv2text

Converting PDF files to text, mainly with a focus on arXiv papers.

grab

Web Scraping Framework

memorious

Lightweight web scraping toolkit for documents and structured data.

newspaper

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

scrapyrt

HTTP API for Scrapy spiders

crawlbase

Fast python library for the Crawlbase API

aioscpy

An asyncio + aiolibs crawler imitate scrapy framework

docscrape

Scrape any documentation site to Markdown in seconds

proxycrawl

ProxyCrawl Python library for scraping and crawling

proxybroker2

The New (auto rotate) Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS :performing_arts:

maalfrid-toolkit

Toolkit for the Målfrid project

lulu

[Unmaintained] A simple and clean video/music/image downloader 👾

shopscout

Scrape any Shopify store - products, collections, pages & metadata from the public JSON API. No API key needed. SDK + CLI + REST API.

mlscraper

🤖 Scrape data from HTML websites automatically by just providing examples

spidy-web-crawler

Spidy is the simple, easy to use command line web crawler.