Webcrawler Python Packages

decrypt-cookies

Get browser cookies and logins. Easily make a request using the authorization data from your browser.

markcrawl

Fast Python web crawler for RAG and AI ingestion. Extracts clean Markdown from any site for LLMs and vector stores.

catch

Web crawler with built in parsers using latest Python technologies

scrapyrt

HTTP API for Scrapy spiders

gne

新闻网页正文通用抽取器 Beta 版.

cloudfeaster

Cloudfeaster Spider Development

web-scraper-toolkit

A powerful, standalone web scraping toolkit using Playwright and various parsers.

googlenewsscraper

A Python package that scrapes Google News article data while remaining undetected by Google. Our scraper can scrape page data up until the last page and never trigger a CAPTCHA (download stats: https://pepy.tech/project/GoogleNewsScraper)

hydrosensorreader

This project provides tools to read files from probes, sensors, or anything used in hydrogeology.

spydy

light-weight high-level web-crawling framework

sslproxies24

Captura e validação de Proxys (Python).

noizze-crawler

A web page crawler which returns (title, og:image, og:description).

trawler

A data gathering framework to search and get information from web sources

scrapio

Asyncio web crawling framework. Work in progress.

scrapegraph-mcp

ScapeGraph MCP Server

generalnewsextractor

新闻网页正文通用抽取器 Beta 版.

webscraping-tools

Tools to make webscraping easier

korean-news-crawler

Python Library for Crawling News Artircles in Korean Top 10 News Websites with Utilities

mseep-scrapegraph-mcp

ScapeGraph MCP Server

yugioh-scraper

Yu-Gi-Oh! Scraper is a project that crawls websites and APIs and extracts Yu-Gi-Oh! related data from their pages.

socialscraper

Social Scraper is a python tool meant for Detection of Child Predators/Cyber Harassers on Social Media

webpage-image-downloader

Tool for extracting and saving specific images from websites.

django-easy-scraper

Django apps to scrape data from web page easily

smith-the-crawler

A toolkit to make easy web scraping the world.