Scrapy Python Packages

itemadapter

Common interface for data container classes

2.4M 69 13

scrapy-playwright

🎭 Playwright integration for Scrapy

922K 1K 160

random-user-agent

A package to get list of user agents based on filters such as operating system, software name etc..

427K 103 12

scrapfly-sdk

Official Python SDK for the Scrapfly platform: web scraping, screenshots, AI extraction, crawling, and a remote anti-bot browser. Integrates with Scrapy, LlamaIndex, and LangChain.

313K 55 15

advertools

advertools - online marketing productivity and analysis tools

147K 1K 240

scrapy-zyte-api

Zyte API integration for Scrapy

109K 40 21

icrawler

A multi-thread crawler framework with many builtin image crawlers provided.

73K 921 179

scrapy-splash

Scrapy+Splash for JavaScript integration

70K 3K 456

scrapy-redis

Redis-based components for Scrapy.

35K 6K 2K

scrapy-impersonate

Scrapy download handler that can impersonate browser' TLS signatures or JA3 fingerprints.

34K 232 27

scrapy-crawlera

Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy

32K 365 91

scrapy-selenium

Scrapy middleware to handle javascript pages using selenium

26K 953 353

scrapy-fake-useragent

Random User-Agent middleware based on fake-useragent

24K 688 94

scrapy-zyte-smartproxy

Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy

22K 365 91

scrapy-rotating-proxies

use multiple proxies with Scrapy

13K 773 157

logparser

A tool for parsing Scrapy log files periodically and incrementally, extending the HTTP JSON API of Scrapyd.

10K 92 24

feapder

🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量数据去重等功能。更有功能强大的爬虫管理系统feaplat为其提供方便的部署及调度

9K 4K 541