Web Scraping Python Packages

curl-cffi

Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.

htmldate

Fast and robust date extraction from web pages, with Python or on the command-line

primp

HTTP client that can impersonate web browsers

trafilatura

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

firecrawl-py

🔥 The API to search, scrape, and interact with the web for AI

selectolax

Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.

patchright

Undetected Python version of the Playwright testing and automation library.

seleniumbase

APIs for browser automation, testing, and bypassing bot-detection.

scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

google-search-results

Google Search Results via SERP API pip Python Package

firecrawl

🔥 The API to search, scrape, and interact with the web for AI

scrapling

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

crawlee

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

rnet

An ergonomic Python HTTP Client with TLS fingerprint

spider-client

Python, Javascript, and Rust libraries for the Spider Cloud API.

scrapfly-sdk

Official Python SDK for the Scrapfly platform: web scraping, screenshots, AI extraction, crawling, and a remote anti-bot browser. Integrates with Scrapy, LlamaIndex, and LangChain.

scrapegraph-py

Official Python SDK for the ScrapeGraph AI API. Smart scraping, search, crawling, markdownify, agentic browser automation, scheduled jobs, and structured data extraction

finvizfinance

Finviz analysis python library.

cloakbrowser

Stealth Chromium that passes every bot detection test. Drop-in Playwright replacement with source-level fingerprint patches. 30/30 tests passed.

httpmorph

httpmorph is a drop-in replacement for Python's requests library that uses a custom C implementation with BoringSSL instead of Python's standard HTTP stack.

rebrowser-playwright

A drop-in replacement for playwright-python patched with rebrowser-patches. It allows to pass modern automation detection tests.

steel-sdk

The official Python library for the Steel API

pytest-sbase

APIs for browser automation, testing, and bypassing bot-detection.

sbase

APIs for browser automation, testing, and bypassing bot-detection.