Spider Python Packages

spider-client

Python, Javascript, and Rust libraries for the Spider Cloud API.

413K 25 9

bilibili-api-python

哔哩哔哩常用API调用。支持视频、番剧、用户、频道、音频等功能。原仓库地址：https://github.com/MoyuScript/bilibili-api

212K 4K 557

crawlerdetect

🕷CrawlerDetect is a Python library designed to identify bots, crawlers, and spiders by analyzing their user agents.

152K 44 11

icrawler

A multi-thread crawler framework with many builtin image crawlers provided.

73K 921 179

is-crawler

Crawler detection from User-Agent strings in 50 ns. Issues and pull requests welcome!

28K 0 0

feapder

🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量数据去重等功能。更有功能强大的爬虫管理系统feaplat为其提供方便的部署及调度

9K 4K 541

spider-rs

Spider ported to Python

7K 106 17

jvav

[NSFW] Useful tools for crawling adult learning resources.

7K 65 10

pyfreeproxy

FreeProxy: Collecting free proxies from internet. (全球海量高质量免费代理，支持爬取数十个免费代理分享源，支持自定义规则代理筛选，爬虫与数据分析必备，每日更新海量免费代理。)

7K 394 69

grab

Web Scraping Framework

6K 2K 278

scrapeops-scrapy

Scrapy extension that gives you all the scraping monitoring, alerting, scheduling, and data validation you will need straight out of the box.

6K 38 13

waifuboard

Asynchronous API for downloading images, tags, and metadata from image board sites (e.g., Danbooru, Safebooru, Yandere). Ignore the downloaded files.

5K 2 0

decryptlogin

DecryptLogin: APIs for loginning some websites by using requests.

4K 3K 748

papercrawlerutil

一套工具组，包括访问链接，获取元素，抽取文件等等也有已经实现好通过scihub获取论文的小工具，还有对于pdf转doc，文本翻译,代理连接获取以及通过api获取代理链接， PDF文件合并，PDF文件截取某些页，CSV，xls文件处理等

4K 18 6

douyin-tiktok-scraper

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。

3K 18K 3K

aio-scrapy

Implement scrapy with asyncio

3K 71 10

bilili

:beers: bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器

2K 1K 88

spkcspider

Your decentral spider for your digital identity

2K 6 0

scrapydweb

Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. Docs 文档 :point_right:

2K 3K 583