Scrapyd Python Packages

logparser

A tool for parsing Scrapy log files periodically and incrementally, extending the HTTP JSON API of Scrapyd.

scrapeops-scrapy

Scrapy extension that gives you all the scraping monitoring, alerting, scheduling, and data validation you will need straight out of the box.

aio-scrapy

Implement scrapy with asyncio

scrapydweb

Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. Docs 文档 :point_right:

gerapy

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

spider-admin-pro

spider-admin-pro 一个集爬虫Scrapy+Scrapyd爬虫项目查看 和 爬虫任务定时调度的可视化管理工具，SpiderAdmin的升级版

spiderkeeper

admin ui for scrapy/open source scrapinghub

spidermanager

admin ui for scrapy/open source scrapinghub

scrapyduler

Scrapyd module that schedules scrapy spiders by time.

scrapyd-heroku

Wrapper for running Scrapyd in Heroku or locally as a service

spider-info-webservice

Scrapy extension for monitoring spider.

scrapyq

Scrapyd queue management using Redis.

scrapyc

Simple client to scrapyd. Done right.

scrapyd-egg-checksum

Extension of scrapyd to get egg's md5 checksum for distributed scrapyd

gerapy-team

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Scrapyd-Client, Scrapyd-API, Django and Vue.js

scrapyd-dash

Scrapyd Dashboard

scrapyd-mongodb

Library designed to replace the SQLite backend by a MongoDB backend on Scrapy queue management

scrapyd-ui

Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. Docs 文档 :point_right:

scrapymon

Simple Web UI for Scrapy spider management via Scrapyd