Search Packages

fastwarc

A robust web archive analytics toolkit

resiliparse

A robust web archive analytics toolkit

warcio

Streaming WARC/ARC library for fast web archive IO

ipwb

InterPlanetary Wayback: A distributed and persistent archive replay system using IPFS

archivebox

🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

cdxj-indexer

CDXJ Indexing of WARC/ARCs

cdx-toolkit

A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine

forum-dl

Scrape posts, threads from forums, news aggregators, mail archives, export to JSONL, mailbox, WARC

warc2zim

Command line tool to convert a file in the WARC format to a file in the ZIM format

internet-archive-extractor

Tool for extracting archived web sites from the Internet Archive saving as WARC files.

scrapy-webarchive

A plugin for Scrapy that allows users to capture and export web archives in the WARC and WACZ formats during crawling.

scrapy-warcio

Support for writing WARC files with Scrapy

warcdb

WarcDB: Web crawl data as SQLite databases

otmt

This system evaluates a collection of mementos (archived web pages) to determine which are off topic. The collection can be part of an Archive-It collection, a single TimeMap, or stored in a WARC file.

cocrawler

CoCrawler is a versatile web crawler built using modern tools and concurrency.

cdxsummary

Summarize web archive capture index (CDX) files

archivebox-likn

The self-hosted internet archive.

metawarc

metawarc: a command-line tool for data extraction from WARC files (web archives)