egg82/domain_archiver

•Updated about 1 year ago

crawls domains and pushes URLs to ArchiveBox

Image

688

egg82/domain_archiver repository overview

⁠📚 domain_archiver

A simple crawler for auto-submitting discovered links from one or more domains to an ArchiveBox instance using its REST API (v0.8.0+).

https://github.com/egg82/archivers/tree/main/domain_archiver⁠

⁠✨ Description

domain_archiver crawls specified domains, respects robots.txt (optionally), parses sitemaps, and finds pages up to a specified depth. Each discovered URL is submitted to your ArchiveBox instance for permanent archiving.

Perfect for keeping entire sites or sets of domains regularly mirrored in your ArchiveBox.

⁠🏗️ How it works

Reads domains from $DOMAIN_LIST (;-separated)
For each domain:
- Parses robots.txt and sitemap if present
- Crawls links recursively to $DEPTH_LIMIT
- Filters links with built-in or custom regex rules
- Submits found URLs to ArchiveBox using its API

⁠🧩 Required Environment Variables

Variable	Description
ARCHIVEBOX_URL	Full URL to your ArchiveBox instance (e.g., https://archivebox.example.com⁠).
API_TOKEN	Bearer token for ArchiveBox API authentication.
DOMAIN_LIST	Semicolon-separated list of domains or URLs to crawl (e.g., example.com;anotherdomain.org).

⁠⚙️ Optional Environment Variables

Variable	Default	Description
TAG	crawler	Tag to attach to all ArchiveBox submissions.
USER_AGENT	Default UA string	Custom User-Agent for requests.
FOLLOW_ROBOTS	1	Respect robots.txt (1, true, yes = on).
REQUEST_TIMEOUT	10	Timeout for HTTP requests (seconds).
DEPTH_LIMIT	4	Maximum crawl depth.
CRAWL_DELAY	0.5	Delay between requests (seconds).
SIMULTANEOUS_DOMAINS	3	How many domains to crawl in parallel.
THREADS_PER_DOMAIN	5	Threads per domain crawler.
LOG_LEVEL	info	One of: debug, info, warn, error, critical.
URL_FILTERS_REGEX	(built-in filters)	Override default regex filters (semicolon-separated).
EXCLUDE_URLS_REGEX	skips various file extensions	Regex to override default exclude URLs.
REDIS_URL	none	Redis URL, e.g. redis://:6379/0. Only required for parallel crawling or very deep (or resumable) crawls.
REDIS_USER	none	Username for Redis authentication.
REDIS_PASS	none	Password for Redis authentication.
REDIS_NAMESPACE	crawler	Namespace/prefix for all Redis keys

⁠✅ Example docker run

docker run \
  -e ARCHIVEBOX_URL="https://archive.example.com" \
  -e API_TOKEN="YOUR_API_TOKEN" \
  -e DOMAIN_LIST="example.com;test.com" \
  -e TAG="crawler" \
  -e DEPTH_LIMIT="2" \
  egg82/domain_archiver:alpine-1.0.0

⁠🗂️ Notes

Default filters include common static file hosts (AWS S3, Azure Blob, Google Storage, etc.) to keep related files.
This tool is designed for periodic use — run it on a schedule (cron, Kubernetes CronJob, etc.) to keep your ArchiveBox up-to-date.
Python 3.9 included by default.

Tag summary

Recent tags

Content type

Image

Digest

sha256:09682a6b7…

Size

28.5 MB

Last updated

about 1 year ago

docker pull egg82/domain_archiver:alpine-1.3.0