egg82/gov_archiver

•Updated about 1 year ago

crawls .gov domains and pushes URLs to ArchiveBox

Image

638

egg82/gov_archiver repository overview

⁠📚 gov_archiver

A crawler for discovering .gov domains from the official CISA dotgov-data⁠ source and auto-submitting discovered links to an ArchiveBox instance using its REST API (requires ArchiveBox v0.8.0+).

https://github.com/egg82/archivers/tree/main/gov_archiver⁠

⁠✨ Description

gov_archiver fetches the latest list of .gov domains, filters out unwanted domain types (like City, County, etc.), crawls the remaining domains, respects robots.txt (optionally), parses sitemaps, and explores pages up to a specified depth. Every discovered URL is submitted to your ArchiveBox for permanent archiving.

Perfect for preserving the public record by keeping snapshots of U.S. government websites automatically.

⁠🏗️ How it works

Downloads the current-full.csv⁠ domain list.
Filters domains by Domain type using $DOMAIN_TYPE_NEGATIVE_FILTER_REGEX.
For each valid domain:
- Parses robots.txt and sitemaps if present.
- Crawls links recursively to $DEPTH_LIMIT.
- Filters links with built-in or custom regex rules.
- Submits found URLs to ArchiveBox using its REST API.

⁠🧩 Required Environment Variables

Variable	Description
ARCHIVEBOX_URL	Full URL to your ArchiveBox instance (e.g., https://archive.example.com⁠).
API_TOKEN	Bearer token for ArchiveBox API authentication.

⁠⚙️ Optional Environment Variables

Variable	Default	Description
TAG	gov	Tag to attach to all ArchiveBox submissions.
DOMAIN_TYPE_NEGATIVE_FILTER_REGEX	`^(City\|County\|State\|Interstate\|School\|Special\|Tribal)`	Regex for excluding certain domain types.
USER_AGENT	ArchiveBox Default UA string	Custom User-Agent for requests.
FOLLOW_ROBOTS	1	Respect `robots.txt` (1, true, yes = on).
REQUEST_TIMEOUT	10	Timeout for HTTP requests (seconds).
DEPTH_LIMIT	4	Maximum crawl depth.
CRAWL_DELAY	0.5	Delay between requests (seconds).
SIMULTANEOUS_DOMAINS	3	How many domains to crawl in parallel.
THREADS_PER_DOMAIN	5	Threads per domain crawler.
LOG_LEVEL	info	One of: debug, info, warn, error, critical.
URL_FILTERS_REGEX	(built-in filters)	Override default regex filters (semicolon-separated).
EXCLUDE_URLS_REGEX	skips various file extensions	Regex to override default exclude URLs.
REDIS_URL	none	Redis URL, e.g. redis://:6379/0. Only required for parallel crawling or very deep (or resumable) crawls.
REDIS_USER	none	Username for Redis authentication.
REDIS_PASS	none	Password for Redis authentication.
REDIS_NAMESPACE	crawler	Namespace/prefix for all Redis keys

⁠✅ Example docker run

docker run \
  -e ARCHIVEBOX_URL="https://archive.example.com" \
  -e API_TOKEN="YOUR_API_TOKEN" \
  -e TAG="gov" \
  -e DEPTH_LIMIT="2" \
  egg82/gov_archiver:alpine-1.0.0

⁠🗂️ Notes

Uses the official cisagov/dotgov-data CSV list to stay up to date.
Built-in domain type filter skips local/state/tribal domains by default — adjust with $DOMAIN_TYPE_NEGATIVE_FILTER_REGEX if needed.
Default filters include common static file hosts (AWS S3, Azure Blob, Google Storage, etc.) to keep related files.
Run on a schedule (cron, Kubernetes CronJob, etc.) to keep your ArchiveBox current.
Python 3.9 included by default.

Tag summary

Recent tags

Content type

Image

Digest

sha256:ba5e04d11…

Size

28.5 MB

Last updated

about 1 year ago

docker pull egg82/gov_archiver:alpine-1.3.0