kareemsasa3/arachne repository overview

Arachne: A Resilient Go Web Scraper Service

Arachne is a production-ready, concurrent web scraper service built in Go. It is designed for resilience and scalability, featuring a REST API, a Redis-backed job queue, and a circuit breaker for fault tolerance.

✨ Features

REST API: Fully asynchronous API to submit and monitor scraping jobs.

Persistent Job Queue: Redis backend ensures job state and results are preserved across restarts.

Fault-Tolerant: Built-in circuit breaker and automatic retries handle network errors and failing services gracefully.

High Performance: Uses Go's concurrency for fast, rate-limited scraping.

Easy to Deploy: Fully containerized and ready to launch with Docker Compose.

🚀 How to Run

Create a docker-compose.yml file with the following content:

version: '3.8'

services:
  scraper:
    image: kareemsasa3/arachne:v1.0.0
    restart: always
    ports:
      - "8080:8080" # Or your desired port
    environment:
      - REDIS_ADDR=redis:6379
    depends_on:
      - redis

  redis:
    image: "redis:alpine"
    restart: always

  redis-commander:
    image: rediscommander/redis-commander:latest
    restart: always
    ports:
      - "8081:8081"
    environment:
      - REDIS_HOSTS=local:redis:6379

Run the service from your terminal:

docker-compose up -d

The Arachne API will be available at http://localhost:8080⁠.