Sign inSign up

pereval/web-page-analyzer

By pereval

•Updated over 1 year ago

CLI tool to fetch, parse, analyze, and transform web pages with static/dynamic support.

Image
Data science
Web analytics
0

272

pereval/web-page-analyzer repository overview

⁠Web Page Analyzer

Version License Java

A command-line tool written in Groovy for retrieving, parsing, analyzing, and transforming web pages.

⁠Features

  • Fetch web pages - Download static or dynamic (JavaScript-rendered) web pages
  • Parse HTML to JSON - Convert HTML DOM to structured JSON format
  • Extract content - Extract specific elements using CSS selectors
  • Generate statistics - Analyze HTML structure, content, links, and more
  • Compare HTML files - Identify differences between web pages
  • Transform HTML - Convert to Markdown, plain text, or JSON
  • UTF-8 character support - Properly handle international character sets

⁠Quick Start

# Pull the Docker image
docker pull yourusername/web-page-analyzer:latest

# Download a webpage
docker run --rm -v "$(pwd)/output:/app/output" \
  web-page-analyzer read https://example.com /app/output/example.html

# Analyze the page
docker run --rm -v "$(pwd)/output:/app/output" \
  web-page-analyzer stats /app/output/example.html /app/output/stats.json

⁠Docker Volumes

The container uses three volume mounts:

  • /app/input - Mount input files here
  • /app/output - Output files will be written here
  • /app/logs - Log files will be written here

⁠Available Commands

⁠read

Retrieve a web page by URL and save as HTML file.

docker run --rm -v "$(pwd)/output:/app/output" \
  web-page-analyzer read https://example.com /app/output/example.html

# With dynamic JavaScript rendering
docker run --rm -v "$(pwd)/output:/app/output" \
  web-page-analyzer read https://example.com /app/output/example.html --dynamic
⁠parse

Parse HTML file and convert DOM to JSON.

docker run --rm -v "$(pwd)/input:/app/input" -v "$(pwd)/output:/app/output" \
  web-page-analyzer parse /app/input/example.html /app/output/dom.json --pretty
⁠extract

Extract specific elements from HTML using CSS selectors.

docker run --rm -v "$(pwd)/input:/app/input" -v "$(pwd)/output:/app/output" \
  web-page-analyzer extract /app/input/example.html "a[href]" /app/output/links.json

# Extract as CSV
docker run --rm -v "$(pwd)/input:/app/input" -v "$(pwd)/output:/app/output" \
  web-page-analyzer extract /app/input/example.html "table tr" /app/output/table.csv --format=csv
⁠stats

Generate statistics about an HTML document.

docker run --rm -v "$(pwd)/input:/app/input" -v "$(pwd)/output:/app/output" \
  web-page-analyzer stats /app/input/example.html /app/output/stats.json
⁠compare

Compare two HTML files and identify differences.

docker run --rm -v "$(pwd)/input:/app/input" -v "$(pwd)/output:/app/output" \
  web-page-analyzer compare /app/input/old.html /app/input/new.html /app/output/diff.json
⁠transform

Transform HTML to another format (Markdown, plain text, or JSON).

docker run --rm -v "$(pwd)/input:/app/input" -v "$(pwd)/output:/app/output" \
  web-page-analyzer transform /app/input/example.html /app/output/example.md

# Convert to plain text
docker run --rm -v "$(pwd)/input:/app/input" -v "$(pwd)/output:/app/output" \
  web-page-analyzer transform /app/input/example.html /app/output/example.txt --format=plain

⁠Configuration

⁠Environment Variables
  • JAVA_OPTS - Configure Java options (e.g., -Xmx2g for more memory)
docker run --rm -e JAVA_OPTS="-Xmx2g" -v "$(pwd)/input:/app/input" -v "$(pwd)/output:/app/output" \
  web-page-analyzer parse /app/input/large-file.html /app/output/output.json
⁠Using Docker Compose

Create a docker-compose.yml file:

version: '3.8'
services:
  web-page-analyzer:
    image: yourusername/web-page-analyzer:latest
    volumes:
      - ./input:/app/input
      - ./output:/app/output
      - ./logs:/app/logs
    command: ["stats", "/app/input/example.html", "/app/output/stats.json"]

Then run:

docker-compose up

⁠International Character Support

The Web Page Analyzer has robust support for international character sets, preserving non-ASCII characters like Cyrillic, Chinese, Japanese, Arabic, etc., in its output without converting them to Unicode escape sequences.

⁠License

This project is licensed under the MIT License⁠.

Tag summary

Content type

Image

Digest

sha256:53040a55c…

Size

262 MB

Last updated

over 1 year ago

docker pull pereval/web-page-analyzer