envirodgi/web-monitoring-diff repository overview

⁠web-monitoring-diff

This Docker image runs an HTTP service that outputs various diffs of types of content commonly found on the web, such as HTML, text files, etc. Some diffs are traditional, programmer-friendly diffs of source text, but others are meant for visual rendering (e.g. diffing the visual content of a web page, diffing the links in the page, etc.).

This was originally developed to aid in monitoring changes to government websites, and is also used by other public good institutions to help people compare changes to web content over time.

You can also use the diff algorithms as a Python library — see web-monitoring-diff on PyPI⁠ or the source code on GitHub⁠.

⁠Documentation

Check out the detailed docs at https://web-monitoring-diff.readthedocs.io/en/stable/⁠.

⁠Using the Server

Start a container based on the Docker image:

$ docker run -p 8888:80 --rm envirodgi/web-monitoring-diff

This starts the web server on port 8888. Modify the -p option if you want to map it to a different port.

Then use cURL, a web browser, or any other HTTP tools to get a list of supported diff types:

$ curl "http://localhost:8888/"

That should output some JSON like:

{"diff_types": ["length", "identical_bytes", "side_by_side_text", "links", "links_json", "html_text_dmp", "html_source_dmp", "html_token", "html_tree", "html_perma_cc", "links_diff", "html_text_diff", "html_source_diff", "html_visual_diff", "html_tree_diff", "html_differ"], "version": "0.1.0"}

You can use each of these diff types by requesting the URL:

http://localhost:8888/<diff_type>?a=<url_to_left_side_of_comparison>&b=<url_to_right_side_of_comparison>

For example, to compare how the links on the National Renewable Energy Laboratory’s “About” page⁠ changed between 2018 and 2020 using data from the Internet Archive⁠:

# URL of a version of the page archived in 2018:
$ VERSION_2018='http://web.archive.org/web/20180918073921id_/https://www.nrel.gov/about/'
# URL of a version of the page archived in 2020:
$ VERSION_2020='http://web.archive.org/web/20201006001420id_/https://www.nrel.gov/about/'
# Use the `links_json` diff to compare the page’s links and output as JSON:
$ curl "http://localhost:8888/links_json?a=${VERSION_2018}&b=${VERSION_2020}"

If you have jq installed, you might want to use it to format the result in a nicer way:

$ curl "http://localhost:8888/links_json?a=${VERSION_2018}&b=${VERSION_2020}" | jq

You can pass additional arguments to the various diffs in the query string. See the full documentation⁠ of the server and of the various diffs for more details.

⁠Contributing

This project wouldn’t exist without a lot of amazing people’s help. It could use yours, too: please file bugs or feature requests⁠ or make a pull request⁠ to address an issue or help improve the documentation.

If you’re looking for ways to help with the project, issues with the label “good-first-issue”⁠ are usually a good place to start.

When contributing to this project, please make sure to follow EDGI's Code of Conduct⁠.

⁠License & Copyright

Copyright (C) 2017-2023 Environmental Data and Governance Initiative (EDGI)

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.0.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

See the LICENSE⁠ file for details.