Make continuous deployment safe by comparing before and after webpage screenshots for each release. Depicted shows when any visual, perceptual differences are found. This is the ultimate, automated end-to-end test.
Depicted is:
Depicted is not finished! Please let us know if you have feedback or find bugs.
See this video for a presentation about how perceptual diffs have made continuous deployment safe.
Here are the steps to making Depicted useful to you:
Depicted organizes your releases by a build ID. You can create a build through the API server's UI. A build is usually synonymous with a binary that's pushed to production. But it could also be a unique view into one product, like one build for desktop web and another for mobile web.
Within a build are releases with names. I recommend naming a release as the day it was branched in your source repository, and maybe an attempt number, like "06-16-r01" for June 16th, release 1. If you use codenames for your releases, like "bumblebee", that works too.
Each release may be attempted many times. The full history of each release attempt is saved in the UI. Releases can be manually marked as good or bad in the UI. When results for a new release are uploaded, they will automatically be compared to the last known-good version within that build.
A release consists of many separate test runs. A test run is a single screenshot of a single page. A test run has a name that is used to pair it up with a baseline test run from the known-good, previous release. Usually the test run is named as the path of the URL being tested (like /foo?bar=meep). This lets the baseline release and new release serve on different hostnames.
The life-cycle of a release:
Final release states: - Bad: The build admin has marked the release and all its test runs as bad. It will never be used as a baseline. - Good: The build admin has marked the release and all of its test runs as passing. The next release created for this build will use this just-approved release as the new baseline.
Depicted is written in portable Python. It uses Flask and SQLAlchemy to make it easy to run in your environment. It works with SQLite out of the box. The API server runs on App Engine. The workers run ImageMagick and PhantomJS as subprocesses. I like to run the worker on a cloud VM, but you could run it on your laptop behind a firewall if that's important to you. See deployment below for details.
Clone this git repo in your terminal:
git clone https://github.com/bslatkin/dpxdt.git
cd to the repo directory:
Update all git submodules in the repo:
git submodule update --init --recursive
Modify common.sh to match your enviornment:
# Edit variables such as ...
export PHANTOMJS_BINARY=/Users/yourname/Downloads/phantomjs-1.9.0-macosx/bin/phantomjs
Write a secrets.py file to the root directory:
SECRET_KEY = 'insert random string of characters here'
Execute ./run_shell.sh and run these commands to initialize your DB:
server.db.drop_all()
server.db.create_all()
Run the combined server/worker with ./run_combined.sh.
Execute the ./run_url_pair_diff.sh tool to verify everything is working:
./run_url_pair_diff.sh \
--upload_build_id=1 \
http://www.google.com \
http://www.yahoo.com
Follow the URL the tool writes to the terminal and verify screenshots are present. Any errors will be printed to the log in the terminal where you are running the server process.
To run the "site diff" script, use:
./run_site_diff.sh \
--upload_build_id=1 \
--crawl_depth=1 \
http://www.example.com
To run the tests to make sure you haven't broken the world:
./run_tests.sh
To run the API server locally, without any worker threads:
./run_server.sh
To run the background workers independently against the local API server:
./run_worker.sh
To run in the App Engine development environment (see the section on deployment for config details):
./appengine_run.sh
You can try out the API on the test instance of Depicted located at https://dpxdt-test.appspot.com. This instance's database will be dropped from time to time, so please don't rely on it.
The API is really simple. All requests are POSTs with parameters that are URL encoded. All responses are JSON. All requests should be over HTTPS. The API server uses HTTP Basic Authentication to verify your client has access to your builds. You can provision API keys for a build on its homepage (at the bottom).
Here's an example request to the API server using curl. Pretty easy.
curl -v \
-u api_key:api_password \
-F build_id=1 \
-F 'run_name=/static/dummy/dummy_page1.html' \
-F 'release_number=1' \
-F 'log=906d3259c103f6fcba4e8164a4dc3ae0d1a685d9' \
-F 'release_name=2013-06-16 17:35:03.327710' \
'http://localhost:5000/api/report_run'
An example client tool that exercises the whole workflow is available in the repo. It's called "Site Diff". It will crawl a webpage, follow all links with the same prefix path, then create a new release that screenshots all the URLs. Running the tool multiple times lets you diff your entire site with little effort. Site Diff is very helpful, for example, when you have a blog with a lot of content and want to make a change to your base template and be sure you haven't broken any pages.
Here's an example invocation of Site Diff:
./dpxdt/tools/site_diff.py \
--upload_build_id=1234 \
--release_server_prefix=https://my-dpxdt-apiserver.example.com/api \
--release_client_id=<your api key> \
--release_client_secret=<your api secret> \
--crawl_depth=1 \
http://www.example.com/my/website/here
Another example tool is available in the repo called Pair Diff. Unlike Site Diff, which establishes a baseline on each subsequent run, Pair Diff takes two live URLs and compares them. This is useful when you have a live version and staging version of your site both available at the same time and can do screenshots of both independently.
Here's an example run of Pair Diff:
./dpxdt/tools/url_pair_diff.py \
--upload_build_id=1234 \
--release_server_prefix=https://my-dpxdt-apiserver.example.com/api \
--release_client_id=<your api key> \
--release_client_secret=<your api secret> \
http://www.example.com/my/before/page \
http://www.example.com/my/after/page
All of these requests are POSTs with URL-encoded or multipart/form-data bodies and require HTTP Basic Authentication using your API key as the username and secret as the password. All responses are JSON. The 'success' key will be present in all responses and true if the request was successful. If 'success' isn't present, a human-readable error message may be present in the response under the key 'error'.
Creates a new release candidate for a build.
Finds the last good run of the given name for a release. Returns an error if no run previous good release exists.
Requests a new run for a release candidate. Causes the API system to take screenshots and do pdiffs. When ref_url and ref_config are supplied, the system will run two sets of captures (one for the baseline, one for the new release) and then compare them. When rel_url and ref_config are not specified, the last good run for this build is found and used for comparison.
configThe config passed to the request_run function may have any or all of these fields. All fields are optional and have reasonably sane defaults.
{
"viewportSize": {
"width": 1024,
"height": 768
},
"injectCss": ".my-css-rules-here { display: none; }",
"injectJs": "document.getElementById('foobar').innerText = 'foo';"
}
Uploads an artifact referenced by a run.
Reports data for a run for a release candidate. May be called multiple times as progress is made for a run. No longer callable once the screenshot image for the run has been assigned.
$ claude mcp add dpxdt \
-- python -m otcore.mcp_server <graph>