Legion Test Infrastructure

This document describes Legion's regression test infrastructure.

There are two key parts of Legion's test infrastructure:

• The test.py script contains all the logic for running tests.
• An automated CI framework is used to run this script on every commit.

This separation has the advantage the test suite itself should run anywhere and ideally produce the same results (though environmental factors may vary).

Running Tests Manually

Everything runs through the main test script:

./test.py

By default, the script will choose an appropriate set of tests and build options to run. The script will print its configuration at the start so you can see what tests are enabled. There are two sets of options that control tests and build flags respectively, which can either be set on the command line or via environment variables.

Tests:

• --test=regent or TEST_REGENT: Regent test suite
• --test=legion_cxx or TEST_LEGION_CXX: Legion C++ examples and tests
• --test=fuzzer or TEST_FUZZER: Legion fuzzer randomized tests
• --test=realm or TEST_REALM: Realm tests
• --test=external or TEST_EXTERNAL: Various external applications
• --test=private or TEST_PRIVATE: Private external applications
• --test=perf or TEST_PERF: Performance tests

Build flags:

• --debug or DEBUG: Enable debug mode (disable with DEBUG=0)
• --use=gasnet or USE_GASNET: Enable GASNet networking
• --use=cuda or USE_CUDA: Enable CUDA for NVIDIA GPUs
• --use=llvm or USE_LLVM: Enable LLVM support
• --use=hdf or USE_HDF: Enable HDF5 I/O support
• --use=spy or USE_SPY: Enable Legion Spy
• --use=cmake or USE_CMAKE: Enable the CMake build system (vs Makefiles)
• --use=rdir or USE_RDIR: Enable RDIR, a plugin for Regent

A note about command-line flags vs environment variables: flags are exclusive, while variables are additive. That is, the following two commands will do different things:

./test --test=regent
TEST_REGENT=1 ./test.py

The first will run only Regent tests. The second will run Regent tests in addition to any other tests already enabled (in this case, whatever defaults are chosen by the script). The same applies to --use and USE_ options as well.

How to Add New Tests

Automated Test Infrastructure

The Legion project currently uses two different CI services to run automated tests: Travis and Gitlab CI. Each service has a number of advantages, so at the moment it makes sense to use both.

Travis:

• Has better Github integration, which is nice for pull requests, etc.
• Supports builds on macOS
• However, Travis tends to be slower and provides a less flexible environment

Gitlab CI:

• Is more flexible:
  • We can provide our own Docker image(s) for tests
  • We can simultaneously use on-premises and cloud-based machines
• Gitlab generally tends to be faster
• Has no built-in Github integration (for obvious reasons)

An advantage of both approaches is that they are entirely configured via text files stored in the Legion repository. Thus there is no long-term state associated with a local installation of a product such as Jenkins, and very little operational knowledge required to run either service.

The configurations are stored in the following files:

• Travis: .travis.yml
• Gitlab CI: .gitlab-ci.yml
  • Docker scripts are in the docker/ directory

The two formats are similar, but different. Since the actual bulk of the test suite is stored in the test.py script, the contents of the two .yml files amount to essentially setting environment variables and kicking off the script.

Gitlab Mirroring

There is one slight wrinkle in the Gitlab CI approach: Gitlab only understands how to run CI jobs on repos hosted on its own service. Therefore, to run Gitlab CI on a Github repo, that repo must first be mirrored onto Gitlab.

Gitlab does provide a mirror feature. However, this only mirrors the contents of the remote repo about once an hour. This means that any CI jobs would also have an average latency of about half an hour, which is unnacceptable.

Instead we do the mirroring ourselves. We accomplish this with an extremely simple cronjob that runs every two minutes (so average latency is one minute). The following steps describe the process to set this up. In normal conditions, it should never be necessary to touch this, but if something goes wrong (e.g. Sapling loses a hard drive) it might be necessary to do this over.

1. Create a new user gitlab-mirror:

   sudo adduser --system gitlab-mirror sudo chsh -s /bin/bash gitlab-mirror

2. Switch to the user gitlab-mirror and initialize the local mirrors:

   git clone --mirror https://github.com/StanfordLegion/legion.git legion.git git clone --mirror https://github.com/StanfordLegion/rdir.git rdir.git

3. Set up the cronjob:

   */2 * * * * git -C /home/gitlab-mirror/rdir.git fetch -p && git -C /home/gitlab-mirror/rdir.git push git@gitlab.com:StanfordLegion/rdir.git --mirror && git -C /home/gitlab-mirror/legion.git fetch -p && git -C /home/gitlab-mirror/legion.git push git@gitlab.com:StanfordLegion/legion.git --mirror

Rebuilding the Docker Image

Docker images are also built with scripts to ensure that the results are reproducible. To update the Docker image for Gitlab, modify the file docker/Dockerfile.gitlab-ci as desired, then commit the change to the Legion repository.

Important: To avoid constantly rebuilding the Docker image, we use a separate branch to trigger the build. After changing the build script, always merge master into docker-snapshot and push the latter branch to Github. If you forget this step the automatic build will not trigger.

While the rebuild process is fully automated, it can take a while. If you get impatient, you can build and upload the image yourself. You will need access to the stanfordlegion organization on Docker Hub before you can do this. The following commands will rebuild the image:

# from the root of the Legion repository
docker build -t stanfordlegion/gitlab-ci -f docker/Dockerfile.gitlab-ci docker
docker login
docker push stanfordlegion/gitlab-ci