Daedalic Test Automation Plugin

The Daedalic Test Automation Plugin facilitates creating and running integration tests with the Gauntlet Automation Framework of Unreal Engine 4.

Automated testing, when applied instead of or in addition to manual testing, provides multiple benefits:

• Clean code. Automating tests requires the tested code to be very modular, which improves maintainability in general.
• Fast results. Running automated tests is usually a lot faster than testing manually, and can be performed unattended.
• Better coverage. As a result, we can test more parts of our game when using automated testing than without, allowing manual testers to focus on other parts of the game.
• Increased stability. Being able to run automated tests greatly increases the confidence of the development team when applying bigger changes to the code base, allowing them to catch regressions early (e.g. after optimizing performance).

After using the plugin for automating tests of The Lord of the Rings™: Gollum™, we decided to share it with the rest of the world. We feel like software testing is far too important not to be supported by automation, and test automation still hasn't fully found its way into game development. We believe that this is party because creating automated tests for games tends to be tedious, and we want to improve on that.

Contents

1. Setup
   1. Prerequisites
   2. Adding The Plugin
2. Creating Tests
   1. Assertions
   2. Delays
   3. Simulating Input
   4. Test Trigger Boxes
   5. Test Timeouts
   6. Test Suite Lifecycle
   7. Parameterized Tests
   8. Skipping Tests
   9. Assumptions
   10. Performance Tests
3. Running Tests
   1. Play In Editor
   2. Automation Window
   3. Gauntlet
4. Configuring Tests
   1. Console Variables
5. Reporting Results
   1. JUnit Test Reports
   2. Custom Test Reports
6. Best Practices
7. Bugs, Questions & Feature Requests
8. Development Cycle
9. Contributing
10. Future Work
11. License
12. References

Setup

Prerequisites

Daedalic Test Automation Plugin currently supports the following Unreal Engine Versions:

• 4.23

Note that you currently have to compile the plugin yourself, and thus need a C++ Unreal project.

If you want to use Gauntlet for running your tests, you'll need a source version of Unreal Engine as well.

Adding The Plugin

1. Clone the repository or download a release.
2. Close the Unreal Editor.
3. Copy the DaedalicTestAutomationPlugin folder to the Plugins folder next to your .uproject file.
4. Copy the DaedalicTestAutomationPlugin.Automation folder to the Build/Scripts folder next to your .uproject file.
5. Start the Unreal Editor.
6. Enable the plugin in Edit > Plugins > Daedalic Entertainment.
7. Close the Unreal Editor.
8. Right-click your .uproject file and select Generate Visual Studio project files.
9. Build the resulting solution in Visual Studio.
10. Start the Unreal Editor.

Creating Tests

Daedalic Test Automation Plugin is fully exposed to blueprints in order to allow everyone to easily create tests. Each level represents a test suite, which in turn can consist of multiple tests.

You'll be using Gauntlet to run one or more test suites, or the Unreal Editor to run a single test suite.

In order to create a new test suite with a single test:

1. Create a new level.
2. Add a Dae Test Suite Actor to the level.
3. Create a Dae Test Actor blueprint (e.g. through right-click in Content Browser > Create Advanced Asset > Test Automation > Test Actor Blueprint).
4. Implement the Arrange, Act and Assert events of the test actor (see below).
5. Add an instance of the test actor blueprint to the level.
6. Add the test actor reference to the list of tests of the test suite actor.

Automated tests in the Daedalic Test Automation Plugin are built with the Arrange-Act-Assert pattern in mind:

• In Arrange, you should set up your test environment, get references to required actors and components, and prepare everything for the actual test.
• In Act, you should perform the actual action to test. Here, you're allowed to use latent actions, such as delays, to test what you want to test. Because we don't know when you're finished, you have to call Finish Act when you're done.
• In Assert, you should use the built-in assertion framework to verify the results of your tests, e.g. check the state of variables or positions of actors.

If any of the assertions performed in the Assert step fail, the test will be marked as failed.

You can verify your test suite by entering PIE and filtering your log by the LogDaeTest log category.

LogDaeTest: Display: ADaeTestSuiteActor::RunAllTests - Test Suite: DaeTestSuiteActor_1
LogDaeTest: Display: ADaeTestSuiteActor::RunNextTest - Test: BP_TestCalculatorAddsNumbers_2
LogDaeTest: Display: ADaeTestSuiteActor::OnTestSuccessful - Test: BP_TestCalculatorAddsNumbers_2
LogDaeTest: Display: ADaeTestSuiteActor::RunNextTest - All tests finished.

You'll also find a handful of example tests in the Content folder of the plugin.

Assertions

There's a whole lot of assertion nodes for use in your automated tests, including equality and range checks for all basic blueprint types, or verifying the state of UMG widgets. Take a look at Documentation/Assertions.md for more details.

Delays

Daedalic Test Automation Plugin comes with additional delay nodes that you might find useful when building your tests. Take a look at Documentation/Delays.md for more details.

Simulating Input

We provide blueprint nodes for simulating player input, both actions and axes. Actions will be applied once immediately, while axes will be applied until explicitly reset by applying it again. You can use all actions and axes defined in your input mappings (Edit > Project Settings > Engine > Input).

Simulated input is especially helpful when combined with Delays:

Test Trigger Boxes

Daedalic Test Automation Plugin ships with a convenience Dae Test Trigger Box that allows you to set up test delays and assertions more quickly. These trigger boxes will just set a flag when triggered, and write a log. Our built-in delays and assertions will use that flag to check if the box has been triggered.

Test Timeouts

At your Dae Gauntlet Test Actor blueprint (or instance), you can specify a timeout for the test (defaults to 30 seconds).

If your test times out during the Act stage, we'll execute all assertions immediately instead of waiting for the Act stage to finish. This allows your test to finish with just warnings instead of errors, in case you just set up some wrong delays, for instance. However, if your assertions actually fail after the timeout, the test will be marked as failed as usual.

LogDaeTest: Display: ADaeTestSuiteActor::RunAllTests - Test Suite: DaeTestSuiteActor_1
LogDaeTest: Display: ADaeTestSuiteActor::RunNextTest - Test: BP_TestMoveForward_2
LogDaeTest: Warning: Timed out after 5.000000 seconds
LogDaeTest: Error: Assertion failed - Trigger box DaeTestTriggerBox_1 wasn't triggered
LogDaeTest: Error: ADaeTestSuiteActor::OnTestFailed - Test: BP_TestMoveForward_2, FailureMessage: Assertion failed - Trigger box DaeTestTriggerBox_1 wasn't triggered
LogDaeTest: Display: ADaeTestSuiteActor::RunNextTest - All tests finished.

Test Suite Lifecycle

Instead of just adding a default Dae Test Suite Actor to your level, you can create a test suite blueprint instead (e.g. through right-click in Content Browser > Create Advanced Asset > Test Automation > Test Suite Actor Blueprint).

Test suite blueprints allow you to implement the following lifecycle events:

• BeforeAll: Executed before running the first test.
• BeforeEach: Executed every time before running a test.
• AfterEach: Executed every time after running a test.
• AfterAll: Executed after running the last test.

After creating your test suite blueprint, you can add instances of that blueprint to your test levels just as you would with the default test suite actor. Then, add test actor references to the list of tests of your test suite as usual.

Parameterized Tests

In case you want to run the same test multiple times with just slightly different configurations, Daedalic Test Automation Plugin offers parameterized tests. You can specify any number of parameters for your test instance (or blueprint).

In order to provide a consistent test API, these parameters have to be of type UObject, so if you have any other type you want to pass in as parameter, you'll need to wrap them with an UObject. Using UObject parameters also enables you to reference other actors in your test level. We're using soft references to the parameter objects, enabling you to reference actors from other streaming levels as well.

The parameter references will be resolved and passed to all test events, where you can perform your test actions on them:

The test will be run once for each parameter, and each test run will be treated exactly as if you'd run a non-parameterized test:

• the test time is reset for each parameter
• BeforeEach and AfterEach are called for each parameter
• test reports include one test per parameter

LogDaeTest: Display: ADaeTestSuiteActor::RunAllTests - Test Suite: DaeTestSuiteActor_1
LogDaeTest: Display: ADaeTestSuiteActor::RunNextTest - Test: BP_TestParameterized_TwoParameters - BP_TestParameter1
LogBlueprintUserMessages: [BP_TestParameterized_TwoParameters] BP_TestParameter1
LogDaeTest: Display: ADaeTestSuiteActor::OnTestSuccessful - Test: BP_TestParameterized_TwoParameters - BP_TestParameter1
LogDaeTest: Display: ADaeTestSuiteActor::RunNextTest - Test: BP_TestParameterized_TwoParameters - BP_TestParameter2
LogBlueprintUserMessages: [BP_TestParameterized_TwoParameters] BP_TestParameter2
LogDaeTest: Display: ADaeTestSuiteActor::OnTestSuccessful - Test: BP_TestParameterized_TwoParameters - BP_TestParameter2
LogDaeTest: Display: ADaeTestSuiteActor::RunNextTest - All tests finished.

Sometimes, you'll want to provide the test parameters dynamically, e.g. when you need to convert them to UObjects. Daedalic Test Automation Plugin features Dae Test Parameter Provider Actors for this: You can create parameter provider blueprints (e.g. through right-click in Content Browser > Create Advanced Asset > Test Automation > Test Parameter Provider Actor Blueprint). Then, you can override the GetParameters function to provide parameters for your test.

Finally, you'll have to add your provider to your test level, and to your test:

For each parameterized test, all parameter providers are applied exactly once, before the first run of that test.

Skipping Tests

If you want to temporarily disable a test, you may specify a Skip Reason at your Dae Test Actor blueprint (or instance). Setting the Skip Reason to a non-empty string will cause the test to be skipped with the specified message. We don't provide a way of skipping a test without specifying a reason, because we feel that people should always know why a test is currently disabled.

Skipped tests will be marked as neither successful nor failed, and will show up explicitly as skipped in test reports.

LogDaeTest: Display: ADaeTestSuiteActor::RunAllTests - Test Suite: DaeTestSuiteActor_1
LogDaeTest: Display: ADaeTestSuiteActor::RunNextTest - Test: BP_TestClimbLedge
LogDaeTest: Display: ADaeTestSuiteActor::OnTestSkipped - Test: BP_TestClimbLedge, SkipReason: Does not seem to succeed reliably. Occasionally, we fail to get a hold of the ledge after jumping if done automatically by the test.
LogDaeTest: Display: ADaeTestSuiteActor::RunNextTest - All tests finished.

Assumptions

You can also specify dynamic conditions for skipping a test through assumptions. This extends the above test model from Arrange-Act-Assert to Assume-Arrange-Act-Assert: You can make an arbitrary number of assumptions before even arranging up your test (e.g. check the value of some configuration variable or command-line parameter, or check the current runtime platform).

If any of your assumptions fail, the test will be skipped instead of bei