We’re pleased to announce a new feature that we’re working on – Evidence Sets. The premise here is that this allows a user to group a set of test run sets together to form a single viewable unit which can be used to provide both evidence (hence the name) of testing and also a way to sign off these changes.
Evidence sets can be used in multiple different ways, including:
- Allowing for failing tests to be re-run if desired
- Grouping multiple pieces of testing together to have a single reference for test results.
The main features of evidence sets are:
- Ability to collate multiple test run sets together, including:
- optional prefixes to create custom hierarchies
- subsets of test runs as desired
- from multiple different products
- Ability to have multiple test runs contribute to a single test (e.g. to handle re-runs). There are several options (best result, worst result, first result, last result or not allowed) for deciding the state of a test if multiple contributing test runs are specified
- Ability to mark tests as having passed (or indeed failed) after review, with full role and test node based security model to control who can make these changes
Internally, we use the evidence set functionality to allow us to have a coherent overview of the state of the application prior to release. For each release, we want to run:
- integration tests for the API layer
- For a fresh install
- For each DB upgrade path
- integration tests for the DB update functionality
- integration tests for the fresh install functionality
Additionally, we would like to be able to show the results of the UI testing
Example – API Integration Tests
The API integration tests are designed to check that a given instance of the API performs as expected and will cover everything from uploading results to checking the security model works as expected. Given that we want to ensure that the functionality is correct regardless of whether it’s a fresh install or an upgraded install, we want to run the same set of integration tests against as many combinations as possible. As the running of these tests is highly automated (one just needs to specify the target server and the appropriate admin user to use to start) then these are trivially easy to run and can generate a large number of result sets to analyse.
By using the evidence sets functionality, we can collate all of these result sets into a single display unit so that it’s very easy to get an overview of the state of the release candidate. We do this by using the ‘prefix’ functionality so it’s very clear where there’d be a problem, e.g.
And then the usual test hierarchy applies underneath each node.
Note that as we wouldn’t release anything which is non-green, we don’t need to leverage the sign-off functionality in evidence sets.
In addition to the functionality above, we then add the installer / upgrade test results to the same evidence set (under appropriate prefixes) so we can demonstrate to the people signing off the release that everything is good.
Test review functionality
Although our internal dogfooding usage pattern doesn’t really require ‘pass after review’ functionality (for us, the functionality either does what the test expects of it or it’s a CI/CD process stopping problem), for a lot of our clients’ use-cases, this ability to sign off on changes (especially numeric changes) is highly valuable.
To this end, when an evidence set is created, the creator can specify the set of roles who can make changes to the state of the test based off of the names of said tests. This allows for ‘sign-off’ functionality to be granted to different categories of users depending on the tests to be signed off.
We would like to add web-hooks to the functionality so that further CI/CD steps can be triggered by sign-offs. Note that although this functionality does encapsulates a large portion of the proposed in-tool release sign-off functionality, it’s not intended to replace that.
We’re putting the final touches to the functionality and we’re hoping to have this work complete in the next week or so and then we’ll make it available to all of our clients in the usual fashion.
In the meantime, if you have any questions, queries or suggestions then please do get in touch with us