Tag: evidencesets

Categories
blog

Creating evidence sets from the command line

As part of the recently added support for evidence sets, we released a .net tool which can be used to create evidence sets from the command line without the need to write a single line of devops code.

The tool – BorsukSoftware.Conical.Tools.EvidenceSetCreator– is available on Nuget and the source is available on GitHub.

Basic Idea

The premise behind the tool is that there’s a programmatic way to identify all of the required inputs to an evidence set, even if this needs to be broken down into multiple criteria.

Following from that principle, the typical workflow is that, for a pipeline which creates the source data, each of the uploaded test run sets is tagged with a unique identifier. Internally, we use ‘ci-%buildNumber%’ but you’re obviously free to come up with something which works for you.

Once you have this identifier defined, it’s trivial to create an additional step in your CI pipeline which runs the tool to create the evidence set. For our internal usage, we have something akin to the following:

dotnet tool run BorsukSoftware.Conical.Tools.EvidenceSetCreator 
  -server https://demo.conical.cloud
  -token "thisIsntOurActualToken"
  -product "dogfood-deployment"
  -searchcriteriacount 2
  -searchcriteria 0 product "dogfood-deployment"
  -searchcriteria 0 tag ci-%build.number%
  -searchcriteria 0 tag api
  -searchcriteria 0 prefix api
  -searchcriteria 1 tag ci-%build.number%
  -searchcriteria 1 tag deployment
  -searchcriteria 1 product "dogfood-deployment"
  -searchcriteria 1 prefix deployment
  -tag ci-%build.number%
  -name "Integration tests"
  -description "Combined view"
  -link "Team City" "%tcLinkRoot%%teamcity.build.id%" "CI job"

This has the following meaning:

  1. There are 2 different sets of test run sets which contribute to our evidence set
    1. Criteria #0 searches for everything in dogfood-deployment which is tagged with both ‘ci-%build.number%’ and ‘api’. These results will have a prefix of ‘api’ (i.e. a test called ‘group1\group2\testName’ would expand to ‘api\group1\group2\testName’ in the evidence set)
    2. Criteria #1 searches for everything in dogfood-deployment which is tagged with both ‘ci-%build.number%’ and ‘deployment’. These tests are then prefixed with ‘deployment’
  2. The generated evidence set will:
    1. be tagged with ‘ci-%build.number%’
    2. have an additional link attached – the URL expands to the actual Teamcity job which generated all of the source data
Getting started

If it’s been a while since you last used .net tools, then full information from Microsoft can be found here.

The very quick ‘getting started’ steps to follow to get you ready are:

# Create the new manifest
dotnet new tool-manifest

# Install the tool
dotnet tool install BorsukSoftware.Conical.Tools.EvidenceSetCreator

Note that if you’re running the above in a CI process, and your tooling reuses the same workspace etc., then the install process won’t necessarily always ensure that the latest version of the tool is available (this caught us out ourselves a few times when we were adding a new feature to the tool!). To handle this, you can also run:

dotnet tool update BorsukSoftware.Conical.Tools.EvidenceSetCreator

As usual, if you have any problems, suggestions or queries about any of this, then please don’t hesitate to get in touch through any of the usual routes.

Happy testing.

Categories
blog

Evidence Sets Released

We’re pleased to announce that we’ve released a new version of Conical containing support for evidence sets. These allow users to be able to have a high level view of the state of their entire release candidate across multiple test run sets and products.

It’s taken a little bit more time than we had originally planned to “dot the ‘i’s and cross the ‘t’s”, but it’s definitely worth the wait. We updated the original implementation to remove the ability to mark tests as ‘pass after review’ (PAR) as the feedback we received was that having an immutable overview was rather useful in its own right. The PAR functionality will be coming soon within the general release approval functionality.

To make it easier to create evidence sets from the command line / CI pipeline, we’ve released a tool on nuget – link – to make it trivially easy to do so without needing to write any code.

We use this tool ourselves in our CI processes prior to release to create an evidence set representing all of the test material that is run against our final candidate docker image, i.e. we can see the whole results of all testing for that package in a single place so that we can be confident that what we’re releasing works.

To get started, simply download Conical and follow the installation instructions. And as always, if you have any requests / comments, please do get in touch with us and we’ll do our very best to help.

Happy testing