Tag: dogfooding

Categories
blog

Creating evidence sets from the command line

As part of the recently added support for evidence sets, we released a .net tool which can be used to create evidence sets from the command line without the need to write a single line of devops code.

The tool – BorsukSoftware.Conical.Tools.EvidenceSetCreator– is available on Nuget and the source is available on GitHub.

Basic Idea

The premise behind the tool is that there’s a programmatic way to identify all of the required inputs to an evidence set, even if this needs to be broken down into multiple criteria.

Following from that principle, the typical workflow is that, for a pipeline which creates the source data, each of the uploaded test run sets is tagged with a unique identifier. Internally, we use ‘ci-%buildNumber%’ but you’re obviously free to come up with something which works for you.

Once you have this identifier defined, it’s trivial to create an additional step in your CI pipeline which runs the tool to create the evidence set. For our internal usage, we have something akin to the following:

dotnet tool run BorsukSoftware.Conical.Tools.EvidenceSetCreator 
  -server https://demo.conical.cloud
  -token "thisIsntOurActualToken"
  -product "dogfood-deployment"
  -searchcriteriacount 2
  -searchcriteria 0 product "dogfood-deployment"
  -searchcriteria 0 tag ci-%build.number%
  -searchcriteria 0 tag api
  -searchcriteria 0 prefix api
  -searchcriteria 1 tag ci-%build.number%
  -searchcriteria 1 tag deployment
  -searchcriteria 1 product "dogfood-deployment"
  -searchcriteria 1 prefix deployment
  -tag ci-%build.number%
  -name "Integration tests"
  -description "Combined view"
  -link "Team City" "%tcLinkRoot%%teamcity.build.id%" "CI job"

This has the following meaning:

  1. There are 2 different sets of test run sets which contribute to our evidence set
    1. Criteria #0 searches for everything in dogfood-deployment which is tagged with both ‘ci-%build.number%’ and ‘api’. These results will have a prefix of ‘api’ (i.e. a test called ‘group1\group2\testName’ would expand to ‘api\group1\group2\testName’ in the evidence set)
    2. Criteria #1 searches for everything in dogfood-deployment which is tagged with both ‘ci-%build.number%’ and ‘deployment’. These tests are then prefixed with ‘deployment’
  2. The generated evidence set will:
    1. be tagged with ‘ci-%build.number%’
    2. have an additional link attached – the URL expands to the actual Teamcity job which generated all of the source data
Getting started

If it’s been a while since you last used .net tools, then full information from Microsoft can be found here.

The very quick ‘getting started’ steps to follow to get you ready are:

# Create the new manifest
dotnet new tool-manifest

# Install the tool
dotnet tool install BorsukSoftware.Conical.Tools.EvidenceSetCreator

Note that if you’re running the above in a CI process, and your tooling reuses the same workspace etc., then the install process won’t necessarily always ensure that the latest version of the tool is available (this caught us out ourselves a few times when we were adding a new feature to the tool!). To handle this, you can also run:

dotnet tool update BorsukSoftware.Conical.Tools.EvidenceSetCreator

As usual, if you have any problems, suggestions or queries about any of this, then please don’t hesitate to get in touch through any of the usual routes.

Happy testing.