The most important thing to remember is that Conical is not a replacement for your existing CI/CD system, it is intended to be used as an additional step in your process to improve your decision making when it comes to dealing with proposed changes which have an impact upon the generated numbers.
There are many ways that Conical can be used within a CI/CD pipeline process. The approach to take is usually a function of the software being tested and its requirements rather than the requirements of the tooling used. To that end, it’s difficult to describe a process in isolation from the process usually followed by a software development team.
Because one of the main aim of Conical is to reduce the effective costs of running tests (due to simpler analysis / orchestration / management etc.) then the next logical step is to be able to run more tests, more frequently. This tends to lead to a model where either:
- The existing pipelines are extended to add more tests
- Dependent pipelines (triggered by the successful completion of the code compilation) which run the additional tests.
The chosen approach is a function of what’s the expected behaviour in case of a test failure. Typically, if regression tests are being run (where the results are known in advance) then we tend to recommend the former, whereas if it’s integration tests against an existing system (i.e. candidate vs. expected) then we tend to the former. This is because where there’s no way of being able to update the expected results, then being unable to build the binaries etc. when there’s an intended change to the generated numbers is far from ideal.
In general, we tend to see a combination of both approaches taken as each test type gives a different level of confidence in the system being tested. For more information on different approaches, see the industry examples page.
Technical requirements
There aren’t any special requirements from a Conical perspective for CI. Note that data can typically be published to Conical in 2 ways:
- As it’s generated – the test app will directly output to Conical
- After the fact
If starting from scratch, we’d recommend #1; whereas #2 is useful for the case where you already have a way of running your tests and therefore the integration of Conical becomes just another results publisher without having to update the rest of your processes.