Monitoring Conical with Prometheus and Grafana

One of our clients recently extended their use of Conical to include UX testing. This proved very useful to them, but also entailed the uploading of ~300MB of data per test cycle. As they’d originally only set themselves up with 16GB of disc space for data, this proved rather problematic as their test jobs started failing because they were unable to upload data as they’d filled their available space.

On talking to the client, we asked what monitoring tools that they had in place and the answer was that they didn’t and they wanted us to recommend a solution. Given that we support reporting server metrics in the Prometheus format (via /api/metrics) the natural answer was ‘use Prometheus and Grafana’.

This allowed the client to be aware of when they needed to proactively clean up their old, no longer relevant, test results rather than having to respond reactively when everything had stopped working.

It then struck us that there was no example of how to configure Prometheus for use with Conical on the web, hence this blog post.

Background

Prometheus and Grafana work together to provide a way of monitoring metrics from a range of services. Prometheus is responsible for gathering the data from a range of sources whilst Grafana can be used to display dashboards of said data.

These tools are incredibly powerful and have a vast range of additional functionality over and above our simple use-case. However, for the purposes of what we’re trying to do, the above is valid.

Note that we run all of our services using Docker containers.

To that end, we needed to do the following:

Conical:

• Create a PAT with ‘servermetrics’ permissions

Prometheus:

• Create a configuration file (see below) containing information about where we want to source data from
• [Optional] Create a ‘webconfig.yml’ file to enable username / password access
• Create a Docker instance
• [Optional] Put behind a reverse proxy (we use nginx)

Grafana:

• Create a Docker instance
• [Optional] Put behind a reverse proxy (we use nginx)

Installing / Configuring Prometheus

Our requirements for configuring Prometheus were:

• fault tolerant
• can source data from multiple Conical instances
• Restricted access

To that end, we went for a simple approach of:

• A mounted volume to contain the prometheus DB
• A config file
• A webconfig.yml file containing the required user details

Configuration File (/datadrive/prometheus/config/prometheus.yml)

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # By default, scrape targets every 15 seconds.

scrape_configs:
  - job_name: 'demo-conical'
    # Override the global default and scrape targets from this job every 30 seconds.
    scrape_interval: 30s
    metrics_path: "/api/metrics"
    scheme: https
    authorization:
      type: Bearer
      credentials: youPATGoesHere
    static_configs:
      - targets: ['demo.conical.cloud']

To monitor additional instances, simply replicate the final block with the additional details.

WebConfig File (/datadrive/prometheus/config/webconfig.yml)

basic_auth_users:
  admin: encryptedPasswordHere

Details of how the basic auth works can be found on the prometheus website along with instructions on how to generate the encrypted password..

Docker Command to launch the container

sudo docker container run \
    -d \
    --name prometheus \
   --network monitoring \
   --hostname prometheus \
   -p 9090:9090 \
   -v /datadrive/prometheus/config:/etc/prometheus \
   -v /datadrive/prometheus/data:/var/prometheus \
   --restart always \
   -u 1002 \
    prom/prometheus:latest \
    --config.file=/etc/prometheus/prometheus.yml \
    --storage.tsdb.path=/var/prometheus \
    --web.config.file=/etc/prometheus/webconfig.yml

Note that we assigned ownership of /datadrive/prometheus to user 1002.

Installing Grafana

Installing the Grafana instance is much simpler and can be done with the following command:

sudo docker run \
  --network monitoring \
  --hostname grafana \
  -d \
  --name grafana \
  -p 3000:3000 \
  -v /datadrive/grafana:/var/lib/grafana \
  -u 1003 \
  --restart always \
  grafana/grafana:latest

Note that /datadrive/grafana is owned by user 1003.

Using Grafana with a reverse proxy (NGINX)

If you’re planning on using a reverse proxy with grafana, then you’ll need to ensure that the forwarding block sets the Host header, e.g.

    location / {
            proxy_set_header Host $http_host;
            proxy_pass http://internalIPAddress:3000;
        }

If this step isn’t done, then Grafana will misbehave subsequently when a user tries to log in.

Configuring Grafana

There are lots of tutorials on the internet as to how to configure data sources in Grafana so we will not seek to duplicate that information here. The main things to remember:

• The URL of the Prometheus server is from the perspective of the Grafana instance. Because we put both services in the same Docker network and specified host names, then we simply use http://prometheus:9000 and bypass all of the nginx functionality (which is reserved for enabling access from outside of our internal network)
• If using basic auth, then you’ll need to remember to set the user name / password combination.

Conclusion

With this set up, we’ve been able to monitor both our own demo instance of Conical and also some of our clients’ instances, allowing us to be proactive in fixing problems (usually disc space) before they become a real problem.

We hope that you will find these instructions helpful if you wish to add monitoring to your own instances. If you have any questions about your Conical instances or how we can help you improve your overall testing, then please do get in touch – contactus@conical.cloud.

Happy testing.