Everytime an healthcheck is executed, Cabourotte will generate and push to my Mirabelle instance an event. The event looks like (in JSON, in reality protobuf is used for performances, like in Riemann):

If you compare that with the healthcheck configuration:

Things should be clear. The metric field is the duration of the healthcheck execution. Let’s now configure Mirabelle in order to alert on these events.

Let’s first put a file name io.edn in /etc/mirabelle/io and define a new Pagerduty I/O named :pagerduty-client:

The Pagerduty I/O options are described in the documentation.

Now, let’s write our streams.

Now, we will create a new file named stream.clj in the directory you want (it will compiled later). What we first want to do is:

Here is the Mirabelle configuration to do this:

We first use not-expired to filter expired events, and then keep only events with :service equal to cabourotte-healthcheck.

We then want to treat each healthcheck independently. We indeed want to alert and resolve alerts based on states transitions for each healthcheck and don’t want conflicts between healthchecks or Cabourotte instances (you can have multiple Cabourotte instances pushing events from multiple hosts).

We do that in Mirabelle by using the by action, here with by [:host :healthcheck]. Everything after by will be splitted for each combination of :host and :healthcheck.

Then, we do (changed :state "ok". This action will let events pass only on states transitions, the default value being "ok". It will prevent Mirabelle to spam the Pagerduty API for each event.

Finally, we use sformat to rebuild the :service value. Events will be updated and will have as :service the value cabourotte-healthcheck-alert-<healthcheck-name>. Indeed, we configured our Pagerduty client with :host and :service as :dedup-key (to detect which event belongs to which alert), so the service value should be unique for each healthcheck.

We then do several things:

And we’re done !

You now need to compile this file into an EDN representation. Everything is explained in the documentation (it’s only a java -jar mirabelle.jar compile <src-directory> <dst-directory>). The result should be put in /etc/mirabelle/streams which is the directory referenced in your configuration file.

A lot of monitoring tools do not allow you to write unit tests on their configurations. It’s a big issue because you are unable to verify the configuration because of that.

Mirabelle has a simple but powerful test system built in. Here is a test I wrote for ths previous configuration:

Events in :input will be injected in Mirabelle and we can then verify what was recorded in the tap actions (in our case, the :cabourotte-pagerduty one).

As you can see, things work as expected: only states transitions are forwarded to Pagerduty, and my :service key was correctly updated.

I killed my blog containers to test the setup. I Immediately receive the alerts in Pagerduty:

Once the website is back, the alert is automatically resolved:

I could to a lot more, like using the Mirabelle stable action to avoid flapping states for example. But for my simple use case this configuration is enough.

Yes ! Mirabelle is powerful and extensible (you can write your own actions). As I said before, the events :metric field contains the healthchecks duration. Why not compute the percentiles on the fly for each healthcheck ?

If you add this code snippet after (not expired in your original configuration, percentiles will be computed for each healthcheck independently (note that :host is not used in by here, we want to compute the percentiles on events for a given healthcheck fromm all Cabourotte instances together).

The events generated for each quantile will be updated with a new service name and then passed to publish!, which allows use to subscribe to it using a websocket client (users can also subscribe to the index).

For example, I could use the Python script referenced in the Mirabelle documentation to subscribe to the percentiles events for the healthcheck "https-mcorbin.fr":

./websocket.py --channel healthchecks-percentiles --query '[:= :healthcheck "https-mcorbin.fr"]':

Mirabelle, like Riemann, is very powerful. You have tons of built-in actions available, can have multiple streams running in parallel with different clocks (for real time computations and continuous queries…​), an API to manage streams, an InfluxDB integration…​ The documentation explains all of that ;)