28 décembre 2021

Mirabelle 0.9.0: hello graphviz !

I released today the version 0.9.0 of Mirabelle, a stream processing engine I built. This release adds a new command named graphviz which generates a .dot file representing your streams configurations.

An example

Let’s take the commented stream example available on the Mirabelle main documentation page:

(streams
  (stream {:name :multiple_branches :description "Example configuration"}
    (where [:= :service "http_request_duration_seconds"]
      (with :ttl 60
        ;; push everything into influxdb
        (push-io! :influxdb)
        ;; index events in memory by host and service
        (index [:host :service])
        ;; by will generate a branch for each :host value. Like that, downstream
        ;; computations will be per host and will not conflict between each other
        (by [:host]
          ;; if the metric is greater than 1 for more than 60 seconds
          ;; Pass events downstream
          (above-dt {:duration 60 :threshold 1}
            ;; pass the state to critical
            (with :state "critical"
              ;; one alert only every 60 sec to avoid flooding pagerduty
              (throttle {:duration 60 :count 1}
                (push-io! :pagerduty)))))))))

The comments should be enough to explain what’s this stream is doing:

  • Filteting events with :service equal to http_request_duration_seconds

  • Setting the :ttl field to 60 if not set

  • forwarding all events to InfluxDB and storing them into the stream index.

  • Then, we send an alert to Pagerduty if the duration is greater than 1 during 60 seconds.

Cool. Let’s now run the new graphviz subcommand: mirabelle graphviz /tmp/graph.dot. This .dot file can now be converted to png using dot -Tpng graph.dot > graph.png.

This is the content of graph.png

Representation of the Mirabelle streams

Another example

Lets now execute the same command on another example:

(streams

 (stream {:name :main :default true :description "All events will arrive into this stream"}
   ;; inject events into the :alert and :influxdb streams
   (reinject! :alert)
   (reinject! :influxdb))

 (stream {:name :alert :description "Send alerts to pagerduty"}
   ;; keep only critical events
   (critical
     ;; split the actions by host/service
     (by [:host :service]
       ;; let only one event pass every 60 sec
       (throttle {:duration 60 :count 1}
         ;; send to pagerduty
         (push-io! :pagerduty)))))

 (stream {:name :influxdb :description "Send all events to InfluxDB"}
   ;; handle influxdb errors
   (exception-stream
     ;; set the :source key to "mirabelle"
     (with :source "mirabelle"
       ;; push to influxdb
       (push-io! :influxdb))
       ;; set the :service key to "mirabelle-influxdb-error"
     (with :service "mirabelle-influxdb-error"
       ;; inject events into the :alert stream
       (reinject! :alert)))))

I have in this configuration 3 streams:

  • The first one named :main has :default true so it will be used by default for events arriving on Mirabelle and not targeting a specific stream. This stream will send all events (using reinject! to the streams :alert and :influxdb.

  • The second one named :alert will keep only events with :state = critical, and then send alerts to Pagerduty (with a by ¨+ throttle to avoid spamming Pagerduty too much: only one event will be send to Pagerduty every 60 seconds for each :host/:service combination).

  • The third one named :influxdb will forward all events to InfluxDB. If pushing to InfluxDB fails (for example, the service is unavailable) an event containing the error will be sent to the :alert stream thank to exception-stream.

Let’s generate the graphical representation of these streams:

Representation of the Mirabelle streams

We can see how the streams are connected between each other.

Conclusion

I’m sure the .dot file generated by mirabelle graphviz can be improved to make it prettier, but I think this feature shows again the benefits of "compiling" the Mirabelle DSL into an intermediate datastructure (EDN) which can be then be consumed if needed (see the documentation for more information about this format).

I’m sure it should be easy for example to do the opposite: building a graphical editor for streams and then generating from the graphical representation the corresponding Mirabelle configuration !

Happy monitoring !

Tags: cloud projects english

Add a comment








If you have a bug/issue with the commenting system, please send me an email (my email is in the "About" section).

Top of page