Pyrra with Prometheus/Grafana on Docker Swarm

[prologic]

Hi 👋

Just some quick feedback on Pyrra so far.

Infrastructure:

• Docker Swarm cluster
• Prometheus (local storage)
• Grafana

The setup was quite smooth once I'd fixed a few local volume permission problems. We might want to fiddle with the Docker images a bit and provide some flexibility there with what UID/GID the process runs as.

Didn't realise I had to add the --generic-rules until afterwards, re-read the docs realizing I needed this option. Maybe make this a default?

On the UI side of things, I'd love it if there was some visual indicator of your SLO(s) being "within budget", "out of budget", etc.

[prologic]

Additionally it might be nice if to provide an API and CLI for creating/managing SLOs? That could be a nicer experience, writing out YAML files is okay too, but this way I don't have to fiddle with volumes, permission etc.

[metalmatze]

We have a generate command to generate the Prometheus recording rules from the configs. But if I'm understanding correctly, you're looking for way to not write the YAML configs? I'm not sure what the CLI would look like. For a API we actually have a branch where we optionally create the config with a UI form.

./pyrra generate --help

Usage: pyrra generate [flags]

Read SLO config files and rewrites them as Prometheus rules and alerts.

Flags:
  -h, --help                   Show context-sensitive help.
      --log-level="info"       Used to set the logging level of the application.
                               Valid options are 'debug', 'info', 'warn' or
                               'error'
      --log-format="logfmt"    Used to set the logging format. Valid options are
                               'logfmt' or 'json'

      --config-files="/etc/pyrra/*.yaml"
                               The folder where Pyrra finds the config files to
                               use.
      --prometheus-folder="/etc/prometheus/pyrra/"
                               The folder where Pyrra writes the generated
                               Prometheus rules and alerts.
      --generic-rules          Enabled generic recording rules generation to
                               make it easier for tools like Grafana.
      --operator-rule          Generate rule files as
                               prometheus-operator PrometheusRule:
                               https://prometheus-operator.dev/docs/operator/api/#monitoring.coreos.com/v1.PrometheusRule.

[prologic]

Nah not to avoid writing the SLO YAML(s), but "upload" them I guess?

[metalmatze]

Didn't realise I had to add the --generic-rules until afterwards, re-read the docs realizing I needed this option. Maybe make this a default?

Fair, however, this mode isn't the default. I don't expect everyone always to want to run Grafana alongside. Therefore, not generating like 5-10x more recording rules seems the better option.

On the UI side of things, I'd love it if there was some visual indicator of your SLO(s) being "within budget", "out of budget", etc.

The error budget is above 0% (so within the budget) or below 0% (out of budget). It'll go into the negative. Check out https://demo.pyrra.dev/ for an example. Is there something else you were expecting?

Additionally it might be nice if to provide an API and CLI for creating/managing SLOs? That could be a nicer experience, writing out YAML files is okay too, but this way I don't have to fiddle with volumes, permission etc.

[prologic]

Fair, however, this mode isn't the default. I don't expect everyone always to want to run Grafana alongside. Therefore, not generating like 5-10x more recording rules seems the better option.

Fair point 👌

The error budget is above 0% (so within the budget) or below 0% (out of budget). It'll go into the negative. Check out https://demo.pyrra.dev/ for an example. Is there something else you were expecting?

Oh! I didn't notice the subtle "red" colored text. Forgive me but I'm vision impaired and I wasn't able to see this. Can we maybe give the row or cell a background color?