How to change Prometheus Alertmanager E-Mail template

linux, monitoring, tech 2 Minutes

The email templates for alerts from Prometheus is one of the things you might want to customize, and yet one of the worst documented features.

The default mail templates are compiled directly in the Alertmanager binary, you can check them out for reference on: https://raw.githubusercontent.com/prometheus/alertmanager/master/template/default.tmpl

You might want to check in particular the templates email.default.subject (that is a redirect to __subject) and email.default.html.

Both templates are quite dense and I suggest to “unpack” them to study them better, by reformatting them.

Create a template file

The first thing we want to do is to create custom templates for the subject and the body of our email.

cat > /opt/prometheus/templates/mail.tmpl <<EOF
{{ define "custom_mail_subject" }}Alert on {{ range .Alerts.Firing }}{{ .Labels.instance }} {{ end }}{{ end }}
{{ define "custom_mail_html" }}
<html>
<head>
<title>Alert!</title>
</head>
<body>
{{ range .Alerts.Firing }}

<p>{{ .Labels.alertname }} on {{ .Labels.instance }}<br/>
{{ if ne .Annotations.summary "" }}{{ .Annotations.summary }}{{ end }}</p>

<p>Details:</p>

<p>
{{ range .Annotations.SortedPairs }}
  {{ .Name }} = {{ .Value }}<br/>
{{ end }}
</p>

<p>
{{ range .Labels.SortedPairs }}
  {{ .Name }} = {{ .Value }}<br/>
{{ end }}
</p>

{{ end }}

</body></html>
{{ end }}

These are super-basic templates, for more customization check out the sources below.

Configure Alertmanager to use our templates

Once we have a template, we need to instruct Alertmanager to actually use it.

We need to include the template in the configuration:

[...]
global:
[...]
templates:
- '/opt/prometheus/templates/*.tmpl'

route:
[...]

And then we must change our email target definitions from:

- name: 'your-target-name'
  email_configs:
  - to: 'you@domain.tld'

to:

- name: 'your-target-name'
  email_configs:
  - to: 'you@domain.tld'
    headers:
      subject: '{{ template "custom_mail_subject" . }}'
    html: '{{ template "custom_mail_html" . }}'

Then we must restart alertmanager for the changes to apply.

Advertisement
Published