Prometheus
Alert Manager & Long-term Storage

Lee Calcote

calcotestudios.com/talks

KubeCon + CloudNativeCon NA 2017
December, 4th 2017

Watch recording

Lee Calcote

linkedin.com/in/leecalcote

@lcalcote

gingergeek.com

lee@calcotestudios.com

clouds, containers, functions, applications and their management

@lcalcote

calcotestudios.com/talks

Show of Hands

AlertManager

Prometheus

Alertmanager is an alert...

@lcalcote

Purpose

ingester

grouper

de-duplicator

silencer

throttler

notifier

Receivers

\ˈnō-mən-ˌklā-chər

a brief Prometheus AlertManager construct review

match alerts to their receiver and how often to notify

where and how to send alerts

Routes

@lcalcote

Silencers - matches alerts with specific labels and prevents them from being included in notifications.

Inhibitors - suppress specific notifications when other specific alerts are already firing.

Grouping - categorizes alerts of similar nature into a single notification.

\ˈnō-mən-ˌklā-chər

a brief Prometheus AlertManager construct review

Muting

Suppressing

Correlating

                                group_wait: 30s

group_by: ['alertname', 'cluster']

group_interval: 5m

@lcalcote

Inhibition

Multiple approaches to suppression

@lcalcote

repeat_interval

vs

Silences

vs

per route

global

via ui / api

Alerts

                                ALERT <alert name>
   IF <PromQL vector expression>
   FOR <duration>
   LABELS { ... }
   ANNOTATIONS { ... }

Supports clients other than Prometheus

is notified when alerts transition state

@lcalcote

a shared construct

Prometheus

AlertManager

inactive

firing

pending

state transition

inactive

firing

notifications

Notification Integrations

@lcalcote

Notifying to Multiple Destinations

Use continue to advance to next receiver.

                                route:
 receiver: email_webhook

receivers:
  - name: email_webhook
    email_configs:
      - to: 'lee@example.io'
    webhook_configs:
      - url: <webhook url here>

Use a receiver that goes to both destinations.

                                route:
 receiver: ops-team-all  # default
 routes:
   - match:
       severity: page
     receiver: ops-team-b
     continue: true
   - match:
       severity: critical
     receiver: ops-team-a

receivers:
  - name: ops-team-all
    email_configs:
      - to: ops-team-all@example.io
  - name: ops-team-a
    email_configs:
      - to: ops-team-a@example.io
  - name: ops-team-b
    email_configs:
      - to: ops-team-b@example.io

or

@lcalcote

api

Inhibitor

de-duplication

Dispatcher

Non-HA AlertManager Architecture

Silencer

Dispatcher sorts incoming alerts into aggregation groups and assigns the correct notifiers to each.

Alert Provider

Silence Provider

store

de-duplication

Router

batched alerts

notification pipeline

Notify Provider

checks for previously sent notifications

Retry

Maintenance Script

@lcalcote

alerts

High Availability

being introduced in 0.5

gossip protocols.

built atop Weave Mesh

With HA, you no longer have to monitor the monitor.

Designed for an alert to be sent to all instances in the cluster.

All Prometheus instances send alerts to all Alertmanager instances.

Guarantees notifications to be sent at least once.

@lcalcote

Understanding and Extending