Monitoring and Alerting Strategy

How do you design a monitoring and alerting strategy? What metrics would you track and how do you avoid alert fatigue?

Observabilityintermediate

// interview question

How do you design a monitoring and alerting strategy? What metrics would you track and how do you avoid alert fatigue?

Sample answer

Start with the four golden signals: latency, traffic, errors, and saturation. Use RED method for services (Rate, Errors, Duration) and USE method for resources (Utilization, Saturation, Errors). To avoid alert fatigue: alert on symptoms not causes, set appropriate thresholds with historical data, use severity levels (page vs ticket), implement alert grouping and deduplication, require runbooks for every alert, and regularly review and tune alerts. Only page for actionable issues that require immediate human intervention.

Why this matters

Effective monitoring enables proactive issue detection and faster incident resolution. Poor alerting leads to alert fatigue where important signals get ignored. This is a critical skill for maintaining reliable systems.

Code examples

Prometheus alerting rules example

yaml

groups:
- name: service-alerts
  rules:
  # High error rate - page immediately
  - alert: HighErrorRate
    expr: |
      sum(rate(http_requests_total{status=~"5.."}[5m]))
      / sum(rate(http_requests_total[5m])) > 0.05
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: High error rate detected
      runbook: https://wiki/runbooks/high-error-rate

  # High latency - create ticket
  - alert: HighLatency
    expr: |
      histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 0.5
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: P95 latency above 500ms

Common mistakes to avoid

Alerting on every metric threshold crossing
Not including runbooks with alerts
Setting thresholds without historical baseline data

Likely follow-ups

What is the difference between metrics, logs, and traces?
How do you decide between warning and critical alert severity?
What is an error budget and how does it relate to alerting?

Answer out loud first, then check yourself against the model answer.

Practice all Mid-Level questions More Observability questions

#monitoring#alerting#observability#sre#metrics

Also worth your time on this topic

Checklist

Monitoring & Observability Checklist

Comprehensive checklist for implementing monitoring, logging, tracing, and alerting across your infrastructure and applications.

60-90 minutes

Interview

Four Golden Signals of Monitoring

What are the four golden signals of monitoring and why are they important?

junior

Article

SLOs, SLIs, and Error Budgets: A Practical Implementation Guide

Your service went down at 2 AM and nobody could agree on whether it was "bad enough" to page someone. SLOs, SLIs, and error budgets fix that. Here is how to define, measure, and act on them with real Prometheus queries and alerting rules.

Monitoring and Alerting Strategy

More Observability interview questions

Also worth your time on this topic

Monitoring & Observability Checklist

Four Golden Signals of Monitoring

SLOs, SLIs, and Error Budgets: A Practical Implementation Guide