Error Budget Burn Investigation

It's Monday morning. You check the dashboard and see that your service burned 80% of its monthly error budget over the weekend. Walk me through how you'd investigate this and what you'd do next.

Senior questions

SREadvanced

// interview question

It's Monday morning. You check the dashboard and see that your service burned 80% of its monthly error budget over the weekend. Walk me through how you'd investigate this and what you'd do next.

Sample answer

First, I need to understand the shape of the burn. Was it one big incident or a slow leak? I'd start by pulling up the error budget burn-down chart for the weekend. If I see a cliff -- a steep drop at a specific time -- that's an incident. I'd look at the deployment history: did someone ship something Friday afternoon? Check the change log, git history, and any config changes. Most budget burns trace back to a change. If the chart shows a steady downward slope, that's a slow degradation. This is harder to track down. I'd look at the error breakdown by endpoint, by error code, and by dependency. Often it's a single endpoint or a downstream service that started flaking. Maybe a database connection pool is saturating under weekend batch job load, or a third-party API started rate-limiting you. Once I know the cause, here's what I'd do: 1. Fix the immediate problem. Roll back the bad deploy, scale up the bottleneck, or disable the flaky feature behind a feature flag. 2. Trigger the error budget policy. With only 20% budget left, we're in the critical zone. That means feature freeze -- no new deployments except reliability fixes. I'd communicate this to the product team immediately, not as a request but as the policy kicking in. 3. Run a blameless postmortem. Focus on why the burn wasn't caught sooner. Were the burn rate alerts misconfigured? Was nobody watching the on-call channel? Did the slow-burn alert fire but get ignored? 4. Build in prevention. If a Friday deploy caused this, maybe we need a policy of no deploys after Thursday, or better canary analysis. If a dependency caused it, we need circuit breakers or fallbacks. The bigger question is why 80% of the budget burned before anyone noticed. That's a monitoring and alerting gap, and it's usually more important to fix than the original incident.

Why this matters

This is a scenario-based senior question that tests incident response thinking, not just SLO knowledge. You want to see structured investigation (not random guessing), awareness of the error budget policy, and a focus on systemic improvements. The best candidates will immediately ask about the burn shape (cliff vs. slope) because the investigation path is completely different. Watch for whether they mention the process gap (why wasn't this caught?) -- that's what separates senior from mid-level thinking.

Code examples

Investigation queries to diagnose budget burn

promql

# 1. Error budget remaining over time (burn-down chart)
1 - (
  sum(increase(http_requests_total{service="checkout", status=~"5.."}[30d]))
  /
  (sum(increase(http_requests_total{service="checkout"}[30d])) * 0.001)
)

# 2. Error rate by endpoint -- find the hot spot
sum by (endpoint) (
  rate(http_requests_total{service="checkout", status=~"5.."}[1h])
)
/ sum by (endpoint) (
  rate(http_requests_total{service="checkout"}[1h])
)

# 3. Error rate by status code -- understand the failure mode
sum by (status) (
  increase(http_requests_total{service="checkout", status=~"5.."}[2d])
)

# 4. Dependency error rates -- is it us or a downstream?
sum by (dependency) (
  rate(dependency_requests_total{service="checkout", status="error"}[2d])
)

Quick weekend incident timeline from deploy history

bash

# Check what was deployed over the weekend
kubectl rollout history deployment/checkout-api -n production
# REVISION  CHANGE-CAUSE
# 14        deploy: v2.3.1 (Thu Apr 9 17:42:00)
# 15        deploy: v2.3.2 (Fri Apr 10 16:58:00)  <-- Friday afternoon deploy
# 16        rollback to v2.3.1 (Mon Apr 13 09:15:00)

# Check the diff between the Friday deploy and the previous version
git log --oneline v2.3.1..v2.3.2
# a1b2c3d Update payment retry logic
# d4e5f6g Remove connection pool size limit  <-- suspicious

# Correlate with error spike timing
kubectl logs deployment/checkout-api -n production --since=48h \
  | grep -c 'connection refused'
# 14,832

Postmortem template for budget burn events

yaml

postmortem:
  title: "80% error budget burn - checkout API (Apr 10-12)"
  date: 2025-04-13
  authors: ["oncall-engineer"]
  status: action-items-pending

  summary: |
    Friday deploy v2.3.2 removed connection pool limits,
    causing DB connection exhaustion under weekend batch load.
    80% of monthly error budget consumed over 48 hours.

  timeline:
    - time: "Fri 16:58" 
      event: "v2.3.2 deployed to production"
    - time: "Fri 23:00"
      event: "Weekend batch jobs begin, connection pool grows unbounded"
    - time: "Sat 02:15"
      event: "Slow-burn alert fires (10x burn rate)"
    - time: "Sat 02:15"
      event: "Alert acknowledged but not investigated (oncall assumed transient)"
    - time: "Mon 09:00"
      event: "Budget burn discovered during morning review"
    - time: "Mon 09:15"
      event: "Rollback to v2.3.1"

  root_cause: |
    Connection pool size limit was removed in v2.3.2.
    Under normal weekday traffic this was fine, but weekend
    batch jobs created enough concurrent connections to
    exhaust the database connection limit.

  action_items:
    - owner: sre-team
      action: "Add connection pool limits to deploy-time config validation"
      priority: P1
    - owner: oncall-lead
      action: "Update runbook: slow-burn alerts require investigation, not just ack"
      priority: P1
    - owner: platform-team
      action: "Add canary analysis gate for Friday deploys"
      priority: P2

Common mistakes to avoid

Jumping straight to 'fix the bug' without first understanding the full scope. Is it one endpoint or the whole service? Is it still happening or did it stop? You need the full picture before acting.
Skipping the process failure analysis. The budget burned over a weekend and nobody noticed until Monday -- that's a bigger problem than whatever caused the errors. Always ask why the alerting and response process failed.
Treating the error budget policy as optional. If the policy says feature freeze at 20% budget remaining, enforce it. Making exceptions undermines the entire system and trains teams to ignore SLOs.

Likely follow-ups

How would your response differ if the burn was caused by a third-party dependency you don't control?
The product team has a major launch next week and is pushing back on the feature freeze. How do you handle that conversation?
Should you adjust the SLO target after repeated budget burns, or is the problem always in the system?

Answer out loud first, then check yourself against the model answer.

Practice all Senior questions More SRE questions

#sre#error-budgets#incident-response#slos#postmortems#on-call

Also worth your time on this topic

Checklist

SLOs, SLIs, and Error Budgets: A Practical Implementation Guide

A step-by-step checklist for defining service level objectives, picking the right service level indicators, and using error budgets to make better decisions about reliability vs. feature velocity.

45-90 minutes

Interview

Error Budget Management

Your service has a 99.9% availability SLO over a 30-day window. How much downtime does that give you, and what do you actually do with that error budget day-to-day?

mid

Article

SLOs, SLIs, and Error Budgets: A Practical Implementation Guide

Your service went down at 2 AM and nobody could agree on whether it was "bad enough" to page someone. SLOs, SLIs, and error budgets fix that. Here is how to define, measure, and act on them with real Prometheus queries and alerting rules.

Error Budget Burn Investigation

More SRE interview questions

Also worth your time on this topic

SLOs, SLIs, and Error Budgets: A Practical Implementation Guide

Error Budget Management

SLOs, SLIs, and Error Budgets: A Practical Implementation Guide