Chaos Engineering Practices

What is chaos engineering and how would you implement it safely in a production environment?

SREadvanced

// interview question

What is chaos engineering and how would you implement it safely in a production environment?

Sample answer

Chaos engineering is the practice of intentionally injecting failures to test system resilience. Implementation: start with hypothesis about system behavior, define minimal blast radius, begin in staging, use tools like Chaos Monkey or Litmus, inject failures (pod terminations, network latency, resource exhaustion), monitor golden signals, automate rollback on unexpected impact, gradually expand to production during low-traffic periods with team monitoring.

Why this matters

Chaos engineering's goal is finding weaknesses before they find you. It builds confidence in system resilience and uncovers hidden dependencies. The key is controlled, observable experiments with safety measures - not random destruction. Netflix pioneered this with Chaos Monkey, and it's now standard practice at large-scale organizations.

Code examples

Litmus ChaosEngine example

yaml

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: nginx-chaos
spec:
  appinfo:
    appns: default
    applabel: 'app=nginx'
    appkind: deployment
  chaosServiceAccount: litmus-admin
  experiments:
  - name: pod-delete
    spec:
      components:
        env:
        - name: TOTAL_CHAOS_DURATION
          value: '30'
        - name: CHAOS_INTERVAL
          value: '10'
        - name: FORCE
          value: 'false'

Manual chaos injection

bash

# Network latency injection
tc qdisc add dev eth0 root netem delay 100ms 20ms

# CPU stress
stress-ng --cpu 4 --timeout 60s

# Kill random pod in deployment
kubectl delete pod $(kubectl get pods -l app=myapp -o name | shuf -n 1)

# Network partition simulation
iptables -A INPUT -s 10.0.0.0/8 -j DROP

Common mistakes to avoid

Running chaos experiments without proper monitoring in place
Starting with production before validating in staging
No automated rollback mechanism when experiments go wrong

Likely follow-ups

How do you define and control the blast radius of chaos experiments?
What metrics should you monitor during chaos experiments?
How do you convince leadership that intentionally breaking production is valuable?

Answer out loud first, then check yourself against the model answer.

Practice all Senior questions More SRE questions

#chaos-engineering#reliability#sre#testing#resilience

Also worth your time on this topic

Article

Running Your First Chaos Engineering Experiment with Litmus

A hands-on walkthrough of installing LitmusChaos on Kubernetes, killing pods on purpose, and watching whether your app actually recovers. Real YAML, real output, no theory.

Checklist

SLOs, SLIs, and Error Budgets: A Practical Implementation Guide

A step-by-step checklist for defining service level objectives, picking the right service level indicators, and using error budgets to make better decisions about reliability vs. feature velocity.

45-90 minutes

Interview

SLO vs SLI vs SLA Differences

Your team just launched a new API service. Your manager asks you to set up SLOs for it. Can you walk me through what SLOs, SLIs, and SLAs are, and how they relate to each other?

junior

Chaos Engineering Practices

More SRE interview questions

Also worth your time on this topic

Running Your First Chaos Engineering Experiment with Litmus

SLOs, SLIs, and Error Budgets: A Practical Implementation Guide

SLO vs SLI vs SLA Differences