Toolmingo
Guides31 min read

50 SRE Interview Questions (With Answers)

Top Site Reliability Engineer interview questions with detailed answers — covering SLOs/SLIs/SLAs, error budgets, toil reduction, incident response, chaos engineering, capacity planning, and on-call best practices.

Site Reliability Engineer interviews test your understanding of reliability principles, SLOs/SLIs/SLAs, error budgets, incident response, system design, and production operations. This guide covers the 50 most common SRE interview questions — with concise answers, formulas, and real examples used at Google, Amazon, Netflix, and other top engineering orgs.

Quick reference

Topic Most asked questions
Core SRE SLO/SLI/SLA definitions, error budget, DevOps vs SRE
Reliability Four nines math, toil, MTTR/MTTF/MTBF
Incident response Severity levels, postmortems, on-call runbooks
Monitoring RED/USE methods, alerting philosophy, SLO-based alerts
Capacity planning Load testing, headroom, scaling strategies
Chaos engineering Game days, failure injection, blast radius
Automation Toil elimination, runbook automation, self-healing
System design Load balancers, caching, retries, circuit breakers
Kubernetes/Linux Resource limits, HPA, node affinity, debugging
Soft skills Blameless culture, stakeholder communication

Core SRE Concepts

1. What is Site Reliability Engineering and how does it differ from DevOps?

Aspect SRE DevOps
Origin Google (2003, Ben Treynor Sloss) Cultural movement (~2009)
Focus Reliability via software engineering Collaboration + delivery speed
Metrics SLO/SLI/error budget Deployment frequency, lead time
Approach Prescriptive (Google SRE book) Principles-based
On-call Dedicated SRE team owns production Shared model (you build it, you run it)
Toil Explicitly measured and capped (< 50%) Not formally defined

"SRE is what you get when you treat operations as a software problem." — Ben Treynor Sloss

SRE is a concrete implementation of DevOps with explicit reliability contracts.


2. Define SLI, SLO, and SLA. Give examples for each.

Term Full Name What it is Example
SLI Service Level Indicator Quantitative metric measuring service behavior 99.7% of requests complete in < 300 ms
SLO Service Level Objective Internal target for an SLI 99.9% availability over 30 days
SLA Service Level Agreement External contract with consequences for breach 99.5% uptime or 10% credit

Relationship:

  • SLI is the measurement
  • SLO is the goal (always stricter than the SLA)
  • SLA is the promise to customers (legal/commercial)

Good SLIs are:

  • Directly user-visible (availability, latency, error rate, throughput)
  • Measurable with high fidelity
  • Not proxy metrics (CPU usage is a bad SLI)

3. What is an error budget and how do you use it?

An error budget is the allowed amount of unreliability derived from the SLO:

Error budget = 1 − SLO target

99.9% SLO → 0.1% error budget = 43.8 minutes/month downtime allowed
99.99% SLO → 0.01% error budget = 4.38 minutes/month

How teams use error budgets:

Budget remaining Action
> 50% Ship features aggressively, run chaos experiments
10–50% Normal pace, watch reliability trends
< 10% Slow feature work, focus on reliability
Exhausted Feature freeze, reliability work only

Error budgets create a shared incentive: developers want to ship features, SREs want reliability. The budget lets both sides negotiate using data instead of opinion.


4. Calculate the error budget for 99.9% and 99.99% SLOs over 30 days.

30 days = 30 × 24 × 60 = 43,200 minutes

99.9% SLO:
  Error budget = 0.1% of 43,200 = 43.2 minutes/month

99.99% SLO:
  Error budget = 0.01% of 43,200 = 4.32 minutes/month

99.999% SLO ("five nines"):
  Error budget = 0.001% of 43,200 = 0.432 minutes ≈ 26 seconds/month

5. What is toil and why does Google cap it at 50%?

Toil = manual, repetitive, automatable, tactical work that scales with service growth and has no enduring value.

Characteristics of toil:

  • Manual: requires human action (e.g., restarting a service)
  • Repetitive: done again and again
  • Automatable: a machine could do it
  • Reactive: triggered by an alert, not a project
  • No enduring value: doesn't improve the system

Google caps SRE toil at < 50% of time. Above that:

  • SREs become ops engineers with no time to improve the system
  • Systems never get more reliable
  • Engineers burn out and leave

The other 50%+ goes to engineering work: automation, observability improvements, architectural changes.


6. What is the difference between MTTR, MTTF, and MTBF?

Metric Full Name Formula Meaning
MTTF Mean Time To Failure Total uptime / number of failures How long system runs before failing
MTTR Mean Time To Recovery Total downtime / number of incidents How fast you recover from failure
MTBF Mean Time Between Failures MTTF + MTTR Average time between start of one failure and next
MTBF = MTTF + MTTR

Availability ≈ MTTF / (MTTF + MTTR) = MTTF / MTBF

SRE focuses heavily on reducing MTTR because:

  • Failures are inevitable (MTTF has limits)
  • Fast recovery keeps error budget intact
  • Good observability, runbooks, and on-call practices reduce MTTR

7. What makes a good SLO?

A good SLO is:

  1. User-centric — measures what users experience, not internal system metrics
  2. Achievable — based on historical performance, not aspirational
  3. Specific — includes measurement window, SLI definition, and target
  4. Actionable — breaching it triggers meaningful action
  5. Minimal — 2-4 SLOs per service (too many dilutes focus)

Bad SLO: "The system should be fast" Good SLO: "99.9% of checkout API requests complete in < 500ms, measured over a rolling 28-day window"


8. How do you handle an SLO that's consistently at 100%?

A service at 100% SLI is a signal that your SLO is too loose. This means:

  • You're over-investing in reliability beyond what users need
  • Your error budget is never consumed (so you can't run experiments)
  • You may be masking real problems with conservative alerting

Actions:

  1. Tighten the SLO to match actual user expectations
  2. Use freed engineering time on feature development
  3. Run chaos experiments to understand real failure modes

Incident Response

9. Walk me through your incident response process.

A standard incident response process:

Detection → Triage → Mitigation → Resolution → Postmortem

Step by step:

  1. Detection: Alert fires (or user report) — assign severity
  2. Incident Commander (IC): One person drives, avoids multitasking
  3. Communication Lead: Updates status page, notifies stakeholders
  4. Triage: Understand scope — who/what is affected, since when
  5. Mitigation: Restore service ASAP (rollback > root cause fix)
  6. Resolution: Confirmed fix, verify SLI recovery
  7. Postmortem: Written within 48 hours, blameless, with action items

Key principle: Mitigate first, diagnose later. A slow rollback beats a fast root-cause hunt.


10. What are severity levels and how do you define them?

Severity Definition Example Response time
SEV-1 Complete outage, all users affected Checkout down Immediate, 24/7
SEV-2 Major functionality degraded 50% error rate on search < 15 min
SEV-3 Partial degradation, workaround exists Slow dashboard loads Business hours
SEV-4 Minor issue, no user impact Log noise, cosmetic bug Next sprint

Definitions vary by company. What matters: severity drives escalation, not blame.


11. What is a blameless postmortem and what should it contain?

A blameless postmortem assumes that people made decisions with the best information available at the time. It focuses on systems, not individuals.

Structure:

## Incident summary
Date, duration, severity, services affected

## Timeline
[Time] Event (what happened, who noticed, what was done)

## Root cause
The technical condition that caused the incident

## Contributing factors
Conditions that made the incident possible or worse

## Resolution
What stopped the incident

## Impact
Users affected, error budget consumed, revenue lost

## Action items
| Action | Owner | Priority | Due date |
|--------|-------|----------|----------|

## Lessons learned
What went well, what went poorly, where we got lucky

Avoid blame language: "John made a mistake" → "The deploy process lacked a safeguard that would have caught this"


12. How do you handle alert fatigue?

Alert fatigue = too many alerts, engineers stop responding to them.

Causes:

  • Alerts not tied to SLOs (alerting on symptoms, not user impact)
  • Alerts without clear action (what do I do when this fires?)
  • Flapping alerts (fires/resolves repeatedly)

Solutions:

Problem Fix
Too many low-value alerts Audit — if no action taken in 6 months, delete
Alerts not actionable Require runbook link in every alert
Flapping Add for: 5m delay, or re-evaluate threshold
Not SLO-linked Replace with burn rate alerts
Wrong severity Re-triage — not every alert is SEV-2

Goal: every alert should be urgent, actionable, and represent user impact.


13. What is an SLO burn rate alert?

Instead of alerting on individual metrics, a burn rate alert fires when you're consuming your error budget too fast.

Burn rate = (error rate) / (1 − SLO)

Example: SLO = 99.9%, current error rate = 1%
Burn rate = 1% / 0.1% = 10x
At 10x burn, you exhaust your monthly budget in 3 days.

Google's multi-window approach:

Window Burn rate Alert type
1h + 5m 14.4x Critical page
6h + 30m 6x Critical page
3d + 6h 1x Ticket

Fast burn = fast window (catches spikes). Slow burn = long window (catches sustained issues). Both needed to avoid missing incidents.


14. How do you write a good runbook?

A runbook documents how to respond to a specific alert or operational task.

Good runbook structure:

## Alert: HighCheckoutErrorRate

### What does this mean?
Checkout error rate > 1% for 5+ minutes. Users cannot complete purchases.

### Severity
SEV-1 if > 5%, SEV-2 if 1–5%

### Immediate checks
1. Check Grafana dashboard: [link]
2. Recent deployments: `kubectl rollout history deployment/checkout`
3. Database health: [link to DB dashboard]

### Common causes + fixes
| Symptom | Likely cause | Fix |
|---------|-------------|-----|
| DB errors in logs | DB connection pool exhausted | Restart app pods: `kubectl rollout restart deployment/checkout` |
| 502s from load balancer | Pod OOM killed | Scale up: `kubectl scale deployment/checkout --replicas=10` |
| Auth errors | JWT signing key rotated | Verify secret: `kubectl get secret jwt-key` |

### Escalation
If not resolved in 15 min: page backend oncall + notify #incidents

### Rollback
`kubectl rollout undo deployment/checkout`

Monitoring & Observability

15. Explain the RED and USE methods.

RED Method (for services/microservices):

  • Rate — requests per second
  • Errors — failed requests per second
  • Duration — latency distribution (p50, p95, p99)

USE Method (for resources: CPU, memory, disk, network):

  • Utilization — % of time resource is busy
  • Saturation — how much work is queued/waiting
  • Errors — error events for that resource
Method Best for Examples
RED APIs, services HTTP endpoints, gRPC, queues
USE Infrastructure CPU, disk I/O, network interfaces

Use RED to detect user-facing problems. Use USE to find the infrastructure bottleneck causing them.


16. What are the four golden signals?

Google's four golden signals (from the SRE book):

Signal What it measures Alert on?
Latency Time to serve a request (successful vs failed separately) p99 exceeds threshold
Traffic Demand on the system (RPS, queries/sec) Unusual spikes (DDoS, viral)
Errors Rate of failed requests (explicit 5xx, implicit wrong results) SLO burn rate
Saturation Fullness of resource (CPU %, disk %, queue depth) Headroom < 20%

These four cover most production problems. If your system measures all four, you'll catch nearly every incident.


17. What is distributed tracing and when do you use it?

Distributed tracing tracks a request as it flows through multiple services by propagating a trace ID through all service calls.

When to use:

  • Debugging latency in microservices (which service is slow?)
  • Finding where errors originate in a request chain
  • Understanding service dependencies

Tools: Jaeger, Zipkin, AWS X-Ray, OpenTelemetry

Key concepts:

  • Trace: end-to-end journey of a request
  • Span: a single operation within a trace (one service call)
  • Context propagation: trace ID passed via HTTP headers (traceparent)
User → API Gateway (span 1)
  → Auth Service (span 2)
  → Product Service (span 3)
    → Database (span 4)
  → Payment Service (span 5)

Without tracing, debugging a 3-second request across 5 services is guesswork.


18. Prometheus vs Datadog vs CloudWatch — when do you use each?

Tool Type Best for Cost
Prometheus Open source, pull-based Kubernetes-native, metrics only Free (infra cost only)
Grafana Open source visualization Dashboards on top of Prometheus Free
Datadog SaaS Full observability (metrics + traces + logs), easy onboarding Expensive per host
CloudWatch AWS-native SaaS AWS resources, Lambda, EC2 Pay per metric/log
New Relic SaaS Full-stack observability, APM Per core/user

Typical stack: Prometheus + Grafana for metrics, OpenTelemetry for traces, Loki for logs (all open source). Or Datadog for teams that want one tool.


Capacity Planning

19. How do you approach capacity planning?

Capacity planning ensures enough resources exist to handle current and future load without over-provisioning.

Process:

  1. Baseline: measure current resource utilization (CPU, memory, disk, network)
  2. Model growth: project traffic growth (historical trend + product roadmap)
  3. Load test: find the breaking point under synthetic load
  4. Headroom target: keep utilization below 70% (30% headroom for spikes)
  5. Alert on headroom: alert when headroom < 20%
  6. Review quarterly: adjust for actual vs projected growth
Example:
  Current: 100 RPS, 60% CPU on 10 servers
  Projected: 2x traffic in 6 months = 200 RPS
  At 200 RPS → 120% CPU (would fail)
  Need: 20 servers by month 6 (or scale-out automation)

20. What is the difference between horizontal and vertical scaling?

Aspect Horizontal scaling (scale out) Vertical scaling (scale up)
What Add more instances Add more CPU/RAM to existing instance
Limit Theoretically unlimited Hardware ceiling
Downtime Zero (rolling adds) Often requires restart
Cost Linear, pay-as-you-go Diminishing returns
Complexity Needs load balancer, stateless design Simpler
SRE preference Preferred for web services Used for DBs (read replicas, then vertical)

Stateless services scale horizontally. Stateful services (databases) are harder — use read replicas, sharding, or managed services.


21. How do you handle a traffic spike that exceeds capacity?

Before the spike (defense in depth):

  1. Auto-scaling configured (HPA in Kubernetes, AWS Auto Scaling)
  2. Load shedding: return 429 at threshold instead of cascading failure
  3. Circuit breakers: stop calling overloaded downstream services
  4. Caching: CDN, Redis to reduce backend load
  5. Rate limiting: per-user/IP limits

During the spike:

  1. Check auto-scaler: is it scaling? Why not?
  2. Identify the bottleneck (RED metrics: which service is saturated?)
  3. Shed traffic if needed (feature flag, CDN rules)
  4. Scale manually if auto-scaler is slow

After the spike:

  • Postmortem: was it expected? Why didn't auto-scaler handle it?
  • Adjust scaling policies

Chaos Engineering

22. What is chaos engineering?

Chaos engineering is the practice of deliberately injecting failures into production (or staging) to discover weaknesses before they cause real incidents.

Principles:

  1. Hypothesize about steady-state behavior
  2. Vary real-world events (server crash, latency spike, dependency failure)
  3. Run experiments in production (with blast radius control)
  4. Automate continuously

Netflix Simian Army examples:

  • Chaos Monkey: randomly kills production instances
  • Chaos Kong: simulates entire AWS region failure
  • Latency Monkey: injects artificial latency

Before running chaos experiments:

  • Have good observability (you can't measure what broke)
  • Have an SLO to measure impact against
  • Start small: test in staging first, then production with small % traffic

23. What is a game day?

A game day is a planned exercise where the team simulates a production failure scenario to test response capabilities.

Format:

  1. Define a failure scenario (e.g., "primary database fails")
  2. Inject the failure (or simulate it with a runbook)
  3. Team responds as if it's a real incident (incident commander, comms, runbook)
  4. Measure: detection time, MTTR, runbook accuracy
  5. Debrief: what worked, what broke, what needs improvement

Game days improve:

  • Runbook accuracy ("did the runbook actually work?")
  • Team confidence and muscle memory
  • On-call readiness
  • Discovery of unknown failure modes

24. How do you control the blast radius of a chaos experiment?

Techniques:

Technique Description
Traffic percentage Kill 1% of instances, not all
Feature flags Enable chaos only for % of users
Start in staging Validate experiment before production
Time-boxing Run for 10 minutes, then stop
Auto-abort If SLI drops below threshold, stop automatically
Read-only first Inject latency before killing services

The goal is to learn without causing a real incident. Start with the smallest failure that teaches you something useful.


Automation & Toil Reduction

25. Give an example of toil you've eliminated through automation.

Example — manual certificate renewal:

Before: Engineer manually ran certbot renew on 20 servers each quarter, checked expiry dates via calendar reminder, 4 hours of toil per quarter.

After: Cert-manager in Kubernetes auto-renews Let's Encrypt certs 30 days before expiry. Prometheus alert fires if any cert expires within 14 days as a safety net. Toil eliminated: 4 hours/quarter → 0.

Framework for toil elimination:

  1. Measure: how long does this take per occurrence?
  2. Frequency: how often does it happen?
  3. Annual cost: time × frequency × number of engineers
  4. Automation cost: how long to automate?
  5. Break-even: payback period

If break-even < 1 year, automate it.


26. What is self-healing infrastructure?

Self-healing infrastructure detects and automatically recovers from failure without human intervention.

Examples:

Failure Self-healing mechanism
Pod OOM killed Kubernetes restarts pod (restartPolicy: Always)
Unhealthy instance Auto Scaling replaces EC2 instance
DB connection exhausted Connection pool health check + restart
Zombie process Systemd Restart=on-failure
Disk full Automated log rotation, cleanup cron
Certificate expired cert-manager auto-renews

Limits of self-healing:

  • Can mask underlying problems (pod keeps restarting = OOM loop)
  • Need alerting on restart count (alert if restarts > 5 in 1 hour)
  • Self-healing is last-resort; fix root cause in parallel

System Design for Reliability

27. What is a circuit breaker pattern?

A circuit breaker prevents a service from repeatedly calling a failing dependency, allowing the dependency time to recover.

States:

CLOSED (normal) → failure threshold exceeded →
OPEN (reject calls) → timeout period elapsed →
HALF-OPEN (test one call) → success → CLOSED
                          → failure → OPEN

Benefits:

  • Prevents cascading failures
  • Reduces load on struggling service
  • Fast failure (immediate 503) instead of timeout cascade
# Pseudocode
if circuit_breaker.is_open():
    return fallback_response()  # don't call dependency
try:
    result = call_dependency()
    circuit_breaker.record_success()
    return result
except Exception:
    circuit_breaker.record_failure()
    raise

Libraries: Resilience4j (Java), Polly (.NET), pybreaker (Python)


28. How do you design a service for 99.99% availability?

99.99% = 52.6 minutes downtime/year.

Architecture requirements:

Layer Requirement
No single point of failure Every component has ≥ 2 instances
Multi-AZ Span at least 2 availability zones
Auto-scaling Handle 3× peak traffic
Health checks Remove unhealthy instances in < 30s
Rolling deploys Zero-downtime deployments
Graceful shutdown Drain connections before terminating
Circuit breakers Fail fast on dependency failure
Retry with backoff Retry transient failures; don't retry 4xx
Caching CDN + application-level cache reduce DB pressure
DB: read replicas Separate read and write load
DB: connection pooling PgBouncer, HikariCP

99.99% is expensive. Validate that users actually need it (most don't).


29. Explain retry with exponential backoff and jitter.

Retry logic handles transient failures. Without backoff, retries cause thundering herd (all clients retry simultaneously, overwhelming the server).

import random
import time

def retry_with_backoff(func, max_retries=5, base_delay=1.0):
    for attempt in range(max_retries):
        try:
            return func()
        except TransientError as e:
            if attempt == max_retries - 1:
                raise
            # Exponential backoff with full jitter
            delay = base_delay * (2 ** attempt)
            jitter = random.uniform(0, delay)
            time.sleep(jitter)

# Delays with jitter: ~0.5s, ~1s, ~2s, ~4s, ~8s (randomized)

Jitter (random delay) spreads retries across time, preventing synchronized retry storms.

Don't retry:

  • 4xx errors (client errors — retry won't fix them)
  • Operations that aren't idempotent (non-idempotent POST)
  • After exhausting budget (use dead letter queue)

30. What is the difference between a hard dependency and a soft dependency?

Type Definition Failure behavior
Hard dependency Service cannot function without it Return error if unavailable
Soft dependency Service degrades gracefully without it Return partial result or default

Example: E-commerce product page

  • Database: hard (can't show products without it)
  • Recommendation engine: soft (show page without "You may also like")
  • Analytics tracking: soft (user doesn't see it fail)
  • Review service: soft (show product without reviews)

SRE principle: make as few things hard dependencies as possible. Each hard dependency directly affects your SLO.


Kubernetes & Linux Operations

31. A pod keeps crashing. How do you debug it?

# 1. Check pod status
kubectl get pod <pod-name> -n <namespace>

# 2. Describe pod (events, resource limits, last termination reason)
kubectl describe pod <pod-name> -n <namespace>

# 3. View current logs
kubectl logs <pod-name> -n <namespace>

# 4. View logs from previous (crashed) container
kubectl logs <pod-name> -n <namespace> --previous

# 5. If OOMKilled: check memory limits vs usage
kubectl top pod <pod-name> -n <namespace>

# 6. Exec into running pod (if it's not crash-looping)
kubectl exec -it <pod-name> -n <namespace> -- /bin/sh

Common crash causes:

Exit code / status Cause Fix
OOMKilled Memory limit too low Increase resources.limits.memory
CrashLoopBackOff App crashes on start Check logs for startup error
Error 1 Application-level error Check app logs
ImagePullBackOff Can't pull image Check image name, registry credentials
Pending No nodes with capacity Check node resources, taints

32. How does Kubernetes HPA work?

Horizontal Pod Autoscaler (HPA) scales replica count based on metrics.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70  # target 70% CPU
  - type: Pods
    pods:
      metric:
        name: requests_per_second
      target:
        type: AverageValue
        averageValue: "1000"

HPA algorithm:

desired replicas = ceil(current replicas × (current metric / desired metric))

SRE considerations:

  • Set minReplicas ≥ 2 (HA)
  • Ensure requests/limits set (HPA requires them for CPU/memory)
  • Scale-down is slow by default (5 min cooldown) — good for stability

33. How do you debug high CPU on a Linux server?

# 1. Which process is consuming CPU?
top -c         # or htop

# 2. Detailed CPU breakdown (user/system/iowait/steal)
vmstat 1 10

# 3. Per-CPU stats (is it one core or all?)
mpstat -P ALL 1

# 4. What is a specific process doing?
pidstat -u -p <PID> 1

# 5. System calls being made (what is the process actually doing?)
strace -p <PID> -c

# 6. If iowait is high (not CPU-bound, but waiting for disk)
iostat -x 1
iotop

# 7. Check if it's a kernel thread
ps aux --sort=-%cpu | head -20

Common high-CPU causes:

  • Infinite loop in application
  • Garbage collection (Java/Go)
  • High request volume
  • Kernel: high iowait = disk bottleneck, not CPU

34. Explain Linux file descriptors and why "too many open files" errors occur.

Every open file, socket, pipe, and device is a file descriptor (fd).

# Current limits
ulimit -n          # soft limit for current process
cat /proc/sys/fs/file-max   # system-wide maximum

# How many fds is a process using?
ls /proc/<PID>/fd | wc -l

# What files are open?
lsof -p <PID>

# System-wide fd usage
lsof | wc -l

# Increase limit for current session
ulimit -n 65536

# Permanent: /etc/security/limits.conf
*    soft    nofile    65536
*    hard    nofile    65536

Common cause: Connection leak — application opens DB connections, HTTP connections, or files and never closes them. Eventually hits the limit (default: 1024).


On-Call & Culture

35. How do you make on-call sustainable?

Symptoms of unsustainable on-call:

  • 2 pages per on-call shift

  • Pages at 2 AM repeatedly
  • No time between alerts to investigate
  • Engineers dread on-call rotation

Solutions:

Problem Fix
Too many alerts Audit and delete non-actionable alerts
Repeated same alert Fix root cause or add auto-remediation
3 AM pages for non-urgent issues Define SEV levels, page only SEV-1/2 at night
Single person on-call Rotate team, add shadow/backup on-call
No runbooks Require runbook link in every alert
Alert fatigue Weekly on-call review + postmortem for repeat alerts

Google's benchmark: on-call load should be < 25% of SRE time. Excess pages = technical debt.


36. How do you prioritize what to fix after an incident?

Use the postmortem action items framework:

Priority Criteria Example
P1 — Do now Prevents reoccurrence, easy to fix Add circuit breaker to dependency
P2 — This sprint Reduces MTTR significantly Write runbook for this alert class
P3 — Backlog Reduces likelihood Migrate off deprecated library
Won't fix Cost > benefit Edge case affecting 0.001% of users

Track action items in JIRA/Linear with an owner, due date, and link to the postmortem. Review completion rate in monthly reliability reviews.


37. What is the principle of defense in depth?

Multiple independent layers of protection so no single failure causes complete disaster.

Applied to SRE:

Layer 1: Code review + testing (prevent bugs entering production)
Layer 2: Canary deployment (affect 1% of traffic before 100%)
Layer 3: Feature flags (disable feature without deploy)
Layer 4: Rate limiting (prevent abuse from taking down service)
Layer 5: Circuit breakers (isolate failing dependency)
Layer 6: Auto-scaling (handle traffic spikes)
Layer 7: Multi-AZ redundancy (survive AZ failure)
Layer 8: Backups + DR (survive data loss)
Layer 9: On-call + runbooks (human last resort)

No single layer is perfect. The combination makes the system robust.


Advanced SRE Topics

38. What is progressive delivery and how does it reduce risk?

Progressive delivery = releasing changes to increasing percentages of users, with automated rollback on SLO breach.

Strategies:

Strategy Description Risk
Canary 1% → 5% → 25% → 100% of traffic Very low
Blue/green Full switch, keep old env running Medium (full traffic at once)
Shadow Mirror traffic to new version (no user impact) Zero
A/B test Different versions for different user segments Low
Feature flag Enable/disable feature without deploy Very low

SRE monitors SLIs during canary. If error rate increases → automated rollback before full rollout.


39. How do you implement SLO-based alerting in Prometheus?

# 1. Define SLI: ratio of successful requests
- record: job:http_request_success_rate:rate5m
  expr: |
    sum(rate(http_requests_total{status!~"5.."}[5m])) by (job)
    /
    sum(rate(http_requests_total[5m])) by (job)

# 2. SLO = 99.9% — error budget = 0.1%
# 3. Alert on burn rate (14.4x = budget exhausted in 2 hours)
- alert: HighErrorBudgetBurn
  expr: |
    (1 - job:http_request_success_rate:rate5m) > (14.4 * 0.001)
    and
    (1 - job:http_request_success_rate:rate1h) > (14.4 * 0.001)
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Error budget burning at 14.4x rate"
    runbook: "https://runbooks.company.com/high-error-rate"

Multi-window approach catches both fast burns (short window) and sustained slow burns (long window).


40. What is the CAP theorem and how does it affect reliability design?

CAP theorem: a distributed system can guarantee only 2 of 3:

Property Definition
Consistency Every read gets the most recent write
Availability Every request gets a response (may be stale)
Partition tolerance System works despite network partitions

Since network partitions are unavoidable, real systems choose between CP or AP.

Choice Examples SRE implication
CP (consistent) ZooKeeper, HBase, etcd Returns error during partition (availability affected)
AP (available) Cassandra, CouchDB, DynamoDB Returns stale data during partition

SRE application: knowing your database's consistency model affects:

  • How you handle read-after-write consistency
  • Whether you can read from replicas
  • How you recover from split-brain scenarios

41. Explain database connection pooling and why SREs care.

Connection pooling reuses existing DB connections instead of creating a new one per request.

Without pooling:
  Each request → new TCP connection → TLS handshake → auth → query → close
  Time: 10-100ms overhead per request

With pooling:
  Connection pool: 20 connections kept open
  Request → borrow connection → query → return connection
  Time: < 1ms overhead

SRE relevance:

  • PostgreSQL max connections: 100-500 (default 100)
  • With 50 pods × 10 connections each = 500 connections (hits limit)
  • Solution: PgBouncer (connection pooler) in front of PostgreSQL
# Signs of connection exhaustion
ERROR: remaining connection slots are reserved for non-replication superuser connections
ERROR: too many connections

# Monitor: connections used vs max
SELECT count(*), max_conn FROM pg_stat_activity, (SELECT setting::int AS max_conn FROM pg_settings WHERE name = 'max_connections') AS mc GROUP BY max_conn;

42. What is graceful degradation? Give examples.

Graceful degradation means the system continues to provide reduced functionality when components fail, rather than a complete outage.

Failure Without degradation With degradation
Recommendation service down Product page 500s Product page loads, no recommendations
Search index stale Users see no results Users see results from 5-min-old index
Auth service slow All users wait 30s Cached auth tokens still work for 5 min
Image CDN down Page with broken images Fallback to origin server
Payment provider A down Checkout fails Automatically try payment provider B

Implementation tools: feature flags, circuit breakers, cached fallbacks, static default responses.


SRE Soft Skills

43. How do you communicate an outage to stakeholders?

During the incident:

Subject: [ONGOING] Checkout Service Degradation — SEV-2

Status: INVESTIGATING
Impact: ~30% of checkout attempts failing since 14:32 UTC
Affected: EU region only, US unaffected
ETR (estimated time to resolution): 15:30 UTC

We have identified the cause (database connection exhaustion after
deployment 2.7.3) and are rolling back now.

Next update: 15:15 UTC or sooner if resolved.

After resolution:

Subject: [RESOLVED] Checkout Service — Back to Normal

Resolved at 15:12 UTC (40 min duration)
Peak impact: 35% error rate
Users affected: ~8,400 checkout attempts
Root cause: DB connection pool exhaustion after config change in deploy 2.7.3
Full postmortem: [link, published within 48h]

Principles: factual, no blame, clear ETR, regular updates, don't go dark.


44. How do you push back when product asks for a feature that would compromise reliability?

Use the error budget as a neutral arbitrator:

"Our current error budget consumption is 85% for this month.
That means we have 6 minutes of downtime left before we breach SLO.

The proposed change touches the payment flow and has a 20% chance
of causing a 30-minute degradation based on staging tests.

This would exhaust our error budget and breach SLA with customers.

Options:
1. Delay launch 2 weeks while we harden the change
2. Launch to 1% of users only, monitor for 48h, then full rollout
3. Launch now and accept SLA breach risk (need VP approval)

Which would you like to proceed with?"

The error budget depersonalizes the conversation. You're not saying "no" — you're showing the math.


45. How do you measure SRE team effectiveness?

Key metrics:

Metric Target Measurement
SLO compliance > 99% of SLOs met Monthly SLO review
Toil ratio < 50% of SRE time Weekly time tracking
MTTR Trending down Incident tracker
Postmortem completion 100% of SEV-1/2 Postmortem tracker
Action item closure > 80% on time JIRA/Linear
On-call load < 2 pages/shift PagerDuty metrics
Change failure rate < 5% Deployment tracker

Anti-metrics (don't optimize for these alone):

  • Number of postmortems (more postmortems = more incidents)
  • Uptime percentage alone (masks slow degradation)
  • Alert count (more alerts ≠ better monitoring)

More Questions

46. What is canary analysis and how is it automated?

Canary analysis automatically compares metrics between a canary deployment and the baseline (production).

Process:

  1. Deploy new version to 5% of traffic (canary)
  2. Compare SLIs: error rate, latency p99, success rate
  3. Statistical test: is canary significantly worse than baseline?
  4. If pass → promote canary to 100%
  5. If fail → automatic rollback

Tools: Spinnaker (Kayenta), Flagger (Kubernetes), Argo Rollouts

# Flagger canary example
apiVersion: flagger.app/v1beta1
kind: Canary
spec:
  analysis:
    interval: 1m
    threshold: 5      # max failed checks
    maxWeight: 50     # max traffic to canary
    stepWeight: 10    # increment per check
    metrics:
    - name: request-success-rate
      thresholdRange:
        min: 99
    - name: request-duration
      thresholdRange:
        max: 500      # max p99 ms

47. How does DNS affect availability and what can go wrong?

DNS translates domain names to IP addresses. DNS failures are invisible to users but catastrophic.

Common DNS problems:

Problem Cause Fix
TTL too high Old IP cached, can't cut over quickly Set TTL to 60s before planned changes
TTL too low DNS resolver overwhelmed, latency spikes Keep at 300s normally
Single DNS provider Provider outage = full outage Use secondary DNS (AWS Route53 + Cloudflare)
No health checks DNS points to dead server Use DNS health checks (Route53 health check)
Long propagation Change takes hours to propagate Plan changes in advance, pre-lower TTL

SRE rule: Lower DNS TTL to 60s 24h before any planned IP change. Raise back to 300s after.


48. What is a service mesh and when does it help?

A service mesh (Istio, Linkerd) adds a sidecar proxy to every pod, handling:

  • mTLS: automatic mutual TLS between services
  • Traffic management: canary, retries, circuit breakers via config
  • Observability: automatic metrics, traces, logs for every service call

When to use:

  • 10+ microservices (managing service-to-service config manually becomes untenable)
  • Need mTLS for compliance (zero-trust networking)
  • Want consistent retry/circuit breaker behavior across all services

Cost: ~10ms latency overhead per call, significant CPU for sidecar proxies.

For small teams or monoliths: not worth the complexity. For large microservices orgs: high value.


49. What is the difference between proactive and reactive SRE work?

Aspect Reactive Proactive
Trigger Alert fires, incident occurs Scheduled review, risk assessment
Examples Incident response, on-call pages Capacity planning, chaos testing, reliability reviews
Value Immediate (stop the bleeding) Long-term (prevent bleeding)
SRE time target < 50% of SRE time > 50% of SRE time

The reactive trap: if SRE teams spend > 50% of time on reactive work (toil, incidents), they never improve the system. The system stays fragile, incidents stay frequent — a self-reinforcing cycle.

Breaking out requires: automate toil → reduce incidents → free time for proactive work → fewer incidents.


50. What would you do on your first 30 days as an SRE?

Week 1: Learn

  • Shadow every on-call shift
  • Read the last 10 postmortems
  • Map all services and dependencies
  • Set up local development environment

Week 2: Observe

  • Review existing SLOs — are they meaningful?
  • Audit alert rules — which are noisy/actionable?
  • Review runbooks — are they accurate and complete?
  • Identify the three most painful recurring issues

Week 3: Small improvements

  • Fix one runbook that was wrong
  • Write one automation for a recurring toil task
  • Shadow an incident commander

Week 4: Propose

  • Identify the highest-impact reliability improvement
  • Present findings to the team with data
  • Own a piece of the on-call rotation

Goal: understand before changing. Learn the culture and existing systems. Earn trust by fixing small things reliably before proposing big changes.


SRE common mistakes

Mistake Why it hurts Fix
Setting SLOs without measuring current baselines SLO is guesswork, immediately violated Measure 30 days of history first
Alerting on every metric Alert fatigue, engineers ignore pages Alert only on SLO burn rate + SEV-1 symptoms
Postmortems with blame People hide incidents, culture suffers Enforce blameless language, lead by example
No error budget policy Teams ignore SLOs — no consequences Write and enforce error budget policy
Automating before understanding Automate the wrong thing Run manually 3 times, then automate
100% availability as target No budget for testing, innovation stops Find the reliability level users actually need
On-call without runbooks MTTR is whatever engineer remembers at 3 AM Every alert must have a runbook
Treating toil as unavoidable Toil grows until SRE team collapses Track toil, cap at 50%, automate aggressively

SRE vs related roles

Role Focus Overlap with SRE
DevOps Engineer CI/CD, delivery pipelines High — SRE is a DevOps implementation
Platform Engineer Internal developer platform (IDP) High — reliability of the platform
Cloud Engineer Cloud infrastructure provisioning Medium — SRE uses cloud, doesn't design it
Software Engineer Feature development Medium — SREs write substantial automation code
Network Engineer Network design, routing Low — SRE uses networking, doesn't configure it
Security Engineer Vulnerability, compliance Low — SRE handles secrets and RBAC

Frequently asked questions

Q: Do I need to know how to code for SRE? Yes. SREs write substantial automation, tooling, and observability code. Python, Go, and Bash are the most common. You don't need to be as strong as a software engineer, but you must be able to write and maintain scripts and services.

Q: What's more important: deep Linux knowledge or Kubernetes? Linux first. Kubernetes is built on Linux primitives (cgroups, namespaces, overlay filesystems). Understanding Linux makes Kubernetes debugging much easier. Employers expect strong Linux fundamentals.

Q: How do SRE interviews differ from software engineer interviews? SRE interviews add: system design for reliability (not just scalability), incident response scenarios, Linux/infrastructure debugging, and on-call philosophy questions. Coding rounds are usually present but weighted lower than for SWE roles.

Q: What is the difference between SRE and platform engineering? SRE focuses on the reliability of production services (SLOs, incident response, toil). Platform engineering focuses on building the internal developer platform — CI/CD, self-service infra, developer experience. They overlap significantly and many orgs combine them.

Q: How do I prepare for SRE interviews at Google specifically? Read the Google SRE book (free at sre.google). Study the chapters on SLOs, error budgets, and toil. Practice Linux debugging. Practice system design with reliability focus. Expect scenario questions like "the database is running slow — walk me through your investigation."

Q: What SRE books should I read?

  1. Site Reliability Engineering (Google, 2016) — the foundational text
  2. The Site Reliability Workbook (Google, 2018) — practical implementation
  3. Seeking SRE (2018) — SRE beyond Google
  4. Database Reliability Engineering (Campbell & Majors, 2017) — database SRE focus

Keep reading

All Toolmingotools are free & run in your browser

No sign-up, no upload, no watermark. Your files never leave your device.

Browse all tools