Site Reliability Engineer interviews test your understanding of reliability principles, SLOs/SLIs/SLAs, error budgets, incident response, system design, and production operations. This guide covers the 50 most common SRE interview questions — with concise answers, formulas, and real examples used at Google, Amazon, Netflix, and other top engineering orgs.
Quick reference
| Topic | Most asked questions |
|---|---|
| Core SRE | SLO/SLI/SLA definitions, error budget, DevOps vs SRE |
| Reliability | Four nines math, toil, MTTR/MTTF/MTBF |
| Incident response | Severity levels, postmortems, on-call runbooks |
| Monitoring | RED/USE methods, alerting philosophy, SLO-based alerts |
| Capacity planning | Load testing, headroom, scaling strategies |
| Chaos engineering | Game days, failure injection, blast radius |
| Automation | Toil elimination, runbook automation, self-healing |
| System design | Load balancers, caching, retries, circuit breakers |
| Kubernetes/Linux | Resource limits, HPA, node affinity, debugging |
| Soft skills | Blameless culture, stakeholder communication |
Core SRE Concepts
1. What is Site Reliability Engineering and how does it differ from DevOps?
| Aspect | SRE | DevOps |
|---|---|---|
| Origin | Google (2003, Ben Treynor Sloss) | Cultural movement (~2009) |
| Focus | Reliability via software engineering | Collaboration + delivery speed |
| Metrics | SLO/SLI/error budget | Deployment frequency, lead time |
| Approach | Prescriptive (Google SRE book) | Principles-based |
| On-call | Dedicated SRE team owns production | Shared model (you build it, you run it) |
| Toil | Explicitly measured and capped (< 50%) | Not formally defined |
"SRE is what you get when you treat operations as a software problem." — Ben Treynor Sloss
SRE is a concrete implementation of DevOps with explicit reliability contracts.
2. Define SLI, SLO, and SLA. Give examples for each.
| Term | Full Name | What it is | Example |
|---|---|---|---|
| SLI | Service Level Indicator | Quantitative metric measuring service behavior | 99.7% of requests complete in < 300 ms |
| SLO | Service Level Objective | Internal target for an SLI | 99.9% availability over 30 days |
| SLA | Service Level Agreement | External contract with consequences for breach | 99.5% uptime or 10% credit |
Relationship:
- SLI is the measurement
- SLO is the goal (always stricter than the SLA)
- SLA is the promise to customers (legal/commercial)
Good SLIs are:
- Directly user-visible (availability, latency, error rate, throughput)
- Measurable with high fidelity
- Not proxy metrics (CPU usage is a bad SLI)
3. What is an error budget and how do you use it?
An error budget is the allowed amount of unreliability derived from the SLO:
Error budget = 1 − SLO target
99.9% SLO → 0.1% error budget = 43.8 minutes/month downtime allowed
99.99% SLO → 0.01% error budget = 4.38 minutes/month
How teams use error budgets:
| Budget remaining | Action |
|---|---|
| > 50% | Ship features aggressively, run chaos experiments |
| 10–50% | Normal pace, watch reliability trends |
| < 10% | Slow feature work, focus on reliability |
| Exhausted | Feature freeze, reliability work only |
Error budgets create a shared incentive: developers want to ship features, SREs want reliability. The budget lets both sides negotiate using data instead of opinion.
4. Calculate the error budget for 99.9% and 99.99% SLOs over 30 days.
30 days = 30 × 24 × 60 = 43,200 minutes
99.9% SLO:
Error budget = 0.1% of 43,200 = 43.2 minutes/month
99.99% SLO:
Error budget = 0.01% of 43,200 = 4.32 minutes/month
99.999% SLO ("five nines"):
Error budget = 0.001% of 43,200 = 0.432 minutes ≈ 26 seconds/month
5. What is toil and why does Google cap it at 50%?
Toil = manual, repetitive, automatable, tactical work that scales with service growth and has no enduring value.
Characteristics of toil:
- Manual: requires human action (e.g., restarting a service)
- Repetitive: done again and again
- Automatable: a machine could do it
- Reactive: triggered by an alert, not a project
- No enduring value: doesn't improve the system
Google caps SRE toil at < 50% of time. Above that:
- SREs become ops engineers with no time to improve the system
- Systems never get more reliable
- Engineers burn out and leave
The other 50%+ goes to engineering work: automation, observability improvements, architectural changes.
6. What is the difference between MTTR, MTTF, and MTBF?
| Metric | Full Name | Formula | Meaning |
|---|---|---|---|
| MTTF | Mean Time To Failure | Total uptime / number of failures | How long system runs before failing |
| MTTR | Mean Time To Recovery | Total downtime / number of incidents | How fast you recover from failure |
| MTBF | Mean Time Between Failures | MTTF + MTTR | Average time between start of one failure and next |
MTBF = MTTF + MTTR
Availability ≈ MTTF / (MTTF + MTTR) = MTTF / MTBF
SRE focuses heavily on reducing MTTR because:
- Failures are inevitable (MTTF has limits)
- Fast recovery keeps error budget intact
- Good observability, runbooks, and on-call practices reduce MTTR
7. What makes a good SLO?
A good SLO is:
- User-centric — measures what users experience, not internal system metrics
- Achievable — based on historical performance, not aspirational
- Specific — includes measurement window, SLI definition, and target
- Actionable — breaching it triggers meaningful action
- Minimal — 2-4 SLOs per service (too many dilutes focus)
Bad SLO: "The system should be fast" Good SLO: "99.9% of checkout API requests complete in < 500ms, measured over a rolling 28-day window"
8. How do you handle an SLO that's consistently at 100%?
A service at 100% SLI is a signal that your SLO is too loose. This means:
- You're over-investing in reliability beyond what users need
- Your error budget is never consumed (so you can't run experiments)
- You may be masking real problems with conservative alerting
Actions:
- Tighten the SLO to match actual user expectations
- Use freed engineering time on feature development
- Run chaos experiments to understand real failure modes
Incident Response
9. Walk me through your incident response process.
A standard incident response process:
Detection → Triage → Mitigation → Resolution → Postmortem
Step by step:
- Detection: Alert fires (or user report) — assign severity
- Incident Commander (IC): One person drives, avoids multitasking
- Communication Lead: Updates status page, notifies stakeholders
- Triage: Understand scope — who/what is affected, since when
- Mitigation: Restore service ASAP (rollback > root cause fix)
- Resolution: Confirmed fix, verify SLI recovery
- Postmortem: Written within 48 hours, blameless, with action items
Key principle: Mitigate first, diagnose later. A slow rollback beats a fast root-cause hunt.
10. What are severity levels and how do you define them?
| Severity | Definition | Example | Response time |
|---|---|---|---|
| SEV-1 | Complete outage, all users affected | Checkout down | Immediate, 24/7 |
| SEV-2 | Major functionality degraded | 50% error rate on search | < 15 min |
| SEV-3 | Partial degradation, workaround exists | Slow dashboard loads | Business hours |
| SEV-4 | Minor issue, no user impact | Log noise, cosmetic bug | Next sprint |
Definitions vary by company. What matters: severity drives escalation, not blame.
11. What is a blameless postmortem and what should it contain?
A blameless postmortem assumes that people made decisions with the best information available at the time. It focuses on systems, not individuals.
Structure:
## Incident summary
Date, duration, severity, services affected
## Timeline
[Time] Event (what happened, who noticed, what was done)
## Root cause
The technical condition that caused the incident
## Contributing factors
Conditions that made the incident possible or worse
## Resolution
What stopped the incident
## Impact
Users affected, error budget consumed, revenue lost
## Action items
| Action | Owner | Priority | Due date |
|--------|-------|----------|----------|
## Lessons learned
What went well, what went poorly, where we got lucky
Avoid blame language: "John made a mistake" → "The deploy process lacked a safeguard that would have caught this"
12. How do you handle alert fatigue?
Alert fatigue = too many alerts, engineers stop responding to them.
Causes:
- Alerts not tied to SLOs (alerting on symptoms, not user impact)
- Alerts without clear action (what do I do when this fires?)
- Flapping alerts (fires/resolves repeatedly)
Solutions:
| Problem | Fix |
|---|---|
| Too many low-value alerts | Audit — if no action taken in 6 months, delete |
| Alerts not actionable | Require runbook link in every alert |
| Flapping | Add for: 5m delay, or re-evaluate threshold |
| Not SLO-linked | Replace with burn rate alerts |
| Wrong severity | Re-triage — not every alert is SEV-2 |
Goal: every alert should be urgent, actionable, and represent user impact.
13. What is an SLO burn rate alert?
Instead of alerting on individual metrics, a burn rate alert fires when you're consuming your error budget too fast.
Burn rate = (error rate) / (1 − SLO)
Example: SLO = 99.9%, current error rate = 1%
Burn rate = 1% / 0.1% = 10x
At 10x burn, you exhaust your monthly budget in 3 days.
Google's multi-window approach:
| Window | Burn rate | Alert type |
|---|---|---|
| 1h + 5m | 14.4x | Critical page |
| 6h + 30m | 6x | Critical page |
| 3d + 6h | 1x | Ticket |
Fast burn = fast window (catches spikes). Slow burn = long window (catches sustained issues). Both needed to avoid missing incidents.
14. How do you write a good runbook?
A runbook documents how to respond to a specific alert or operational task.
Good runbook structure:
## Alert: HighCheckoutErrorRate
### What does this mean?
Checkout error rate > 1% for 5+ minutes. Users cannot complete purchases.
### Severity
SEV-1 if > 5%, SEV-2 if 1–5%
### Immediate checks
1. Check Grafana dashboard: [link]
2. Recent deployments: `kubectl rollout history deployment/checkout`
3. Database health: [link to DB dashboard]
### Common causes + fixes
| Symptom | Likely cause | Fix |
|---------|-------------|-----|
| DB errors in logs | DB connection pool exhausted | Restart app pods: `kubectl rollout restart deployment/checkout` |
| 502s from load balancer | Pod OOM killed | Scale up: `kubectl scale deployment/checkout --replicas=10` |
| Auth errors | JWT signing key rotated | Verify secret: `kubectl get secret jwt-key` |
### Escalation
If not resolved in 15 min: page backend oncall + notify #incidents
### Rollback
`kubectl rollout undo deployment/checkout`
Monitoring & Observability
15. Explain the RED and USE methods.
RED Method (for services/microservices):
- Rate — requests per second
- Errors — failed requests per second
- Duration — latency distribution (p50, p95, p99)
USE Method (for resources: CPU, memory, disk, network):
- Utilization — % of time resource is busy
- Saturation — how much work is queued/waiting
- Errors — error events for that resource
| Method | Best for | Examples |
|---|---|---|
| RED | APIs, services | HTTP endpoints, gRPC, queues |
| USE | Infrastructure | CPU, disk I/O, network interfaces |
Use RED to detect user-facing problems. Use USE to find the infrastructure bottleneck causing them.
16. What are the four golden signals?
Google's four golden signals (from the SRE book):
| Signal | What it measures | Alert on? |
|---|---|---|
| Latency | Time to serve a request (successful vs failed separately) | p99 exceeds threshold |
| Traffic | Demand on the system (RPS, queries/sec) | Unusual spikes (DDoS, viral) |
| Errors | Rate of failed requests (explicit 5xx, implicit wrong results) | SLO burn rate |
| Saturation | Fullness of resource (CPU %, disk %, queue depth) | Headroom < 20% |
These four cover most production problems. If your system measures all four, you'll catch nearly every incident.
17. What is distributed tracing and when do you use it?
Distributed tracing tracks a request as it flows through multiple services by propagating a trace ID through all service calls.
When to use:
- Debugging latency in microservices (which service is slow?)
- Finding where errors originate in a request chain
- Understanding service dependencies
Tools: Jaeger, Zipkin, AWS X-Ray, OpenTelemetry
Key concepts:
- Trace: end-to-end journey of a request
- Span: a single operation within a trace (one service call)
- Context propagation: trace ID passed via HTTP headers (
traceparent)
User → API Gateway (span 1)
→ Auth Service (span 2)
→ Product Service (span 3)
→ Database (span 4)
→ Payment Service (span 5)
Without tracing, debugging a 3-second request across 5 services is guesswork.
18. Prometheus vs Datadog vs CloudWatch — when do you use each?
| Tool | Type | Best for | Cost |
|---|---|---|---|
| Prometheus | Open source, pull-based | Kubernetes-native, metrics only | Free (infra cost only) |
| Grafana | Open source visualization | Dashboards on top of Prometheus | Free |
| Datadog | SaaS | Full observability (metrics + traces + logs), easy onboarding | Expensive per host |
| CloudWatch | AWS-native SaaS | AWS resources, Lambda, EC2 | Pay per metric/log |
| New Relic | SaaS | Full-stack observability, APM | Per core/user |
Typical stack: Prometheus + Grafana for metrics, OpenTelemetry for traces, Loki for logs (all open source). Or Datadog for teams that want one tool.
Capacity Planning
19. How do you approach capacity planning?
Capacity planning ensures enough resources exist to handle current and future load without over-provisioning.
Process:
- Baseline: measure current resource utilization (CPU, memory, disk, network)
- Model growth: project traffic growth (historical trend + product roadmap)
- Load test: find the breaking point under synthetic load
- Headroom target: keep utilization below 70% (30% headroom for spikes)
- Alert on headroom: alert when headroom < 20%
- Review quarterly: adjust for actual vs projected growth
Example:
Current: 100 RPS, 60% CPU on 10 servers
Projected: 2x traffic in 6 months = 200 RPS
At 200 RPS → 120% CPU (would fail)
Need: 20 servers by month 6 (or scale-out automation)
20. What is the difference between horizontal and vertical scaling?
| Aspect | Horizontal scaling (scale out) | Vertical scaling (scale up) |
|---|---|---|
| What | Add more instances | Add more CPU/RAM to existing instance |
| Limit | Theoretically unlimited | Hardware ceiling |
| Downtime | Zero (rolling adds) | Often requires restart |
| Cost | Linear, pay-as-you-go | Diminishing returns |
| Complexity | Needs load balancer, stateless design | Simpler |
| SRE preference | Preferred for web services | Used for DBs (read replicas, then vertical) |
Stateless services scale horizontally. Stateful services (databases) are harder — use read replicas, sharding, or managed services.
21. How do you handle a traffic spike that exceeds capacity?
Before the spike (defense in depth):
- Auto-scaling configured (HPA in Kubernetes, AWS Auto Scaling)
- Load shedding: return 429 at threshold instead of cascading failure
- Circuit breakers: stop calling overloaded downstream services
- Caching: CDN, Redis to reduce backend load
- Rate limiting: per-user/IP limits
During the spike:
- Check auto-scaler: is it scaling? Why not?
- Identify the bottleneck (RED metrics: which service is saturated?)
- Shed traffic if needed (feature flag, CDN rules)
- Scale manually if auto-scaler is slow
After the spike:
- Postmortem: was it expected? Why didn't auto-scaler handle it?
- Adjust scaling policies
Chaos Engineering
22. What is chaos engineering?
Chaos engineering is the practice of deliberately injecting failures into production (or staging) to discover weaknesses before they cause real incidents.
Principles:
- Hypothesize about steady-state behavior
- Vary real-world events (server crash, latency spike, dependency failure)
- Run experiments in production (with blast radius control)
- Automate continuously
Netflix Simian Army examples:
- Chaos Monkey: randomly kills production instances
- Chaos Kong: simulates entire AWS region failure
- Latency Monkey: injects artificial latency
Before running chaos experiments:
- Have good observability (you can't measure what broke)
- Have an SLO to measure impact against
- Start small: test in staging first, then production with small % traffic
23. What is a game day?
A game day is a planned exercise where the team simulates a production failure scenario to test response capabilities.
Format:
- Define a failure scenario (e.g., "primary database fails")
- Inject the failure (or simulate it with a runbook)
- Team responds as if it's a real incident (incident commander, comms, runbook)
- Measure: detection time, MTTR, runbook accuracy
- Debrief: what worked, what broke, what needs improvement
Game days improve:
- Runbook accuracy ("did the runbook actually work?")
- Team confidence and muscle memory
- On-call readiness
- Discovery of unknown failure modes
24. How do you control the blast radius of a chaos experiment?
Techniques:
| Technique | Description |
|---|---|
| Traffic percentage | Kill 1% of instances, not all |
| Feature flags | Enable chaos only for % of users |
| Start in staging | Validate experiment before production |
| Time-boxing | Run for 10 minutes, then stop |
| Auto-abort | If SLI drops below threshold, stop automatically |
| Read-only first | Inject latency before killing services |
The goal is to learn without causing a real incident. Start with the smallest failure that teaches you something useful.
Automation & Toil Reduction
25. Give an example of toil you've eliminated through automation.
Example — manual certificate renewal:
Before: Engineer manually ran certbot renew on 20 servers each quarter, checked expiry dates via calendar reminder, 4 hours of toil per quarter.
After: Cert-manager in Kubernetes auto-renews Let's Encrypt certs 30 days before expiry. Prometheus alert fires if any cert expires within 14 days as a safety net. Toil eliminated: 4 hours/quarter → 0.
Framework for toil elimination:
- Measure: how long does this take per occurrence?
- Frequency: how often does it happen?
- Annual cost: time × frequency × number of engineers
- Automation cost: how long to automate?
- Break-even: payback period
If break-even < 1 year, automate it.
26. What is self-healing infrastructure?
Self-healing infrastructure detects and automatically recovers from failure without human intervention.
Examples:
| Failure | Self-healing mechanism |
|---|---|
| Pod OOM killed | Kubernetes restarts pod (restartPolicy: Always) |
| Unhealthy instance | Auto Scaling replaces EC2 instance |
| DB connection exhausted | Connection pool health check + restart |
| Zombie process | Systemd Restart=on-failure |
| Disk full | Automated log rotation, cleanup cron |
| Certificate expired | cert-manager auto-renews |
Limits of self-healing:
- Can mask underlying problems (pod keeps restarting = OOM loop)
- Need alerting on restart count (alert if restarts > 5 in 1 hour)
- Self-healing is last-resort; fix root cause in parallel
System Design for Reliability
27. What is a circuit breaker pattern?
A circuit breaker prevents a service from repeatedly calling a failing dependency, allowing the dependency time to recover.
States:
CLOSED (normal) → failure threshold exceeded →
OPEN (reject calls) → timeout period elapsed →
HALF-OPEN (test one call) → success → CLOSED
→ failure → OPEN
Benefits:
- Prevents cascading failures
- Reduces load on struggling service
- Fast failure (immediate 503) instead of timeout cascade
# Pseudocode
if circuit_breaker.is_open():
return fallback_response() # don't call dependency
try:
result = call_dependency()
circuit_breaker.record_success()
return result
except Exception:
circuit_breaker.record_failure()
raise
Libraries: Resilience4j (Java), Polly (.NET), pybreaker (Python)
28. How do you design a service for 99.99% availability?
99.99% = 52.6 minutes downtime/year.
Architecture requirements:
| Layer | Requirement |
|---|---|
| No single point of failure | Every component has ≥ 2 instances |
| Multi-AZ | Span at least 2 availability zones |
| Auto-scaling | Handle 3× peak traffic |
| Health checks | Remove unhealthy instances in < 30s |
| Rolling deploys | Zero-downtime deployments |
| Graceful shutdown | Drain connections before terminating |
| Circuit breakers | Fail fast on dependency failure |
| Retry with backoff | Retry transient failures; don't retry 4xx |
| Caching | CDN + application-level cache reduce DB pressure |
| DB: read replicas | Separate read and write load |
| DB: connection pooling | PgBouncer, HikariCP |
99.99% is expensive. Validate that users actually need it (most don't).
29. Explain retry with exponential backoff and jitter.
Retry logic handles transient failures. Without backoff, retries cause thundering herd (all clients retry simultaneously, overwhelming the server).
import random
import time
def retry_with_backoff(func, max_retries=5, base_delay=1.0):
for attempt in range(max_retries):
try:
return func()
except TransientError as e:
if attempt == max_retries - 1:
raise
# Exponential backoff with full jitter
delay = base_delay * (2 ** attempt)
jitter = random.uniform(0, delay)
time.sleep(jitter)
# Delays with jitter: ~0.5s, ~1s, ~2s, ~4s, ~8s (randomized)
Jitter (random delay) spreads retries across time, preventing synchronized retry storms.
Don't retry:
- 4xx errors (client errors — retry won't fix them)
- Operations that aren't idempotent (non-idempotent POST)
- After exhausting budget (use dead letter queue)
30. What is the difference between a hard dependency and a soft dependency?
| Type | Definition | Failure behavior |
|---|---|---|
| Hard dependency | Service cannot function without it | Return error if unavailable |
| Soft dependency | Service degrades gracefully without it | Return partial result or default |
Example: E-commerce product page
- Database: hard (can't show products without it)
- Recommendation engine: soft (show page without "You may also like")
- Analytics tracking: soft (user doesn't see it fail)
- Review service: soft (show product without reviews)
SRE principle: make as few things hard dependencies as possible. Each hard dependency directly affects your SLO.
Kubernetes & Linux Operations
31. A pod keeps crashing. How do you debug it?
# 1. Check pod status
kubectl get pod <pod-name> -n <namespace>
# 2. Describe pod (events, resource limits, last termination reason)
kubectl describe pod <pod-name> -n <namespace>
# 3. View current logs
kubectl logs <pod-name> -n <namespace>
# 4. View logs from previous (crashed) container
kubectl logs <pod-name> -n <namespace> --previous
# 5. If OOMKilled: check memory limits vs usage
kubectl top pod <pod-name> -n <namespace>
# 6. Exec into running pod (if it's not crash-looping)
kubectl exec -it <pod-name> -n <namespace> -- /bin/sh
Common crash causes:
| Exit code / status | Cause | Fix |
|---|---|---|
| OOMKilled | Memory limit too low | Increase resources.limits.memory |
| CrashLoopBackOff | App crashes on start | Check logs for startup error |
| Error 1 | Application-level error | Check app logs |
| ImagePullBackOff | Can't pull image | Check image name, registry credentials |
| Pending | No nodes with capacity | Check node resources, taints |
32. How does Kubernetes HPA work?
Horizontal Pod Autoscaler (HPA) scales replica count based on metrics.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: api-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: api
minReplicas: 2
maxReplicas: 20
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70 # target 70% CPU
- type: Pods
pods:
metric:
name: requests_per_second
target:
type: AverageValue
averageValue: "1000"
HPA algorithm:
desired replicas = ceil(current replicas × (current metric / desired metric))
SRE considerations:
- Set
minReplicas ≥ 2(HA) - Ensure requests/limits set (HPA requires them for CPU/memory)
- Scale-down is slow by default (5 min cooldown) — good for stability
33. How do you debug high CPU on a Linux server?
# 1. Which process is consuming CPU?
top -c # or htop
# 2. Detailed CPU breakdown (user/system/iowait/steal)
vmstat 1 10
# 3. Per-CPU stats (is it one core or all?)
mpstat -P ALL 1
# 4. What is a specific process doing?
pidstat -u -p <PID> 1
# 5. System calls being made (what is the process actually doing?)
strace -p <PID> -c
# 6. If iowait is high (not CPU-bound, but waiting for disk)
iostat -x 1
iotop
# 7. Check if it's a kernel thread
ps aux --sort=-%cpu | head -20
Common high-CPU causes:
- Infinite loop in application
- Garbage collection (Java/Go)
- High request volume
- Kernel: high iowait = disk bottleneck, not CPU
34. Explain Linux file descriptors and why "too many open files" errors occur.
Every open file, socket, pipe, and device is a file descriptor (fd).
# Current limits
ulimit -n # soft limit for current process
cat /proc/sys/fs/file-max # system-wide maximum
# How many fds is a process using?
ls /proc/<PID>/fd | wc -l
# What files are open?
lsof -p <PID>
# System-wide fd usage
lsof | wc -l
# Increase limit for current session
ulimit -n 65536
# Permanent: /etc/security/limits.conf
* soft nofile 65536
* hard nofile 65536
Common cause: Connection leak — application opens DB connections, HTTP connections, or files and never closes them. Eventually hits the limit (default: 1024).
On-Call & Culture
35. How do you make on-call sustainable?
Symptoms of unsustainable on-call:
2 pages per on-call shift
- Pages at 2 AM repeatedly
- No time between alerts to investigate
- Engineers dread on-call rotation
Solutions:
| Problem | Fix |
|---|---|
| Too many alerts | Audit and delete non-actionable alerts |
| Repeated same alert | Fix root cause or add auto-remediation |
| 3 AM pages for non-urgent issues | Define SEV levels, page only SEV-1/2 at night |
| Single person on-call | Rotate team, add shadow/backup on-call |
| No runbooks | Require runbook link in every alert |
| Alert fatigue | Weekly on-call review + postmortem for repeat alerts |
Google's benchmark: on-call load should be < 25% of SRE time. Excess pages = technical debt.
36. How do you prioritize what to fix after an incident?
Use the postmortem action items framework:
| Priority | Criteria | Example |
|---|---|---|
| P1 — Do now | Prevents reoccurrence, easy to fix | Add circuit breaker to dependency |
| P2 — This sprint | Reduces MTTR significantly | Write runbook for this alert class |
| P3 — Backlog | Reduces likelihood | Migrate off deprecated library |
| Won't fix | Cost > benefit | Edge case affecting 0.001% of users |
Track action items in JIRA/Linear with an owner, due date, and link to the postmortem. Review completion rate in monthly reliability reviews.
37. What is the principle of defense in depth?
Multiple independent layers of protection so no single failure causes complete disaster.
Applied to SRE:
Layer 1: Code review + testing (prevent bugs entering production)
Layer 2: Canary deployment (affect 1% of traffic before 100%)
Layer 3: Feature flags (disable feature without deploy)
Layer 4: Rate limiting (prevent abuse from taking down service)
Layer 5: Circuit breakers (isolate failing dependency)
Layer 6: Auto-scaling (handle traffic spikes)
Layer 7: Multi-AZ redundancy (survive AZ failure)
Layer 8: Backups + DR (survive data loss)
Layer 9: On-call + runbooks (human last resort)
No single layer is perfect. The combination makes the system robust.
Advanced SRE Topics
38. What is progressive delivery and how does it reduce risk?
Progressive delivery = releasing changes to increasing percentages of users, with automated rollback on SLO breach.
Strategies:
| Strategy | Description | Risk |
|---|---|---|
| Canary | 1% → 5% → 25% → 100% of traffic | Very low |
| Blue/green | Full switch, keep old env running | Medium (full traffic at once) |
| Shadow | Mirror traffic to new version (no user impact) | Zero |
| A/B test | Different versions for different user segments | Low |
| Feature flag | Enable/disable feature without deploy | Very low |
SRE monitors SLIs during canary. If error rate increases → automated rollback before full rollout.
39. How do you implement SLO-based alerting in Prometheus?
# 1. Define SLI: ratio of successful requests
- record: job:http_request_success_rate:rate5m
expr: |
sum(rate(http_requests_total{status!~"5.."}[5m])) by (job)
/
sum(rate(http_requests_total[5m])) by (job)
# 2. SLO = 99.9% — error budget = 0.1%
# 3. Alert on burn rate (14.4x = budget exhausted in 2 hours)
- alert: HighErrorBudgetBurn
expr: |
(1 - job:http_request_success_rate:rate5m) > (14.4 * 0.001)
and
(1 - job:http_request_success_rate:rate1h) > (14.4 * 0.001)
for: 2m
labels:
severity: critical
annotations:
summary: "Error budget burning at 14.4x rate"
runbook: "https://runbooks.company.com/high-error-rate"
Multi-window approach catches both fast burns (short window) and sustained slow burns (long window).
40. What is the CAP theorem and how does it affect reliability design?
CAP theorem: a distributed system can guarantee only 2 of 3:
| Property | Definition |
|---|---|
| Consistency | Every read gets the most recent write |
| Availability | Every request gets a response (may be stale) |
| Partition tolerance | System works despite network partitions |
Since network partitions are unavoidable, real systems choose between CP or AP.
| Choice | Examples | SRE implication |
|---|---|---|
| CP (consistent) | ZooKeeper, HBase, etcd | Returns error during partition (availability affected) |
| AP (available) | Cassandra, CouchDB, DynamoDB | Returns stale data during partition |
SRE application: knowing your database's consistency model affects:
- How you handle read-after-write consistency
- Whether you can read from replicas
- How you recover from split-brain scenarios
41. Explain database connection pooling and why SREs care.
Connection pooling reuses existing DB connections instead of creating a new one per request.
Without pooling:
Each request → new TCP connection → TLS handshake → auth → query → close
Time: 10-100ms overhead per request
With pooling:
Connection pool: 20 connections kept open
Request → borrow connection → query → return connection
Time: < 1ms overhead
SRE relevance:
- PostgreSQL max connections: 100-500 (default 100)
- With 50 pods × 10 connections each = 500 connections (hits limit)
- Solution: PgBouncer (connection pooler) in front of PostgreSQL
# Signs of connection exhaustion
ERROR: remaining connection slots are reserved for non-replication superuser connections
ERROR: too many connections
# Monitor: connections used vs max
SELECT count(*), max_conn FROM pg_stat_activity, (SELECT setting::int AS max_conn FROM pg_settings WHERE name = 'max_connections') AS mc GROUP BY max_conn;
42. What is graceful degradation? Give examples.
Graceful degradation means the system continues to provide reduced functionality when components fail, rather than a complete outage.
| Failure | Without degradation | With degradation |
|---|---|---|
| Recommendation service down | Product page 500s | Product page loads, no recommendations |
| Search index stale | Users see no results | Users see results from 5-min-old index |
| Auth service slow | All users wait 30s | Cached auth tokens still work for 5 min |
| Image CDN down | Page with broken images | Fallback to origin server |
| Payment provider A down | Checkout fails | Automatically try payment provider B |
Implementation tools: feature flags, circuit breakers, cached fallbacks, static default responses.
SRE Soft Skills
43. How do you communicate an outage to stakeholders?
During the incident:
Subject: [ONGOING] Checkout Service Degradation — SEV-2
Status: INVESTIGATING
Impact: ~30% of checkout attempts failing since 14:32 UTC
Affected: EU region only, US unaffected
ETR (estimated time to resolution): 15:30 UTC
We have identified the cause (database connection exhaustion after
deployment 2.7.3) and are rolling back now.
Next update: 15:15 UTC or sooner if resolved.
After resolution:
Subject: [RESOLVED] Checkout Service — Back to Normal
Resolved at 15:12 UTC (40 min duration)
Peak impact: 35% error rate
Users affected: ~8,400 checkout attempts
Root cause: DB connection pool exhaustion after config change in deploy 2.7.3
Full postmortem: [link, published within 48h]
Principles: factual, no blame, clear ETR, regular updates, don't go dark.
44. How do you push back when product asks for a feature that would compromise reliability?
Use the error budget as a neutral arbitrator:
"Our current error budget consumption is 85% for this month.
That means we have 6 minutes of downtime left before we breach SLO.
The proposed change touches the payment flow and has a 20% chance
of causing a 30-minute degradation based on staging tests.
This would exhaust our error budget and breach SLA with customers.
Options:
1. Delay launch 2 weeks while we harden the change
2. Launch to 1% of users only, monitor for 48h, then full rollout
3. Launch now and accept SLA breach risk (need VP approval)
Which would you like to proceed with?"
The error budget depersonalizes the conversation. You're not saying "no" — you're showing the math.
45. How do you measure SRE team effectiveness?
Key metrics:
| Metric | Target | Measurement |
|---|---|---|
| SLO compliance | > 99% of SLOs met | Monthly SLO review |
| Toil ratio | < 50% of SRE time | Weekly time tracking |
| MTTR | Trending down | Incident tracker |
| Postmortem completion | 100% of SEV-1/2 | Postmortem tracker |
| Action item closure | > 80% on time | JIRA/Linear |
| On-call load | < 2 pages/shift | PagerDuty metrics |
| Change failure rate | < 5% | Deployment tracker |
Anti-metrics (don't optimize for these alone):
- Number of postmortems (more postmortems = more incidents)
- Uptime percentage alone (masks slow degradation)
- Alert count (more alerts ≠ better monitoring)
More Questions
46. What is canary analysis and how is it automated?
Canary analysis automatically compares metrics between a canary deployment and the baseline (production).
Process:
- Deploy new version to 5% of traffic (canary)
- Compare SLIs: error rate, latency p99, success rate
- Statistical test: is canary significantly worse than baseline?
- If pass → promote canary to 100%
- If fail → automatic rollback
Tools: Spinnaker (Kayenta), Flagger (Kubernetes), Argo Rollouts
# Flagger canary example
apiVersion: flagger.app/v1beta1
kind: Canary
spec:
analysis:
interval: 1m
threshold: 5 # max failed checks
maxWeight: 50 # max traffic to canary
stepWeight: 10 # increment per check
metrics:
- name: request-success-rate
thresholdRange:
min: 99
- name: request-duration
thresholdRange:
max: 500 # max p99 ms
47. How does DNS affect availability and what can go wrong?
DNS translates domain names to IP addresses. DNS failures are invisible to users but catastrophic.
Common DNS problems:
| Problem | Cause | Fix |
|---|---|---|
| TTL too high | Old IP cached, can't cut over quickly | Set TTL to 60s before planned changes |
| TTL too low | DNS resolver overwhelmed, latency spikes | Keep at 300s normally |
| Single DNS provider | Provider outage = full outage | Use secondary DNS (AWS Route53 + Cloudflare) |
| No health checks | DNS points to dead server | Use DNS health checks (Route53 health check) |
| Long propagation | Change takes hours to propagate | Plan changes in advance, pre-lower TTL |
SRE rule: Lower DNS TTL to 60s 24h before any planned IP change. Raise back to 300s after.
48. What is a service mesh and when does it help?
A service mesh (Istio, Linkerd) adds a sidecar proxy to every pod, handling:
- mTLS: automatic mutual TLS between services
- Traffic management: canary, retries, circuit breakers via config
- Observability: automatic metrics, traces, logs for every service call
When to use:
- 10+ microservices (managing service-to-service config manually becomes untenable)
- Need mTLS for compliance (zero-trust networking)
- Want consistent retry/circuit breaker behavior across all services
Cost: ~10ms latency overhead per call, significant CPU for sidecar proxies.
For small teams or monoliths: not worth the complexity. For large microservices orgs: high value.
49. What is the difference between proactive and reactive SRE work?
| Aspect | Reactive | Proactive |
|---|---|---|
| Trigger | Alert fires, incident occurs | Scheduled review, risk assessment |
| Examples | Incident response, on-call pages | Capacity planning, chaos testing, reliability reviews |
| Value | Immediate (stop the bleeding) | Long-term (prevent bleeding) |
| SRE time target | < 50% of SRE time | > 50% of SRE time |
The reactive trap: if SRE teams spend > 50% of time on reactive work (toil, incidents), they never improve the system. The system stays fragile, incidents stay frequent — a self-reinforcing cycle.
Breaking out requires: automate toil → reduce incidents → free time for proactive work → fewer incidents.
50. What would you do on your first 30 days as an SRE?
Week 1: Learn
- Shadow every on-call shift
- Read the last 10 postmortems
- Map all services and dependencies
- Set up local development environment
Week 2: Observe
- Review existing SLOs — are they meaningful?
- Audit alert rules — which are noisy/actionable?
- Review runbooks — are they accurate and complete?
- Identify the three most painful recurring issues
Week 3: Small improvements
- Fix one runbook that was wrong
- Write one automation for a recurring toil task
- Shadow an incident commander
Week 4: Propose
- Identify the highest-impact reliability improvement
- Present findings to the team with data
- Own a piece of the on-call rotation
Goal: understand before changing. Learn the culture and existing systems. Earn trust by fixing small things reliably before proposing big changes.
SRE common mistakes
| Mistake | Why it hurts | Fix |
|---|---|---|
| Setting SLOs without measuring current baselines | SLO is guesswork, immediately violated | Measure 30 days of history first |
| Alerting on every metric | Alert fatigue, engineers ignore pages | Alert only on SLO burn rate + SEV-1 symptoms |
| Postmortems with blame | People hide incidents, culture suffers | Enforce blameless language, lead by example |
| No error budget policy | Teams ignore SLOs — no consequences | Write and enforce error budget policy |
| Automating before understanding | Automate the wrong thing | Run manually 3 times, then automate |
| 100% availability as target | No budget for testing, innovation stops | Find the reliability level users actually need |
| On-call without runbooks | MTTR is whatever engineer remembers at 3 AM | Every alert must have a runbook |
| Treating toil as unavoidable | Toil grows until SRE team collapses | Track toil, cap at 50%, automate aggressively |
SRE vs related roles
| Role | Focus | Overlap with SRE |
|---|---|---|
| DevOps Engineer | CI/CD, delivery pipelines | High — SRE is a DevOps implementation |
| Platform Engineer | Internal developer platform (IDP) | High — reliability of the platform |
| Cloud Engineer | Cloud infrastructure provisioning | Medium — SRE uses cloud, doesn't design it |
| Software Engineer | Feature development | Medium — SREs write substantial automation code |
| Network Engineer | Network design, routing | Low — SRE uses networking, doesn't configure it |
| Security Engineer | Vulnerability, compliance | Low — SRE handles secrets and RBAC |
Frequently asked questions
Q: Do I need to know how to code for SRE? Yes. SREs write substantial automation, tooling, and observability code. Python, Go, and Bash are the most common. You don't need to be as strong as a software engineer, but you must be able to write and maintain scripts and services.
Q: What's more important: deep Linux knowledge or Kubernetes? Linux first. Kubernetes is built on Linux primitives (cgroups, namespaces, overlay filesystems). Understanding Linux makes Kubernetes debugging much easier. Employers expect strong Linux fundamentals.
Q: How do SRE interviews differ from software engineer interviews? SRE interviews add: system design for reliability (not just scalability), incident response scenarios, Linux/infrastructure debugging, and on-call philosophy questions. Coding rounds are usually present but weighted lower than for SWE roles.
Q: What is the difference between SRE and platform engineering? SRE focuses on the reliability of production services (SLOs, incident response, toil). Platform engineering focuses on building the internal developer platform — CI/CD, self-service infra, developer experience. They overlap significantly and many orgs combine them.
Q: How do I prepare for SRE interviews at Google specifically? Read the Google SRE book (free at sre.google). Study the chapters on SLOs, error budgets, and toil. Practice Linux debugging. Practice system design with reliability focus. Expect scenario questions like "the database is running slow — walk me through your investigation."
Q: What SRE books should I read?
- Site Reliability Engineering (Google, 2016) — the foundational text
- The Site Reliability Workbook (Google, 2018) — practical implementation
- Seeking SRE (2018) — SRE beyond Google
- Database Reliability Engineering (Campbell & Majors, 2017) — database SRE focus