DevOps interviews test Linux fundamentals, container knowledge, CI/CD pipelines, infrastructure as code, cloud platforms, and observability. This guide covers the 50 most common questions — with concise answers and real examples.
Quick reference
| Topic | Most asked questions |
|---|---|
| Core concepts | DevOps vs SRE, CI vs CD, SLA/SLO/SLI |
| Linux | Processes, file permissions, networking, systemd |
| Docker | Images vs containers, Dockerfile, multi-stage builds |
| Kubernetes | Pods, Deployments, Services, HPA, ConfigMaps |
| CI/CD | Pipeline stages, GitHub Actions, deployment strategies |
| IaC | Terraform state, Ansible idempotency, drift detection |
| Cloud | IAM, VPC, auto-scaling, object storage |
| Monitoring | Prometheus, Grafana, log aggregation, alerting |
| Networking | DNS, TCP/IP, load balancers, TLS |
| Security | Secrets management, RBAC, image scanning |
Core Concepts
1. What is DevOps and how does it differ from SRE?
| Aspect | DevOps | SRE |
|---|---|---|
| Origin | Cultural movement | Google engineering discipline |
| Focus | Collaboration + automation | Reliability + error budgets |
| Ownership | Shared between Dev + Ops | SRE team owns production |
| Metrics | Deployment frequency, lead time | SLO/SLI, error budget |
| On-call | Shared model | Dedicated SRE on-call |
DevOps is a culture and set of practices. SRE is a concrete implementation of DevOps principles with a focus on reliability using software engineering.
2. What is the difference between CI, CD (Delivery), and CD (Deployment)?
| Term | Definition | Manual step? |
|---|---|---|
| Continuous Integration | Merge code frequently, run automated tests on every commit | Build/test |
| Continuous Delivery | Automatically build + test + prepare for release; deploy button required | Deploy to prod |
| Continuous Deployment | Every passing commit automatically deploys to production | None |
# GitHub Actions: CI + CD pipeline
name: CI/CD
on:
push:
branches: [main]
jobs:
build-and-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run tests
run: npm ci && npm test
deploy:
needs: build-and-test
runs-on: ubuntu-latest
steps:
- name: Deploy to production
run: ./scripts/deploy.sh
3. What are SLA, SLO, and SLI?
| Term | Definition | Example |
|---|---|---|
| SLA (Service Level Agreement) | Contract with customers about uptime/performance | 99.9% uptime per month |
| SLO (Service Level Objective) | Internal target (tighter than SLA) | 99.95% uptime (internal goal) |
| SLI (Service Level Indicator) | Actual measurement | % of successful HTTP requests |
Error budget = 1 - SLO. At 99.9% SLO, you have 43.8 minutes/month of allowed downtime. When the budget is burned, new features stop and reliability work begins.
4. What are the four DORA metrics?
The DORA (DevOps Research and Assessment) metrics measure software delivery performance:
| Metric | Description | Elite target |
|---|---|---|
| Deployment Frequency | How often code is deployed to production | Multiple times/day |
| Lead Time for Changes | Commit to production time | < 1 hour |
| Change Failure Rate | % of deployments causing failures | < 5% |
| Time to Restore Service | How long to recover from failure | < 1 hour |
5. What is Infrastructure as Code (IaC)?
IaC means managing infrastructure through version-controlled code files instead of manual processes:
- Declarative: Describe desired state (Terraform, Pulumi)
- Imperative: Describe steps to reach state (Ansible in some modes)
Benefits: reproducibility, auditability, version control, code review for infrastructure changes.
Linux
6. How do you find and kill a process in Linux?
# Find process by name
ps aux | grep nginx
pgrep nginx # prints PID(s)
# Kill by PID
kill 1234 # sends SIGTERM (graceful)
kill -9 1234 # sends SIGKILL (force)
# Kill by name
pkill nginx
killall nginx
# Find what's using a port
ss -tlnp | grep :8080
lsof -i :8080
7. What does chmod 755 mean?
Linux permissions use three octet groups: owner, group, others.
| Octet | Permissions | Meaning |
|---|---|---|
| 7 | rwx (4+2+1) | Read, write, execute |
| 5 | r-x (4+0+1) | Read, execute only |
| 5 | r-x (4+0+1) | Read, execute only |
chmod 755 script.sh → owner can do anything, group/others can read and execute (typical for scripts and web server directories).
chmod 755 script.sh # rwxr-xr-x
chmod 644 file.txt # rw-r--r-- (typical file)
chmod 600 ~/.ssh/id_rsa # rw------- (private key — required by SSH)
8. What is the difference between a hard link and a soft (symbolic) link?
| Aspect | Hard link | Symbolic link |
|---|---|---|
| Points to | Inode (data blocks) | Path (another filename) |
| Crosses filesystems | No | Yes |
| Works on directories | No | Yes |
| Original deleted | Link still works | Link becomes dangling |
| Different inode | No (same inode) | Yes (own inode) |
ln original.txt hardlink.txt # hard link
ln -s /path/to/original.txt symlink.txt # symbolic link
9. How does systemd work and what are common commands?
systemd is the init system on most modern Linux distros. It manages services (units) through .service files.
# Service management
systemctl start nginx
systemctl stop nginx
systemctl restart nginx
systemctl reload nginx # reload config without restart
systemctl enable nginx # start on boot
systemctl disable nginx
systemctl status nginx # check status + recent logs
# View logs for a service
journalctl -u nginx -f # follow live
journalctl -u nginx --since "1 hour ago"
# Write a unit file
cat /etc/systemd/system/myapp.service
[Unit]
Description=My Application
After=network.target
[Service]
User=appuser
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/server
Restart=always
RestartSec=5
Environment=NODE_ENV=production
[Install]
WantedBy=multi-user.target
systemctl daemon-reload # reload after editing unit files
10. What is the difference between ssh -L, -R, and -D?
| Flag | Type | Use case |
|---|---|---|
-L local_port:host:remote_port |
Local forward | Access remote service locally |
-R remote_port:host:local_port |
Remote forward | Expose local service via SSH server |
-D port |
Dynamic (SOCKS proxy) | Route browser traffic through SSH |
# Access a DB on remote server locally
ssh -L 5432:localhost:5432 user@server
# Expose local dev server on remote machine
ssh -R 8080:localhost:3000 user@server
# Use remote as SOCKS5 proxy for browser
ssh -D 1080 user@server
Docker
11. What is the difference between a Docker image and a container?
| Concept | Description |
|---|---|
| Image | Read-only layered filesystem snapshot (blueprint) |
| Container | Running instance of an image (writable layer added on top) |
| Registry | Storage for images (Docker Hub, ECR, GCR) |
| Dockerfile | Instructions to build an image |
docker build -t myapp:1.0 . # image
docker run -d -p 8080:80 myapp:1.0 # container
docker ps # list running containers
docker exec -it <id> sh # shell into running container
12. What are Docker layers and why do they matter?
Each Dockerfile instruction creates a new read-only layer. Layers are cached and shared:
# Bad: invalidates cache on every build when code changes
FROM node:20-alpine
COPY . . # cache miss every time
RUN npm ci
# Good: dependencies cached unless package.json changes
FROM node:20-alpine
COPY package*.json ./ # only invalidated when deps change
RUN npm ci
COPY . . # code copied after deps installed
Smaller layers = faster builds, smaller pushes, less storage.
13. What is a multi-stage Docker build?
Multi-stage builds let you use multiple FROM statements to produce a minimal final image:
# Stage 1: build
FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build
# Stage 2: production image (no dev tools, no source code)
FROM node:20-alpine AS production
WORKDIR /app
ENV NODE_ENV=production
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
EXPOSE 3000
CMD ["node", "dist/server.js"]
Result: production image contains only runtime artifacts — often 5–10x smaller.
14. How does Docker networking work?
| Network type | Description | Use case |
|---|---|---|
| bridge (default) | Private internal network on host | Containers on same host |
| host | Container shares host network | High-performance, no isolation |
| none | No networking | Security-sensitive tasks |
| overlay | Multi-host networking | Docker Swarm / Kubernetes |
| macvlan | Container gets its own MAC/IP | Legacy apps expecting physical NIC |
# Create custom bridge network
docker network create mynet
# Containers on same network can reach each other by name
docker run -d --name db --network mynet postgres
docker run -d --name app --network mynet -e DB_HOST=db myapp
15. What is Docker Compose and when do you use it?
Docker Compose defines multi-container applications in a single compose.yaml file:
services:
web:
build: .
ports:
- "3000:3000"
environment:
- DATABASE_URL=postgres://user:pass@db:5432/mydb
depends_on:
db:
condition: service_healthy
restart: unless-stopped
db:
image: postgres:16-alpine
environment:
POSTGRES_USER: user
POSTGRES_PASSWORD: pass
POSTGRES_DB: mydb
volumes:
- pgdata:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U user"]
interval: 5s
retries: 5
volumes:
pgdata:
docker compose up -d # start in background
docker compose logs -f web # follow logs
docker compose down -v # stop + remove volumes
Use Compose for local development and simple single-host deployments. Use Kubernetes for production multi-host orchestration.
Kubernetes
16. What are the main Kubernetes objects?
| Object | Description |
|---|---|
| Pod | Smallest deployable unit; one or more containers |
| Deployment | Manages ReplicaSet; handles rolling updates |
| Service | Stable network endpoint for a set of Pods |
| ConfigMap | Non-secret configuration data |
| Secret | Sensitive data (base64-encoded) |
| Ingress | HTTP routing from outside the cluster |
| PersistentVolumeClaim | Request for durable storage |
| HorizontalPodAutoscaler | Auto-scale Pods based on metrics |
| Namespace | Virtual cluster for resource isolation |
17. What is the difference between a ClusterIP, NodePort, and LoadBalancer Service?
| Type | Accessible from | Use case |
|---|---|---|
| ClusterIP | Inside cluster only | Internal microservices |
| NodePort | <NodeIP>:<30000-32767> |
Dev/testing only |
| LoadBalancer | External via cloud LB | Production ingress (non-Ingress) |
| ExternalName | DNS CNAME alias | Route to external service |
18. How does a rolling update work in Kubernetes?
# Deployment with rolling update strategy
spec:
replicas: 4
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1 # max extra pods during update
maxUnavailable: 0 # no downtime — always 4 pods ready
Kubernetes replaces pods one at a time. New pods must pass readiness probes before old ones are terminated.
kubectl set image deployment/myapp app=myapp:2.0 # trigger update
kubectl rollout status deployment/myapp # watch progress
kubectl rollout undo deployment/myapp # rollback
kubectl rollout history deployment/myapp # view history
19. What are liveness and readiness probes?
| Probe | Question answered | Failure action |
|---|---|---|
| Readiness | Is the pod ready to receive traffic? | Remove from Service endpoints |
| Liveness | Is the pod still alive? | Restart the container |
| Startup | Has the app finished starting? | Kill container if startup takes too long |
containers:
- name: app
readinessProbe:
httpGet:
path: /health/ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
livenessProbe:
httpGet:
path: /health/live
port: 8080
initialDelaySeconds: 15
failureThreshold: 3
20. What is a Kubernetes HPA (HorizontalPodAutoscaler)?
HPA automatically scales the number of Pods based on observed CPU/memory utilization or custom metrics:
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: myapp-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: myapp
minReplicas: 2
maxReplicas: 20
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
kubectl get hpa
kubectl describe hpa myapp-hpa
21. How do you manage secrets in Kubernetes?
Kubernetes Secrets are base64-encoded (not encrypted by default). Best practices:
# Create a secret
kubectl create secret generic db-creds \
--from-literal=username=admin \
--from-literal=password=s3cr3t
# Use in Pod
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: db-creds
key: password
Better approaches for production:
- Enable etcd encryption at rest
- Use External Secrets Operator with AWS Secrets Manager / HashiCorp Vault
- Use Sealed Secrets (encrypt in git, decrypt in cluster)
CI/CD
22. What are common CI/CD pipeline stages?
| Stage | What happens | Tools |
|---|---|---|
| Source | Code commit triggers pipeline | GitHub, GitLab |
| Build | Compile, bundle, create image | Docker, Maven, npm |
| Test | Unit, integration, e2e tests | Jest, pytest, Playwright |
| Security scan | SAST, container scanning, dependency audit | Trivy, Snyk, Semgrep |
| Publish | Push image/artifact to registry | ECR, Docker Hub, Artifactory |
| Deploy (staging) | Deploy to pre-prod environment | Helm, kubectl, Terraform |
| Smoke test | Verify deploy is healthy | curl, Playwright |
| Deploy (production) | Rollout to production | Helm, ArgoCD |
| Notify | Alert on success/failure | Slack, PagerDuty |
23. What is the difference between blue-green and canary deployments?
| Strategy | How it works | Risk | Rollback speed |
|---|---|---|---|
| Recreate | Shut down v1, start v2 | Downtime | Fast |
| Rolling | Gradually replace old pods | Minimal | kubectl rollout undo |
| Blue-Green | Run v1 and v2 simultaneously, flip traffic | Low (but 2x infra) | Instant (flip DNS/LB) |
| Canary | Send small % of traffic to v2 | Very low | Remove canary |
| A/B Testing | Route specific users to v2 | Low | Remove route |
Blue-green is best for databases that can't run two schema versions simultaneously. Canary is best for gradual confidence building with real traffic.
24. Write a GitHub Actions workflow for a Node.js app
name: Node.js CI/CD
on:
push:
branches: [main]
pull_request:
branches: [main]
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
node-version: [20, 22]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: ${{ matrix.node-version }}
cache: "npm"
- run: npm ci
- run: npm test
- run: npm run lint
build-and-push:
needs: test
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- uses: actions/checkout@v4
- uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- uses: docker/build-push-action@v5
with:
push: true
tags: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
deploy:
needs: build-and-push
runs-on: ubuntu-latest
environment: production
steps:
- name: Deploy to cluster
run: |
kubectl set image deployment/myapp \
app=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
25. What is GitOps?
GitOps applies Git as the single source of truth for infrastructure and application state. The cluster continuously reconciles with the git repo.
| Traditional CD | GitOps |
|---|---|
| CI/CD pipeline pushes to cluster | Operator pulls from git |
| Cluster state diverges over time | Cluster auto-heals drift |
| Credentials in CI system | Cluster pulls — fewer credentials needed |
| Rollback = re-run pipeline | Rollback = git revert |
Tools: ArgoCD, Flux CD
Infrastructure as Code
26. What is Terraform state and why does it matter?
Terraform stores the current state of managed infrastructure in terraform.tfstate. It uses this to:
- Know what resources exist (diff against config)
- Detect out-of-band changes (drift)
- Map config resources to real cloud resources
Best practice: Remote state backend:
terraform {
backend "s3" {
bucket = "mycompany-tf-state"
key = "prod/terraform.tfstate"
region = "us-east-1"
encrypt = true
dynamodb_table = "terraform-lock" # prevents concurrent apply
}
}
Never commit terraform.tfstate to git — it may contain secrets.
27. What is the difference between terraform plan and terraform apply?
| Command | Effect |
|---|---|
terraform init |
Initialize provider plugins, backend |
terraform validate |
Syntax/config check (no API calls) |
terraform plan |
Preview changes (read-only) |
terraform apply |
Create/update/destroy real resources |
terraform destroy |
Destroy all managed resources |
terraform import |
Import existing resource into state |
terraform state list |
List all managed resources |
# Safe workflow
terraform plan -out=tfplan # save plan to file
terraform apply tfplan # apply exactly the saved plan
28. What is Ansible idempotency?
Idempotency means running a playbook multiple times produces the same result as running it once. The system ends in the desired state regardless of its starting state.
- name: Ensure nginx is installed and running
hosts: webservers
become: yes
tasks:
- name: Install nginx
apt:
name: nginx
state: present # idempotent — won't reinstall if already present
- name: Start nginx
service:
name: nginx
state: started # idempotent — no-op if already running
enabled: yes
- name: Deploy config
template:
src: nginx.conf.j2
dest: /etc/nginx/nginx.conf
notify: Reload nginx # handler only runs if file changed
handlers:
- name: Reload nginx
service:
name: nginx
state: reloaded
29. What is Terraform locals vs variables?
| Concept | Purpose | Set by |
|---|---|---|
variable |
Input value (parameterizes module) | User/tfvars file/CI env |
local |
Computed value within module | Module author (not settable outside) |
output |
Expose value to parent module or CLI | Module |
variable "environment" {
type = string
default = "production"
}
locals {
name_prefix = "${var.environment}-${var.region}"
common_tags = {
Environment = var.environment
ManagedBy = "terraform"
}
}
resource "aws_s3_bucket" "data" {
bucket = "${local.name_prefix}-data"
tags = local.common_tags
}
Cloud
30. What is the AWS Shared Responsibility Model?
| Responsibility | AWS | Customer |
|---|---|---|
| Physical hardware | ✓ | |
| Hypervisor | ✓ | |
| Network infrastructure | ✓ | |
| Managed service patching (RDS, Lambda) | ✓ | |
| OS patching (EC2) | ✓ | |
| Application code | ✓ | |
| IAM policies | ✓ | |
| Data encryption | ✓ | |
| Security groups / NACLs | ✓ |
31. What is IAM and what are least-privilege permissions?
AWS Identity and Access Management (IAM) controls who can do what to which resources.
Least privilege: Grant only the minimum permissions needed.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": "arn:aws:s3:::my-bucket/*"
}
]
}
Common IAM concepts:
- User: Long-term credentials for a person
- Role: Temporary credentials for a service/app (preferred)
- Group: Collection of users with shared permissions
- Policy: JSON document defining permissions
32. What is the difference between an AWS Security Group and a NACL?
| Feature | Security Group | Network ACL |
|---|---|---|
| Applies to | EC2 instance (ENI) | Subnet |
| State | Stateful (response allowed automatically) | Stateless (must allow both directions) |
| Rules | Allow only | Allow + Deny |
| Rule evaluation | All rules evaluated | Rules evaluated in number order |
| Default | Deny all inbound, allow all outbound | Allow all in and out |
33. What is auto-scaling and what triggers it?
Auto-scaling adjusts the number of EC2 instances (or containers) based on demand:
| Scaling policy | Trigger |
|---|---|
| Target tracking | Keep metric at target (e.g., 70% CPU) |
| Step scaling | Add N instances when metric crosses threshold |
| Scheduled | Scale up for peak hours (e.g., 9 AM weekdays) |
| Predictive | ML-based forecast using historical patterns |
# AWS CLI: create simple scaling policy
aws autoscaling put-scaling-policy \
--auto-scaling-group-name my-asg \
--policy-name cpu-scale-out \
--policy-type TargetTrackingScaling \
--target-tracking-configuration '{
"TargetValue": 70.0,
"PredefinedMetricSpecification": {
"PredefinedMetricType": "ASGAverageCPUUtilization"
}
}'
Monitoring & Observability
34. What are the three pillars of observability?
| Pillar | Description | Tools |
|---|---|---|
| Metrics | Numerical time-series data (CPU, requests/sec) | Prometheus, CloudWatch, Datadog |
| Logs | Timestamped event records | ELK Stack, Loki, CloudWatch Logs |
| Traces | End-to-end request journey across services | Jaeger, Zipkin, AWS X-Ray |
A fourth pillar is sometimes added: Events (state changes, deploys, alerts).
35. How does Prometheus work?
Prometheus is a pull-based monitoring system. It scrapes metrics from HTTP endpoints (/metrics):
# prometheus.yml
global:
scrape_interval: 15s
scrape_configs:
- job_name: "myapp"
static_configs:
- targets: ["myapp:3000"]
- job_name: "node"
static_configs:
- targets: ["node-exporter:9100"]
# Prometheus metrics format (text/plain)
# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET",status="200"} 1234
http_requests_total{method="POST",status="500"} 12
PromQL queries:
# Request rate (per second) over 5 minutes
rate(http_requests_total[5m])
# Error rate
rate(http_requests_total{status=~"5.."}[5m]) /
rate(http_requests_total[5m])
# 95th percentile latency
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))
36. What is the ELK stack?
| Component | Role |
|---|---|
| Elasticsearch | Distributed search and analytics engine |
| Logstash | Log collection, parsing, transformation pipeline |
| Kibana | Visualization and dashboards |
| Beats (extension) | Lightweight log shippers (Filebeat, Metricbeat) |
Modern alternative: Grafana Loki (cheaper, label-based, integrates with Prometheus).
37. What is a dead man's switch in alerting?
A heartbeat alert (dead man's switch) fires when a signal stops arriving — used to detect when a job or service silently dies without error:
# Prometheus alerting rule
- alert: CronJobMissing
expr: absent(job_last_success_timestamp_seconds{job="backup"} > time() - 3600)
for: 5m
annotations:
summary: "Backup cron job has not run in over 1 hour"
Networking
38. What happens when you type a URL in a browser?
- DNS resolution: Browser checks cache → OS cache → DNS resolver → authoritative NS
- TCP handshake: SYN → SYN-ACK → ACK
- TLS handshake (HTTPS): ClientHello → ServerHello → certificate verify → session keys
- HTTP request:
GET / HTTP/2 - Server processing: Load balancer → app server → database
- HTTP response: HTML payload returned
- Rendering: Browser parses HTML, fetches CSS/JS, renders page
39. What is the difference between TCP and UDP?
| Feature | TCP | UDP |
|---|---|---|
| Connection | Connection-oriented (3-way handshake) | Connectionless |
| Delivery guarantee | Yes (retransmit on loss) | No |
| Order guarantee | Yes | No |
| Speed | Slower (overhead) | Faster |
| Use cases | HTTP, SSH, databases | DNS, video streaming, gaming |
40. What is a load balancer and what are common algorithms?
A load balancer distributes incoming requests across multiple backend servers:
| Algorithm | Description | Best for |
|---|---|---|
| Round Robin | Each server in sequence | Uniform requests |
| Least Connections | Send to least busy server | Varying request durations |
| IP Hash | Same client → same server | Session persistence |
| Weighted Round Robin | More capacity = more requests | Mixed server sizes |
| Random | Random selection | Simple, stateless |
L4 vs L7:
- L4 (TCP/UDP): Fast, no content inspection — used for raw TCP proxying
- L7 (HTTP/HTTPS): Content-aware — path-based routing, SSL termination, header manipulation
41. What is a CDN and how does it work?
A Content Delivery Network (CDN) caches static content at edge nodes close to users:
- User requests
https://example.com/logo.png - DNS resolves to nearest CDN edge (GeoDNS)
- Cache hit: CDN returns cached response (< 10ms)
- Cache miss: CDN fetches from origin, caches for
Cache-Control: max-age=..., returns to user
Benefits: lower latency, reduced origin load, DDoS protection, automatic TLS.
Security
42. What is the principle of least privilege in DevOps?
Every user, service, and process should have only the minimum access required for its function:
- Service accounts: narrow IAM policies (not
*actions) - Containers: drop Linux capabilities, run as non-root
- Kubernetes: RBAC roles scoped to namespace + specific resources
- Database users: read-only for read replicas, limited tables for app users
- SSH: deploy keys with read-only access; disable root login
43. How do you manage secrets in a CI/CD pipeline?
| Approach | Risk | Recommendation |
|---|---|---|
| Hardcoded in code | ❌ Critical | Never |
| Environment variables in CI | ⚠️ Medium | Use CI secret store |
.env file in git |
❌ Critical | Add to .gitignore |
| CI platform secrets (GitHub Secrets) | ✓ OK | Acceptable |
| HashiCorp Vault | ✓ Best | Recommended for production |
| AWS Secrets Manager + OIDC | ✓ Best | Keyless, no long-lived credentials |
# GitHub Actions: OIDC-based AWS credentials (no stored secrets)
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789:role/GitHubActionsRole
aws-region: us-east-1
44. What is container image scanning?
Scanning container images for known CVEs (Common Vulnerabilities and Exposures) in base OS packages and language dependencies:
# Trivy (open source scanner)
trivy image myapp:1.0
# Scan in CI pipeline
trivy image --exit-code 1 --severity CRITICAL myapp:1.0
# exit code 1 = fail CI on critical vulnerabilities
Best practices:
- Use minimal base images (
alpine,distroless) - Scan in CI before pushing to registry
- Set up registry scanning (ECR, Docker Hub, GCR)
- Rebuild images regularly for OS patches
45. What is RBAC in Kubernetes?
Role-Based Access Control grants Kubernetes API permissions through:
- Role / ClusterRole: Define permissions (verbs + resources)
- RoleBinding / ClusterRoleBinding: Assign roles to subjects
# Role: read-only access to pods in 'staging' namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: staging
name: pod-reader
rules:
- apiGroups: [""]
resources: ["pods", "pods/log"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
namespace: staging
name: read-pods
subjects:
- kind: ServiceAccount
name: monitoring-agent
namespace: staging
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io
Advanced Topics
46. What is a service mesh and when do you need one?
A service mesh (Istio, Linkerd) adds a sidecar proxy to every Pod for:
| Feature | Without mesh | With mesh |
|---|---|---|
| mTLS between services | Manual cert management | Automatic |
| Observability | App-level only | Automatic traces/metrics |
| Traffic shaping | Manual LB rules | Declarative policies |
| Circuit breaking | App code | Sidecar config |
| Retries | App code | Sidecar config |
Add a service mesh when you have many microservices with complex networking requirements. Avoid it for simple architectures — it adds operational overhead.
47. What is a circuit breaker pattern?
Prevents cascading failures when a downstream service is degraded:
| State | Behavior |
|---|---|
| Closed | Requests pass through normally |
| Open | Requests fail fast (no calls to failing service) |
| Half-Open | A few test requests; reset if successful |
The circuit "opens" after N consecutive failures within a time window. It "half-opens" after a timeout to test recovery.
48. How do you handle database schema migrations in CI/CD?
| Approach | How | Pros/Cons |
|---|---|---|
| Run on startup | App applies migrations at boot | Simple; dangerous at scale |
| Pre-deploy job | Kubernetes Job runs before Deployment | Controlled; rollback risk |
| Flyway / Liquibase | Versioned migration files in source | Auditable; transactional |
| Blue-green with expand-contract | Backward-compatible migrations first | Zero downtime; complex |
Expand-contract pattern for zero-downtime:
- Expand: Add new column (nullable, no breaking change) → deploy app v1
- Backfill: Populate new column
- Contract: Remove old column → deploy app v2
49. What is chaos engineering?
Deliberately inject failures into production to discover weaknesses before they cause incidents:
| Tool | What it does |
|---|---|
| Chaos Monkey (Netflix) | Randomly terminate EC2 instances |
| Gremlin | Inject CPU/memory/network/latency failures |
| Litmus (Kubernetes) | Chaos experiments via CRDs |
| Toxiproxy | Simulate network conditions in staging |
Chaos engineering principles:
- Define steady state (SLI baseline)
- Hypothesize the system will maintain steady state under failure
- Introduce failures in production (start small)
- Look for differences from hypothesis
- Fix weaknesses found
50. What is an error budget and how does it drive decisions?
Error budget = total allowed downtime under an SLO.
At 99.9% monthly SLO: 0.1% × 43,800 min = 43.8 minutes allowed downtime.
| Budget status | Team action |
|---|---|
| Budget healthy (> 50%) | Release new features aggressively |
| Budget draining (20–50%) | Slow releases, investigate reliability |
| Budget nearly exhausted | Feature freeze; reliability work only |
| Budget exhausted | Stop all deploys until next period |
Error budgets make reliability a shared responsibility between product and engineering — reliability is not "Ops' problem" when releases burn the budget.
Common mistakes
| Mistake | Fix |
|---|---|
| Running containers as root | Add USER nonroot in Dockerfile |
| Storing secrets in environment variables in plain text | Use Vault, AWS Secrets Manager, or Kubernetes Secrets |
| No resource requests/limits on Pods | Set CPU/memory requests + limits on every container |
kubectl apply directly to production |
Use GitOps (ArgoCD/Flux) or deployment pipelines |
| Single point of failure in pipeline | Redundant runners; multi-AZ deployments |
| Ignoring Terraform state locking | Use S3 + DynamoDB or Terraform Cloud |
| Fat Docker images | Multi-stage builds + .dockerignore |
| No rollback plan | Always test kubectl rollout undo and DB migration revert |
DevOps vs SRE vs Platform Engineering
| Role | Focus | Owns |
|---|---|---|
| DevOps Engineer | Automation + CI/CD + culture | Pipelines, IaC, tooling |
| SRE | Reliability + error budgets + on-call | SLOs, incident response, capacity |
| Platform Engineer | Internal developer platform | Self-service infra, paved paths, IDPs |
| Cloud Architect | Cloud strategy + design | Architecture decisions, cost |
6 FAQ
Q: Is Kubernetes overkill for a small team?
A: Yes, usually. Start with a single server + Docker Compose or a managed PaaS (Railway, Render, Fly.io). Add Kubernetes when you need multi-region deployments, per-service scaling, or have > 5 services.
Q: Terraform vs Pulumi vs Ansible — which do I choose?
A: Terraform/Pulumi for infrastructure provisioning (declarative, stateful). Ansible for configuration management (imperative, agentless). They complement each other; many teams use both.
Q: How do I handle long-running jobs in Kubernetes?
A: Use kind: Job for one-off tasks and kind: CronJob for scheduled tasks. For streaming jobs, use Deployments with appropriate liveness probes.
Q: What's the difference between kubectl apply and kubectl create?
A: create fails if the resource exists. apply creates or updates (merge patch). Always use apply in automation; it's idempotent.
Q: How do I reduce Docker image build times in CI?
A: Layer caching (copy dependencies before source), multi-stage builds, --cache-from pointing to previously built images, and tools like BuildKit cache mounts.
Q: What certifications are useful for DevOps roles?
A: CKA (Certified Kubernetes Administrator), CKS (Kubernetes Security), AWS SAA (Solutions Architect Associate), Terraform Associate, and Google Cloud Professional DevOps Engineer are most valued.