Data Engineer Roadmap 2025 (Complete Guide)

A data engineer builds the infrastructure that moves, transforms, and stores data so analysts and data scientists can use it. This roadmap shows you exactly what to learn, in what order, and realistic timelines to go from beginner to job-ready data engineer in 2025.

At a Glance

Phase 1 — Foundations (Months 1–2)

Before touching data pipelines, get comfortable with the command line, version control, and the basics of Python.

Linux & Shell

Data engineers spend a lot of time in terminals — SSH-ing into servers, inspecting logs, running scripts. Learn:

• File system navigation (ls, cd, find, grep)
• File manipulation (cp, mv, rm, cat, head, tail)
• Process management (ps, kill, top, nohup)
• Text processing (awk, sed, cut, sort, uniq, wc)
• Shell scripting (for loops, conditionals, cron for scheduling)

Git

All pipeline code lives in Git. Learn:

git init / clone / add / commit / push / pull
git branch / checkout / merge / rebase
git log --oneline --graph

Use Conventional Commits (feat:, fix:, chore:) from day one.

Python Basics

You don't need to be a Python expert yet — just comfortable:

• Variables, types, control flow, functions
• List/dict comprehensions
• File I/O (open, csv, json)
• Exception handling (try/except)
• Virtual environments (venv, pip, or uv)

Phase 2 — SQL & Databases (Months 2–3)

SQL is the most important skill for a data engineer. You need to be fluent, not just functional.

Core SQL

-- Aggregations + GROUP BY
SELECT customer_id, SUM(amount) AS total, COUNT(*) AS orders
FROM orders
WHERE created_at >= '2025-01-01'
GROUP BY customer_id
HAVING SUM(amount) > 1000
ORDER BY total DESC;

-- JOINs
SELECT o.id, c.name, o.amount
FROM orders o
JOIN customers c ON o.customer_id = c.id
LEFT JOIN refunds r ON o.id = r.order_id
WHERE r.id IS NULL; -- orders with no refund

Window Functions (essential for analytics)

-- Ranking + running total
SELECT
  date,
  revenue,
  SUM(revenue) OVER (ORDER BY date) AS running_total,
  LAG(revenue, 1) OVER (ORDER BY date) AS prev_day,
  ROUND(100.0 * (revenue - LAG(revenue, 1) OVER (ORDER BY date))
        / LAG(revenue, 1) OVER (ORDER BY date), 2) AS day_over_day_pct
FROM daily_revenue;

Data Modeling

Database Internals to Understand

• Indexes (B-tree, hash, composite) — know when to add them
• Query execution plans (EXPLAIN ANALYZE in PostgreSQL)
• ACID transactions and isolation levels
• N+1 problem and how to avoid it

Phase 3 — Python for Data Engineering (Months 3–4)

Now go deeper with Python focused on data tasks.

pandas for Data Manipulation

import pandas as pd

df = pd.read_csv("orders.csv", parse_dates=["created_at"])

# Clean
df = df.dropna(subset=["customer_id"])
df["amount"] = pd.to_numeric(df["amount"], errors="coerce").fillna(0)

# Transform
monthly = (
    df.groupby(df["created_at"].dt.to_period("M"))
    .agg(revenue=("amount", "sum"), orders=("id", "count"))
    .reset_index()
)
print(monthly)

Writing ETL Scripts

A simple extract-transform-load script pattern:

import psycopg2, requests, json

def extract():
    resp = requests.get("https://api.example.com/orders", timeout=30)
    resp.raise_for_status()
    return resp.json()

def transform(records):
    return [
        {
            "order_id": r["id"],
            "customer_id": r["customer"]["id"],
            "amount": float(r["total"]),
        }
        for r in records
        if r.get("status") == "completed"
    ]

def load(rows, conn_str):
    with psycopg2.connect(conn_str) as conn, conn.cursor() as cur:
        cur.executemany(
            "INSERT INTO orders (order_id, customer_id, amount) VALUES (%s, %s, %s) ON CONFLICT DO NOTHING",
            [(r["order_id"], r["customer_id"], r["amount"]) for r in rows],
        )

if __name__ == "__main__":
    raw = extract()
    clean = transform(raw)
    load(clean, "postgresql://user:pass@localhost/db")

Key Libraries

Phase 4 — Data Warehousing (Months 4–5)

Data warehouses store analytical data at scale, separate from operational databases.

Core Concepts

• OLTP vs OLAP: transactional (row-oriented, many small writes) vs analytical (column-oriented, large scans)
• Columnar storage: Parquet, ORC — compress well, fast for aggregations
• Partitioning: slice tables by date/region to prune scans
• Clustering / Z-ordering: co-locate related rows for faster filtering

Major Data Warehouses

Star Schema Example

-- Dimension table
CREATE TABLE dim_customer (
    customer_sk   SERIAL PRIMARY KEY,
    customer_id   TEXT NOT NULL,
    name          TEXT,
    country       TEXT,
    valid_from    DATE,
    valid_to      DATE
);

-- Fact table
CREATE TABLE fact_orders (
    order_id      TEXT PRIMARY KEY,
    customer_sk   INT REFERENCES dim_customer(customer_sk),
    date_sk       INT,          -- FK to dim_date
    product_sk    INT,          -- FK to dim_product
    quantity      INT,
    amount        NUMERIC(12,2)
);

Phase 5 — ETL / ELT with dbt (Months 5–6)

Modern data engineering favors ELT (extract → load raw → transform in warehouse) over traditional ETL.

dbt (Data Build Tool)

dbt lets you write SQL SELECT statements; it handles materialisation, dependency resolution, and documentation.

-- models/staging/stg_orders.sql
{{ config(materialized='view') }}

SELECT
    id            AS order_id,
    customer_id,
    CAST(total AS NUMERIC) AS amount,
    status,
    created_at::DATE AS order_date
FROM {{ source('raw', 'orders') }}
WHERE status != 'cancelled'

-- models/marts/fct_monthly_revenue.sql
{{ config(materialized='table', partition_by={'field': 'month', 'data_type': 'date'}) }}

SELECT
    DATE_TRUNC('month', order_date) AS month,
    COUNT(*)                         AS orders,
    SUM(amount)                      AS revenue
FROM {{ ref('stg_orders') }}
GROUP BY 1

dbt Core Concepts

Ingestion Tools

Phase 6 — Orchestration (Months 6–7)

Orchestration tools schedule and monitor data pipelines, handle retries, and manage dependencies between tasks.

Apache Airflow

Airflow uses DAGs (Directed Acyclic Graphs) written in Python.

from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.providers.postgres.operators.postgres import PostgresOperator
from datetime import datetime, timedelta

default_args = {"retries": 2, "retry_delay": timedelta(minutes=5)}

with DAG(
    "daily_orders_pipeline",
    schedule_interval="0 6 * * *",   # 06:00 UTC daily
    start_date=datetime(2025, 1, 1),
    catchup=False,
    default_args=default_args,
) as dag:

    extract = PythonOperator(
        task_id="extract_orders",
        python_callable=extract_orders_from_api,
    )

    load = PostgresOperator(
        task_id="load_to_staging",
        sql="sql/load_staging_orders.sql",
        postgres_conn_id="warehouse",
    )

    transform = BashOperator(
        task_id="dbt_run",
        bash_command="dbt run --select marts.fct_monthly_revenue",
    )

    extract >> load >> transform

Orchestration Comparison

Phase 7 — Big Data & Apache Spark (Months 7–8)

When data exceeds a single machine (typically 10GB+), you need distributed computing.

Spark Concepts

• RDD → low-level distributed collection (avoid in new code)
• DataFrame → structured, SQL-like API (use this)
• Lazy evaluation → transformations build a plan; actions execute it
• Partitions → units of parallelism; aim for 100–200MB per partition

PySpark Example

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = SparkSession.builder.appName("OrdersETL").getOrCreate()

df = spark.read.parquet("s3://data-lake/raw/orders/")

result = (
    df
    .filter(F.col("status") == "completed")
    .withColumn("month", F.date_trunc("month", F.col("created_at")))
    .groupBy("month")
    .agg(
        F.sum("amount").alias("revenue"),
        F.count("*").alias("orders"),
    )
    .orderBy("month")
)

result.write.mode("overwrite").parquet("s3://data-lake/processed/monthly_revenue/")

Spark Performance Tips

Phase 8 — Streaming Data (Months 8–10)

Batch processing is great for nightly jobs, but many use cases need near-real-time data.

Apache Kafka

Kafka is the industry-standard event streaming platform.

from confluent_kafka import Producer, Consumer

# Producer
p = Producer({"bootstrap.servers": "localhost:9092"})
p.produce("orders", key="order-123", value='{"amount": 99.99}')
p.flush()

# Consumer
c = Consumer({
    "bootstrap.servers": "localhost:9092",
    "group.id": "order-processor",
    "auto.offset.reset": "earliest",
})
c.subscribe(["orders"])
while True:
    msg = c.poll(1.0)
    if msg and not msg.error():
        print(f"Received: {msg.value().decode()}")

Streaming Frameworks

Phase 9 — Cloud Data Platforms (Months 10–12)

Most data engineering happens in the cloud. Pick one cloud to go deep on first.

AWS Data Stack

GCP Data Stack

Azure Data Stack

File Formats to Know

Phase 10 — DataOps (Months 12–14)

DataOps applies DevOps principles to data pipelines: testing, CI/CD, monitoring, and data quality.

Data Quality with Great Expectations

import great_expectations as ge

df = ge.read_csv("orders.csv")

df.expect_column_values_to_not_be_null("order_id")
df.expect_column_values_to_be_unique("order_id")
df.expect_column_values_to_be_between("amount", min_value=0, max_value=100_000)
df.expect_column_values_to_be_in_set("status", ["completed", "pending", "refunded"])

results = df.validate()
print(results["success"])  # True/False

dbt Tests

# models/staging/schema.yml
models:
  - name: stg_orders
    columns:
      - name: order_id
        tests: [not_null, unique]
      - name: amount
        tests:
          - not_null
          - dbt_utils.accepted_range:
              min_value: 0
      - name: status
        tests:
          - accepted_values:
              values: [completed, pending, refunded]

CI/CD for Data Pipelines

# .github/workflows/dbt-ci.yml
name: dbt CI
on: [pull_request]
jobs:
  dbt-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with: {python-version: "3.12"}
      - run: pip install dbt-bigquery
      - run: dbt deps
      - run: dbt compile
      - run: dbt test --select state:modified+
        env:
          DBT_ENV_PROFILES_DIR: .

Observability

Full Technology Map

┌─────────────────────────────────────────────────────────────────┐
│                    DATA ENGINEER TECH MAP                       │
├──────────────┬──────────────────────────────────────────────────┤
│ Foundations  │ Linux · Git · Python · SQL                       │
├──────────────┼──────────────────────────────────────────────────┤
│ Storage      │ PostgreSQL · S3/GCS/ADLS · Delta/Iceberg/Parquet │
├──────────────┼──────────────────────────────────────────────────┤
│ Warehousing  │ Snowflake · BigQuery · Redshift · Databricks     │
├──────────────┼──────────────────────────────────────────────────┤
│ Ingestion    │ Airbyte · Fivetran · Kafka Connect · Custom      │
├──────────────┼──────────────────────────────────────────────────┤
│ Transform    │ dbt · PySpark · pandas · polars · SQL            │
├──────────────┼──────────────────────────────────────────────────┤
│ Orchestration│ Airflow · Prefect · Dagster · dbt Cloud          │
├──────────────┼──────────────────────────────────────────────────┤
│ Batch        │ Spark · Glue · Dataflow · EMR                    │
├──────────────┼──────────────────────────────────────────────────┤
│ Streaming    │ Kafka · Flink · Spark Streaming · Kinesis        │
├──────────────┼──────────────────────────────────────────────────┤
│ DataOps      │ Great Expectations · dbt tests · OpenLineage     │
├──────────────┼──────────────────────────────────────────────────┤
│ Cloud        │ AWS (Glue/Redshift/EMR) · GCP · Azure            │
└──────────────┴──────────────────────────────────────────────────┘

Realistic Timeline

Portfolio Project Ideas

Data Engineer Roles & Salary

Common Mistakes

Data Engineer vs Related Roles

Frequently Asked Questions

Do I need a CS degree to become a data engineer?

No. Many data engineers come from software development, data analysis, or self-taught backgrounds. What matters is demonstrable skill in SQL, Python, and building pipelines. A portfolio of working projects is more persuasive than a degree.

Should I learn Scala or Python for Spark?

Start with Python (PySpark). It has a larger community, easier learning curve, and most new Spark jobs use PySpark. Learn Scala only if your employer or a specific job requires it — it offers better performance but isn't necessary to get hired.

Spark or dbt first?

dbt first. Most data engineering roles start in the warehouse layer (Snowflake/BigQuery + dbt). Spark is needed once data scale exceeds what the warehouse handles well, which may be months or years into a role.

How important is real-time/streaming?

Valuable but not required for most roles. The majority of data engineering work is batch. Get a solid batch foundation, then add streaming. Kafka is by far the most common streaming technology to know.

Which cloud should I learn first?

AWS is the largest market share and most job postings. GCP is best if you plan to use BigQuery (easiest warehouse to start with). Azure is dominant in enterprise/Microsoft shops. Pick based on the jobs you're targeting, or learn BigQuery on GCP first for its low-cost serverless model.

What SQL questions get asked in data engineering interviews?

Expect window functions (ROW_NUMBER, RANK, LAG/LEAD, running totals), complex JOINs, de-duplication, finding the Nth highest value, and writing incremental queries. Practice on real datasets — not just toy examples.