50 Apache Spark Interview Questions (With Answers)

Apache Spark interviews test your understanding of distributed data processing — from RDDs and DataFrames to Structured Streaming, memory management, and cluster tuning. This guide covers the 50 most common questions with concise, accurate answers.

Quick reference

Core Architecture

1. What is Apache Spark and how does it differ from Hadoop MapReduce?

Apache Spark is an open-source, in-memory distributed data processing engine designed for batch, streaming, SQL, machine learning, and graph workloads.

When to choose Spark: iterative ML, interactive analytics, complex multi-stage pipelines, real-time streaming.

When MapReduce still fits: stable, simple ETL on massive HDFS data where memory is scarce.

2. Describe Spark's architecture.

┌─────────────────────────────────┐
│         Cluster Manager         │  ← YARN / Kubernetes / Standalone
│   (allocates resources)         │
└──────────────┬──────────────────┘
               │
┌──────────────▼──────────────────┐
│           Driver Node           │
│  ┌────────────────────────┐     │
│  │ SparkContext / Session  │     │
│  │  DAGScheduler          │     │
│  │  TaskScheduler         │     │
│  └────────────────────────┘     │
└──────┬──────────────┬───────────┘
       │              │
  ┌────▼────┐    ┌────▼────┐
  │Executor │    │Executor │   ← Worker Nodes
  │ Task    │    │ Task    │
  │ Task    │    │ Task    │
  │ Cache   │    │ Cache   │
  └─────────┘    └─────────┘

Key components:

3. What is a DAG in Spark?

A Directed Acyclic Graph (DAG) is Spark's representation of the computation pipeline.

• Spark builds a DAG of transformations when you chain operations.
• The DAG is executed when an action (e.g., collect(), count(), write()) is called.
• DAGScheduler divides the DAG into stages at shuffle boundaries (wide dependencies).
• Within a stage, tasks run in parallel on different partitions.

# This builds a DAG — nothing executes yet
result = (
    spark.read.parquet("s3://data/events")   # read
         .filter("event_type = 'click'")      # narrow transformation
         .groupBy("user_id")                  # wide — shuffle boundary
         .agg({"event_id": "count"})          # aggregation
         .filter("count > 10")                # narrow
)

result.show()   # ← action triggers DAG execution

4. Explain the difference between transformations and actions.

Narrow transformations — each input partition contributes to at most one output partition (no shuffle): map, filter, flatMap, union, sample

Wide transformations — require shuffling data across partitions (stage boundary): groupByKey, reduceByKey, join, distinct, repartition, sortBy

5. What is a Spark Session vs SparkContext?

In Spark 2+, SparkSession is preferred. You can still access SparkContext via spark.sparkContext.

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("MyApp") \
    .config("spark.executor.memory", "4g") \
    .getOrCreate()

sc = spark.sparkContext   # access underlying SparkContext

RDDs, DataFrames, Datasets

6. What is an RDD? What are its properties?

An RDD (Resilient Distributed Dataset) is Spark's foundational data abstraction.

Properties:

1. Resilient — fault-tolerant via lineage (can recompute lost partitions)
2. Distributed — data partitioned across nodes
3. Dataset — collection of records (any Python/Java/Scala object)
4. Immutable — transformations produce new RDDs
5. Lazily evaluated — computed only when action is called
6. Typed — RDD[T] in Scala/Java; Python RDDs are untyped

Creating RDDs:

# From collection
rdd = sc.parallelize([1, 2, 3, 4, 5], numSlices=4)

# From file
rdd = sc.textFile("hdfs://path/to/file.txt")

# From another RDD (transformation)
squares = rdd.map(lambda x: x * x)

7. What is the difference between RDD, DataFrame, and Dataset?

In PySpark: only RDD and DataFrame are available (Dataset is a JVM concept).

8. What are the common DataFrame transformations?

df = spark.read.parquet("s3://bucket/sales/")

# Selection
df.select("name", "amount", (col("amount") * 1.1).alias("amount_with_tax"))

# Filtering
df.filter(col("amount") > 100)
df.where("region = 'EU'")

# Aggregation
df.groupBy("region").agg(
    sum("amount").alias("total"),
    avg("amount").alias("avg"),
    count("*").alias("cnt")
)

# Joins
orders.join(customers, orders.customer_id == customers.id, "left")

# Window functions
from pyspark.sql.window import Window
w = Window.partitionBy("region").orderBy(desc("amount"))
df.withColumn("rank", rank().over(w))

# Deduplication
df.dropDuplicates(["order_id"])

# Column operations
df.withColumn("year", year(col("order_date"))) \
  .withColumn("is_high_value", col("amount") > 1000) \
  .drop("internal_id")

9. What is the difference between map and flatMap?

rdd = sc.parallelize(["hello world", "apache spark"])

# map — each string becomes one element
rdd.map(lambda s: s.split()).collect()
# [['hello', 'world'], ['apache', 'spark']]

# flatMap — flattens nested lists
rdd.flatMap(lambda s: s.split()).collect()
# ['hello', 'world', 'apache', 'spark']

10. What are reduceByKey and groupByKey? Which is preferred?

pairs = rdd.map(lambda x: (x['key'], x['value']))

# BAD — shuffles all values
pairs.groupByKey().mapValues(sum)

# GOOD — reduces locally first, then shuffles
pairs.reduceByKey(lambda a, b: a + b)

Rule: prefer reduceByKey, aggregateByKey, or DataFrame groupBy().agg() over groupByKey.

Spark SQL & Catalyst

11. What is the Catalyst optimizer?

Catalyst is Spark SQL's extensible query optimizer. It transforms a logical query plan into an optimized physical plan through 4 phases:

SQL / DataFrame API
        ↓
1. Unresolved Logical Plan   (parse SQL string)
        ↓
2. Analyzed Logical Plan     (resolve column names, types)
        ↓
3. Optimized Logical Plan    (predicate pushdown, constant folding, etc.)
        ↓
4. Physical Plans            (multiple physical strategies)
        ↓
5. Best Physical Plan        (cost model selection)
        ↓
6. Code Generation (Tungsten)

Key optimizations Catalyst applies:

• Predicate pushdown — move filters as close to the data source as possible
• Column pruning — read only needed columns
• Constant folding — evaluate 1 + 1 → 2 at compile time
• Join reordering — put smaller tables first
• Partition pruning — skip entire Parquet/Hive partitions

12. What is Tungsten execution engine?

Tungsten is Spark's physical execution layer that improves upon the JVM's default behaviour:

13. How do you run SQL queries in Spark?

# Register as temp view
df.createOrReplaceTempView("sales")

# Run SQL
result = spark.sql("""
    SELECT region,
           SUM(amount) AS total,
           COUNT(*) AS orders
    FROM sales
    WHERE order_date >= '2025-01-01'
    GROUP BY region
    ORDER BY total DESC
""")

# Global temp view (survives across sessions)
df.createOrReplaceGlobalTempView("global_sales")
spark.sql("SELECT * FROM global_temp.global_sales")

14. What is the difference between cache() and persist()?

Storage levels:

from pyspark import StorageLevel

# Cache when dataset fits in memory and reused multiple times
df.cache()

# Persist to disk when memory is limited
df.persist(StorageLevel.MEMORY_AND_DISK)

# Unpersist when done
df.unpersist()

When to cache: reuse a DataFrame 2+ times in the same job (e.g., iterative ML, multiple aggregations on the same filtered dataset).

15. What is a broadcast join and when should you use it?

A broadcast join sends the smaller table to all executors, avoiding a full shuffle.

from pyspark.sql.functions import broadcast

# Spark auto-broadcasts tables < spark.sql.autoBroadcastJoinThreshold (10 MB default)
result = large_df.join(broadcast(small_df), "product_id")

# Increase threshold
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "50m")

# Disable broadcast
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1")

When to use:

• One table < 10 MB (or your configured threshold)
• Star schema fact/dimension joins
• Lookup tables (country codes, product categories)

When NOT to use:

• Both tables are large — causes OOM on executors

Partitioning & Shuffles

16. What is a partition in Spark?

A partition is the basic unit of parallelism — a logical chunk of data processed by one task.

# Check number of partitions
df.rdd.getNumPartitions()

# Repartition — full shuffle, even distribution, accepts expression
df.repartition(200)
df.repartition(200, col("region"))  # partition by column

# Coalesce — reduce partitions, no full shuffle (merge adjacent)
df.coalesce(50)   # only for reducing; cannot increase

# Rule of thumb: 2–4 partitions per CPU core; aim for 128–256 MB per partition

17. What is a shuffle and why is it expensive?

A shuffle redistributes data across partitions based on a key. It is triggered by wide transformations (groupBy, join, distinct, repartition).

Why it's expensive:

1. Disk I/O — shuffle writes to local disk before sending
2. Network I/O — data moves across nodes
3. Serialisation/deserialisation — objects encoded and decoded
4. Sorting — shuffle sort phase

How to reduce shuffles:

• Use reduceByKey / aggregateByKey instead of groupByKey
• Use broadcast joins for small tables
• Partition data by join key before joining (bucket joins)
• Use coalesce instead of repartition when reducing
• Enable Adaptive Query Execution (AQE)

18. What is Adaptive Query Execution (AQE)?

AQE (enabled by default in Spark 3.0+) dynamically re-optimises the query plan at runtime using statistics collected during execution.

# Enable AQE (default in Spark 3.0+)
spark.conf.set("spark.sql.adaptive.enabled", "true")

# Key AQE features:
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")   # merge small post-shuffle partitions
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")             # split skewed partitions
spark.conf.set("spark.sql.adaptive.localShuffleReader.enabled", "true")   # avoid unnecessary data movement

AQE optimisations:

19. What is data skew and how do you handle it?

Data skew occurs when some partitions have significantly more data than others — causing a few slow tasks that delay the entire stage.

Detection:

# Check partition sizes
df.rdd.mapPartitionsWithIndex(
    lambda i, it: [(i, sum(1 for _ in it))]
).toDF(["partition", "count"]).orderBy(desc("count")).show(10)

Fixes:

# Salting example
import random

# Add salt to skewed dataset
salted = skewed_df.withColumn(
    "salted_key",
    concat(col("join_key"), lit("_"), (rand() * 10).cast("int"))
)

# Explode salt on small side
small_salted = small_df.crossJoin(
    spark.range(10).withColumnRenamed("id", "salt")
).withColumn("salted_key", concat(col("join_key"), lit("_"), col("salt")))

salted.join(small_salted, "salted_key")

20. Explain bucketing in Spark.

Bucketing pre-partitions data on disk by a column, so joins/groupBys on that column skip the shuffle entirely.

# Write bucketed table — one-time cost
df.write \
  .bucketBy(64, "user_id") \
  .sortBy("user_id") \
  .saveAsTable("events_bucketed")

# Join avoids shuffle when both sides bucketed on same key with same # buckets
events_bucketed.join(users_bucketed, "user_id")

Requirements for shuffle-free join:

• Both tables bucketed on join key
• Same number of buckets
• Spark reads from Hive/managed tables (not arbitrary paths)

Structured Streaming

21. What is Structured Streaming?

Structured Streaming is Spark's streaming engine built on the DataFrame/Dataset API. It models a stream as an unbounded table that continuously grows.

Streaming input:   row 1, row 2, row 3, row 4, ...
                   └──────── treated as ──────────┘
Unbounded table:   | col_a | col_b | ... |
                   |  v1   |  v2   | ... |   ← new rows appended

# Read from Kafka
stream_df = spark.readStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", "broker:9092") \
    .option("subscribe", "events") \
    .load()

# Parse and process
result = stream_df \
    .selectExpr("CAST(value AS STRING) as json") \
    .select(from_json("json", schema).alias("data")) \
    .select("data.*") \
    .groupBy(window("timestamp", "5 minutes"), "region") \
    .count()

# Write output
query = result.writeStream \
    .outputMode("update") \
    .format("console") \
    .trigger(processingTime="10 seconds") \
    .start()

query.awaitTermination()

22. What are the output modes in Structured Streaming?

23. What is a watermark in Structured Streaming?

A watermark tells Spark how long to wait for late-arriving data before considering a time window closed.

result = stream_df \
    .withWatermark("event_time", "10 minutes") \   # allow up to 10 min late events
    .groupBy(
        window("event_time", "5 minutes"),          # 5-min tumbling window
        "device_id"
    ) \
    .count()

Without watermark, Spark keeps state for all windows forever → OOM.

With watermark of 10 minutes: Spark drops events more than 10 minutes late and cleans up old state.

24. What are the trigger types in Structured Streaming?

25. How does Structured Streaming handle fault tolerance?

Spark uses checkpointing to save progress:

query = result.writeStream \
    .option("checkpointLocation", "s3://bucket/checkpoints/query1") \
    .format("delta") \
    .outputMode("append") \
    .start()

On restart:

1. Reads checkpoint to find last committed offset
2. Reprocesses from that offset
3. Guarantees exactly-once with idempotent sinks (Delta Lake, Kafka, JDBC with upsert)

Memory Management

26. How does Spark manage memory?

Spark JVM memory is split into two main regions:

Total Executor JVM Heap
├── Reserved Memory (300 MB) — internal Spark use
└── Usable Memory (heap - 300 MB)
    ├── spark.memory.fraction (default 0.6)
    │   ├── Execution Memory — shuffle, sort, joins, aggregations
    │   └── Storage Memory — cached RDD/DataFrame data
    │       (these two share the fraction dynamically)
    └── User Memory (0.4) — Python UDFs, user data structures, metadata

# Tune executor memory split
spark.conf.set("spark.memory.fraction", "0.7")        # more for Spark internals
spark.conf.set("spark.memory.storageFraction", "0.4") # within the 0.7, 40% for cache

Off-heap memory (for Tungsten):

spark.conf.set("spark.memory.offHeap.enabled", "true")
spark.conf.set("spark.memory.offHeap.size", "4g")

27. What causes OOM errors in Spark and how do you fix them?

28. What is spilling in Spark?

Spilling occurs when Spark runs out of execution memory and writes intermediate data to disk.

• Spill to disk is automatic — prevents OOM but causes significant slowdown.
• Monitor via Spark UI: Stages → Shuffle Write/Read bytes spilled.

Reduce spilling:

# More executor memory
spark.conf.set("spark.executor.memory", "8g")

# More parallelism (smaller partitions)
spark.conf.set("spark.sql.shuffle.partitions", "400")

# Increase memory fraction
spark.conf.set("spark.memory.fraction", "0.8")

Performance Tuning

29. How do you tune the number of shuffle partitions?

# Default is 200 — often too high for small datasets, too low for large
spark.conf.set("spark.sql.shuffle.partitions", "400")

# With AQE enabled, Spark auto-coalesces small partitions
spark.conf.set("spark.sql.adaptive.enabled", "true")

# Rule of thumb: target 128–256 MB per partition after shuffle
# estimated_shuffle_size / target_partition_size

30. What is the difference between repartition and coalesce?

# After heavy filter that leaves few rows — reduce before write
df.filter(col("country") == "US") \
  .coalesce(10) \
  .write.parquet("s3://output/us/")

# Need even distribution for join/aggregation
df.repartition(200, col("user_id"))

31. What are common Spark performance anti-patterns?

32. What is predicate pushdown and how does Spark use it?

Predicate pushdown moves filter conditions as close to the data source as possible, reducing the amount of data read.

# Spark pushes this filter down to Parquet reader
# Only reads row groups where year = 2025 — skips others
df = spark.read.parquet("s3://sales/") \
         .filter(col("year") == 2025)

# Verify pushdown in query plan
df.explain(True)
# Look for: PushedFilters: [IsNotNull(year), EqualTo(year,2025)]

Works with: Parquet, ORC, JDBC (pushes SQL WHERE), Delta Lake, Iceberg.

Partition pruning — Spark skips entire directory partitions:

# If data is partitioned by year=.../month=...
# Only reads year=2025/ directory
df.filter("year = 2025 AND month = 3")

33. How do you use EXPLAIN to debug query plans?

df.explain()           # Physical plan only
df.explain("simple")  # Same as above
df.explain("extended") # Logical + physical plans
df.explain("cost")    # + cost model estimates
df.explain("formatted") # Indented tree (Spark 3+)
df.explain(True)       # Same as "extended"

Reading the output (bottom to top):

== Physical Plan ==
AdaptiveSparkPlan (1)                ← AQE wrapper
+- Project (2)                       ← final column selection
   +- BroadcastHashJoin (3)          ← broadcast join (small table)
      :- Filter (4)                  ← pushed-down filter
      :  +- FileScan parquet (5)     ← reads only needed columns
      +- BroadcastExchange (6)       ← sends small table to all executors
         +- Filter (7)
            +- FileScan parquet (8)

Cluster & Deployment

34. What cluster managers does Spark support?

# Standalone
spark-submit --master spark://master:7077 app.py

# YARN
spark-submit --master yarn --deploy-mode cluster app.py

# Kubernetes
spark-submit --master k8s://https://k8s-api-server:443 \
  --deploy-mode cluster \
  --conf spark.kubernetes.container.image=my-spark:3.5 \
  app.py

35. What is the difference between client and cluster deploy modes?

36. How do you configure Spark executor resources?

spark-submit \
  --num-executors 10 \          # total executors (YARN)
  --executor-cores 4 \          # cores per executor
  --executor-memory 8g \        # heap per executor
  --driver-memory 4g \          # driver heap
  --conf spark.executor.memoryOverhead=2g \   # off-heap (shuffle, python)
  --conf spark.dynamicAllocation.enabled=true \ # scale up/down automatically
  --conf spark.dynamicAllocation.minExecutors=2 \
  --conf spark.dynamicAllocation.maxExecutors=50 \
  app.py

Recommended executor sizing (YARN):

• Leave 1 core + 1 GB per node for OS/YARN daemons
• Fat executors (5 cores, ~20 GB) perform well; avoid 1-core executors (no intra-executor parallelism for broadcast joins)

PySpark & MLlib

37. What is PySpark?

PySpark is Python's API for Spark. It runs Python code in a worker process that communicates with the JVM Spark engine via Py4J (for small data) and Arrow (for bulk data transfer with Pandas UDFs).

# py4j: individual object operations — slower
df.count()

# Arrow (Pandas UDF) — vectorised batch transfer — fast
from pyspark.sql.functions import pandas_udf
import pandas as pd

@pandas_udf("double")
def multiply(s: pd.Series) -> pd.Series:
    return s * 2.0

df.withColumn("doubled", multiply(col("value")))

38. What is the difference between a UDF and a Pandas UDF?

# Pandas UDF — much faster than row-by-row UDF
from pyspark.sql.functions import pandas_udf

@pandas_udf("string")
def clean_name(name: pd.Series) -> pd.Series:
    return name.str.strip().str.lower()

df.withColumn("name", clean_name(col("raw_name")))

Best practice: prefer built-in Spark SQL functions; use Pandas UDFs only when native functions can't do it.

39. What is Spark MLlib?

MLlib is Spark's distributed machine learning library.

from pyspark.ml import Pipeline
from pyspark.ml.feature import VectorAssembler, StandardScaler
from pyspark.ml.classification import LogisticRegression
from pyspark.ml.evaluation import BinaryClassificationEvaluator

assembler = VectorAssembler(
    inputCols=["age", "income", "credit_score"],
    outputCol="features"
)
scaler = StandardScaler(inputCol="features", outputCol="scaled_features")
lr = LogisticRegression(featuresCol="scaled_features", labelCol="default")

pipeline = Pipeline(stages=[assembler, scaler, lr])
model = pipeline.fit(train_df)

predictions = model.transform(test_df)
evaluator = BinaryClassificationEvaluator(labelCol="default")
print(f"AUC: {evaluator.evaluate(predictions):.4f}")

40. What is Delta Lake and how does it relate to Spark?

Delta Lake is an open-source storage layer built on Parquet that brings ACID transactions to Spark.

# Write
df.write.format("delta").save("s3://lake/events/")

# MERGE (upsert)
from delta.tables import DeltaTable

DeltaTable.forPath(spark, "s3://lake/events/") \
  .alias("t") \
  .merge(updates.alias("s"), "t.id = s.id") \
  .whenMatchedUpdateAll() \
  .whenNotMatchedInsertAll() \
  .execute()

# Time travel
spark.read.format("delta") \
    .option("versionAsOf", 5) \
    .load("s3://lake/events/")

Advanced Topics

41. What is lazy evaluation and why does Spark use it?

Lazy evaluation means transformations are not executed until an action is called. Spark records each transformation in a logical plan (DAG).

Benefits:

1. Optimisation — Catalyst can optimise the entire DAG before execution (e.g., push filters before joins)
2. Fault tolerance — lineage graph allows recomputing only lost partitions
3. Efficiency — eliminates unnecessary computations if action produces early result

# Nothing executes here
df = spark.read.parquet("huge_table.parquet")      # no I/O
filtered = df.filter(col("country") == "US")       # no computation
selected = filtered.select("user_id", "amount")    # no computation

# Execution happens here — Catalyst optimises all steps together
selected.show()   # ← triggers read → filter → select in one pass

42. What is the Spark UI and what can you monitor in it?

Spark UI runs on port 4040 (or 18080 for History Server) and provides:

Key metrics to watch:

• Shuffle spill (memory/disk) → add memory or reduce partition size
• GC time > 5% → memory pressure, consider off-heap
• Skewed tasks → some tasks 10× longer than median → salting needed
• Input/output rows — huge drop means predicate pushdown is working

43. How do you handle schema evolution in Spark?

# Read with schema merge (slow — scans all files for schema)
df = spark.read \
    .option("mergeSchema", "true") \
    .parquet("s3://data/events/")

# Better: use Delta Lake with schema evolution
spark.conf.set("spark.databricks.delta.schema.autoMerge.enabled", "true")
new_df.write.format("delta").mode("append").save("s3://lake/events/")

# Explicit schema definition (best for production)
from pyspark.sql.types import StructType, StructField, StringType, LongType

schema = StructType([
    StructField("id", LongType(), nullable=False),
    StructField("name", StringType(), nullable=True),
    StructField("amount", LongType(), nullable=True),
])
df = spark.read.schema(schema).json("s3://landing/events/")

44. What is checkpointing in Spark?

Checkpointing saves the RDD/DataFrame to a reliable storage (HDFS, S3), breaking the lineage chain.

# Set checkpoint directory
sc.setCheckpointDir("hdfs:///spark/checkpoints")

# Checkpoint an RDD (cuts lineage)
rdd.checkpoint()
rdd.count()   # trigger write

# Use when: lineage becomes very long (iterative ML), prevents stack overflow

Streaming checkpoint (different):

# Saves streaming state and offsets to recover after failure
query = df.writeStream \
    .option("checkpointLocation", "s3://checkpoints/query1") \
    .start()

45. What is the difference between Spark and Flink?

46. How does Spark handle fault tolerance?

# Enable speculative execution
spark.conf.set("spark.speculation", "true")
spark.conf.set("spark.speculation.multiplier", "1.5")  # task 1.5× median → speculate

# Retries
spark.conf.set("spark.task.maxFailures", "4")

47. What is the difference between saveAsTable and write.parquet()?

48. What are accumulators in Spark?

Accumulators are distributed counters that executors update; only the driver reads the final value.

# Built-in accumulator
error_count = sc.accumulator(0)

def parse_row(row):
    global error_count
    try:
        return parse(row)
    except Exception:
        error_count += 1
        return None

rdd.map(parse_row).filter(lambda x: x is not None).collect()
print(f"Parsing errors: {error_count.value}")

Caveats:

• Accumulators in transformations may run multiple times (retries, speculation) — can double-count
• More reliable inside foreach / actions
• Don't use as primary logic — only for monitoring/diagnostics

49. How do you read and write to Kafka in Spark?

# Read from Kafka
df = spark.readStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", "broker1:9092,broker2:9092") \
    .option("subscribe", "orders") \
    .option("startingOffsets", "latest") \
    .option("maxOffsetsPerTrigger", 100_000) \
    .load()

# Parse JSON payload
from pyspark.sql.functions import from_json, col
from pyspark.sql.types import StructType, StructField, StringType, DoubleType

schema = StructType([
    StructField("order_id", StringType()),
    StructField("amount", DoubleType()),
])

parsed = df.select(
    from_json(col("value").cast("string"), schema).alias("data"),
    col("timestamp")
).select("data.*", "timestamp")

# Write back to Kafka
parsed.selectExpr("order_id AS key", "to_json(struct(*)) AS value") \
      .writeStream \
      .format("kafka") \
      .option("kafka.bootstrap.servers", "broker1:9092") \
      .option("topic", "processed-orders") \
      .option("checkpointLocation", "s3://checkpoints/kafka-sink") \
      .start()

50. What are common Spark interview gotchas?

Common mistakes

Apache Spark vs alternatives

FAQ

Q: What is the best way to learn Spark for interviews?

A: Practice with real PySpark code on a local cluster (local[*]). Understand the Spark UI — interviewers love when you can explain performance issues by reading execution plans. Focus on DataFrames/Spark SQL rather than RDDs (most modern Spark is DataFrame-based). Study AQE, shuffle tuning, and broadcast joins.

Q: When should you use Spark instead of pandas?

A: Use Spark when your data doesn't fit in a single machine's memory (typically > 10–20 GB), when you need distributed processing across a cluster, or when you need streaming. For small data, pandas is simpler and faster (no cluster overhead).

Q: What is the difference between Databricks and Apache Spark?

A: Databricks is a managed cloud platform (AWS/Azure/GCP) built on top of open-source Spark. It adds Delta Lake, Unity Catalog (data governance), MLflow (ML tracking), collaborative notebooks, and auto-scaling clusters. Open-source Spark is the engine; Databricks is the managed service.

Q: How does Spark handle a large join between two big tables?

A: Spark uses a sort-merge join: both sides are sorted by the join key, then merged in parallel. This requires two shuffles (one per side). To avoid the shuffle: use bucketing (pre-partition both tables by join key) or apply predicate pushdown to make one side small enough for a broadcast join.

Q: What is the difference between count() and countDistinct()?

A: count("*") counts all rows including nulls; count(col) excludes nulls. countDistinct(col) counts distinct non-null values — it requires a shuffle (all data for a column must be compared). For approximate distinct counts at scale, use approx_count_distinct(col, rsd=0.05) which uses HyperLogLog.

Q: What checkpointing strategy should I use for production Structured Streaming jobs?

A: Always use a reliable storage like S3 or HDFS (not local filesystem). Use a unique checkpoint directory per query. Implement monitoring for query lag (query.lastProgress). Set maxOffsetsPerTrigger on Kafka sources to control throughput. Test restart recovery in staging before production.