50 Apache Kafka Interview Questions (With Answers)

Apache Kafka interviews test your understanding of distributed event streaming — from core architecture and partitioning to exactly-once semantics, consumer groups, and Kafka Streams. This guide covers the 50 most common questions with concise, accurate answers.

Quick reference

Core Architecture

1. What is Apache Kafka and what problems does it solve?

Apache Kafka is a distributed event streaming platform designed for high-throughput, fault-tolerant, publish-subscribe messaging.

Problems it solves:

Core use cases: event sourcing, log aggregation, CDC (change data capture), stream processing, metrics pipelines, activity tracking.

2. Describe Kafka's architecture.

Producers → [Topic: order-events]
               Partition 0  [P0] ─── Broker 1 (Leader) ─── Broker 2 (Follower)
               Partition 1  [P1] ─── Broker 2 (Leader) ─── Broker 3 (Follower)
               Partition 2  [P2] ─── Broker 3 (Leader) ─── Broker 1 (Follower)
                                          ↓
                                   Consumer Group A
                                   (one consumer per partition)

Key components:

3. What is a Kafka partition and why does it matter?

A partition is an ordered, append-only log of records. Each record has an offset — an immutable sequential ID.

Why partitions matter:

• Parallelism — multiple consumers in a group can read in parallel; one partition per consumer
• Throughput — data is striped across brokers; more partitions = more I/O parallelism
• Ordering — ordering is guaranteed within a partition, not across partitions
• Scalability — increasing partitions lets you scale consumers horizontally

Rule of thumb: start with max(target throughput / per-consumer throughput, num brokers) partitions. More partitions have costs: more file handles, longer leader election.

4. How does Kafka achieve fault tolerance?

Through replication. Each partition has one leader and zero or more followers (replicas).

Topic: orders  replication.factor=3
  Partition 0: Leader=Broker1, Followers=[Broker2, Broker3]
  Partition 1: Leader=Broker2, Followers=[Broker3, Broker1]

• ISR (In-Sync Replicas) — set of replicas fully caught up with the leader
• Leader handles all reads and writes; followers replicate asynchronously
• If a leader fails, one ISR follower is elected as new leader
• min.insync.replicas — minimum ISR size required for a produce to succeed (prevents data loss)

5. What is the difference between ZooKeeper mode and KRaft mode?

KRaft uses the Raft consensus algorithm for controller quorum. The transition from ZooKeeper reduces operational overhead significantly.

Producers

6. How does a Kafka producer work?

Application → ProducerRecord(topic, key, value)
    → Serializer (key + value to bytes)
    → Partitioner (which partition?)
    → RecordAccumulator (batch buffer)
    → Sender thread → Broker leader
    ← Acknowledgement (RecordMetadata or exception)

Key producer configs:

7. What are producer acknowledgement levels (acks)?

The acks config controls durability vs throughput:

Production recommendation: acks=all + min.insync.replicas=2 + replication.factor=3.

8. How does Kafka partition messages?

The partitioner decides which partition a record goes to:

1. Explicit partition — producer sets partition field → used directly
2. Key-based — murmur2(key) % numPartitions → same key always goes to same partition (ordering guarantee)
3. No key — default partitioner uses sticky partitioning (fill one batch at a time, then switch) since Kafka 2.4; previously round-robin

When to use keys: order events by customer ID, user actions by user ID, related events that must arrive in order.

9. What is idempotent producer and why use it?

With default settings, retries can cause duplicate messages if the broker received the record but the ack was lost.

Idempotent producer (enable.idempotence=true) assigns each producer a PID (Producer ID) and sequence number per partition. The broker deduplicates retried records.

enable.idempotence=true
acks=all               # automatically set
retries=MAX_INT        # automatically set
max.in.flight.requests.per.connection=5  # max for idempotency

Result: exactly-once delivery within a single partition from a single producer session.

10. What is a transactional producer?

Extends idempotency to atomic writes across multiple partitions (and topics).

producer.initTransactions();
producer.beginTransaction();
producer.send(new ProducerRecord<>("topic-a", key, value));
producer.send(new ProducerRecord<>("topic-b", key, value));
producer.commitTransaction(); // or producer.abortTransaction();

Requires transactional.id config. Enables:

• Atomic multi-partition writes
• Exactly-once read-process-write loops (with consumer isolation.level=read_committed)

Consumers

11. How does a Kafka consumer work?

Consumer → poll(Duration timeout)
    → Fetch request to partition leaders
    ← Batch of records
    → Process records
    → commitSync() / commitAsync()  (commits offset)

The consumer pulls records at its own pace. Broker does not push.

12. What is a consumer group?

A consumer group is a set of consumers that jointly consume a topic:

• Each partition is assigned to exactly one consumer in the group
• Multiple groups can independently read the same topic (pub-sub)
• Groups are identified by group.id

Topic: orders (3 partitions)
Consumer Group A (3 consumers): C1→P0, C2→P1, C3→P2
Consumer Group B (1 consumer):  C4→P0,P1,P2  (one consumer handles all)

If consumers > partitions, extra consumers are idle.

13. What is a consumer group rebalance?

A rebalance redistributes partition assignments when:

• A consumer joins the group
• A consumer leaves or crashes (detected via heartbeat timeout)
• Partitions are added to the topic
• subscribe() topic list changes

Rebalance protocols:

Cooperative rebalancing (default since Kafka 3.1 with CooperativeStickyAssignor) greatly reduces downtime.

14. How does offset management work?

An offset is the position of the next record to read in a partition.

Offsets are stored in the internal __consumer_offsets topic.

// Auto-commit (every auto.commit.interval.ms = 5000ms)
props.put("enable.auto.commit", "true");

// Manual commit - more control
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
process(records);
consumer.commitSync(); // blocks until committed
// or
consumer.commitAsync(); // non-blocking, callback on completion

15. What is the difference between seekToBeginning and resetting auto.offset.reset?

auto.offset.reset only triggers on first read or after offset expiry; seek* methods override at runtime.

16. What causes consumer lag and how do you monitor it?

Consumer lag = latest offset (end) − committed offset (current position). High lag means consumers are falling behind producers.

# Check consumer group lag
kafka-consumer-groups.sh --bootstrap-server broker:9092 \
  --describe --group my-group

# Output
TOPIC        PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
order-events 0          1050            1200            150
order-events 1          2000            2000            0

Monitoring: export records-lag-max JMX metric to Prometheus/Grafana. Alert when lag exceeds threshold.

Causes: slow consumer processing, GC pauses, network issues, under-partitioned topic.

Replication & Durability

17. What is ISR (In-Sync Replicas)?

The ISR is the set of replicas that are fully caught up with the leader (within replica.lag.time.max.ms).

• A replica falls out of ISR if it's too far behind
• Leader waits for all ISR replicas to acknowledge a write when acks=all
• If ISR shrinks below min.insync.replicas, producers with acks=all get NotEnoughReplicasException

Safety: replication.factor=3, min.insync.replicas=2 → tolerate 1 broker failure without data loss.

18. What is unclean leader election?

By default, only ISR members can be elected as leader. If the ISR is empty (all replicas are down), the partition is unavailable.

unclean.leader.election.enable=true allows an out-of-sync replica to become leader → data loss risk but improved availability.

Recommendation: keep false for financial/critical data; true only when availability > durability.

19. How does Kafka persist data to disk?

Kafka uses a segmented log per partition:

/data/kafka-logs/orders-0/
  00000000000000000000.log    ← segment (messages)
  00000000000000000000.index  ← offset index
  00000000000000000000.timeindex ← timestamp index
  00000000000001048576.log    ← next segment (after roll)

• Messages are appended sequentially (fast disk writes, no random I/O)
• Reads use sendfile() system call (zero-copy) to transfer from page cache to network
• Segments are rolled when log.segment.bytes (1 GB) or log.roll.ms is reached
• Old segments are deleted when log.retention.bytes or log.retention.hours is exceeded

Delivery Semantics

20. What are at-most-once, at-least-once, and exactly-once semantics?

Implementation for EOS end-to-end:

1. Producer: enable.idempotence=true + transactional.id
2. Consumer: isolation.level=read_committed
3. Within Kafka Streams: processing.guarantee=exactly_once_v2

21. How do you achieve exactly-once in a read-process-write loop?

producer.initTransactions();
while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    producer.beginTransaction();
    for (ConsumerRecord<String, String> r : records) {
        // Process and produce to output topic
        producer.send(new ProducerRecord<>("output-topic", r.key(), transform(r.value())));
    }
    // Commit consumer offsets as part of the same transaction
    producer.sendOffsetsToTransaction(
        getOffsets(records),
        new ConsumerGroupMetadata(consumer.groupMetadata())
    );
    producer.commitTransaction();
}

The consumer offsets are written atomically with the output records → no partial processing on restart.

Log Compaction & Retention

22. What is log compaction?

Log compaction keeps the last value for each key, discarding old records with the same key.

Before compaction:
offset 0: key=user-1, value={"name":"Alice"}
offset 1: key=user-2, value={"name":"Bob"}
offset 2: key=user-1, value={"name":"Alice Smith"}  ← newer value for user-1

After compaction:
offset 1: key=user-2, value={"name":"Bob"}
offset 2: key=user-1, value={"name":"Alice Smith"}  ← offset 0 removed

Use cases: CDC (change data capture), user preferences, KTable source topics.

Configured with cleanup.policy=compact (or compact,delete for both).

23. What is the difference between delete and compact retention policies?

Kafka Streams

24. What is Kafka Streams?

Kafka Streams is a Java library (not a cluster) for stream processing directly on top of Kafka. No separate processing cluster is needed.

StreamsBuilder builder = new StreamsBuilder();
KStream<String, Order> orders = builder.stream("orders");

KStream<String, Invoice> invoices = orders
    .filter((k, v) -> v.getAmount() > 100)
    .mapValues(v -> new Invoice(v.getId(), v.getAmount() * 1.2));

invoices.to("invoices");

KafkaStreams streams = new KafkaStreams(builder.build(), props);
streams.start();

Key abstractions:

25. What is the difference between KStream and KTable?

26. How does windowing work in Kafka Streams?

Windows group records by time for aggregations:

orders.groupByKey()
    .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(1)))
    .count()
    .toStream()
    .to("order-counts-per-minute");

27. What is a state store in Kafka Streams?

State stores persist local state (backed by RocksDB) needed for stateful operations (joins, aggregations, count).

• Changelog topic in Kafka backs the store → fault-tolerant (rebuild on restart)
• Interactive queries allow other microservices to query local state via REST
• Standby replicas (num.standby.replicas) keep a warm copy on another instance

// Access state store
ReadOnlyKeyValueStore<String, Long> store =
    streams.store(StoreQueryParameters.fromNameAndType(
        "my-store", QueryableStoreTypes.keyValueStore()));
Long count = store.get("user-1");

28. What is processing.guarantee in Kafka Streams?

exactly_once_v2 uses one producer per task, transactional producers, and read_committed isolation.

Kafka Connect

29. What is Kafka Connect?

Kafka Connect is a framework for streaming data between Kafka and external systems (databases, object stores, search engines) without writing custom code.

Source Connector → [Kafka Topic] → Sink Connector
  (MySQL CDC)        (orders)       (Elasticsearch)

Modes: standalone (single process) and distributed (cluster, config stored in Kafka).

Popular connectors: Debezium (CDC), JDBC Source/Sink, S3 Sink, Elasticsearch Sink, MongoDB Source/Sink.

30. What is Schema Registry and why is it used?

Schema Registry (Confluent) stores and manages Avro/Protobuf/JSON Schema schemas. Producers register schemas; consumers look them up.

Benefits:

• Schema evolution with compatibility checks (backward, forward, full)
• Compact encoding — Avro binary is smaller than JSON
• Contract enforcement — rejects messages breaking the schema

Producer → serialize with schema ID → [byte prefix: 0x00 + schema_id + payload]
Consumer → reads schema ID → fetches schema from registry → deserializes

Kafka Security

31. What authentication and authorization does Kafka support?

# Grant a user produce access to a topic
kafka-acls.sh --bootstrap-server broker:9092 \
  --add --allow-principal User:alice \
  --operation Write --topic orders

32. What is mTLS in Kafka?

Mutual TLS — both client and broker present certificates:

1. Broker presents its certificate → client verifies
2. Client presents its certificate → broker verifies (authorizes by CN/SAN)

Config: ssl.client.auth=required on the broker, client provides ssl.keystore.* properties.

Performance Tuning

33. How do you increase producer throughput?

34. How do you increase consumer throughput?

• Add consumers (up to partition count) in the same group
• Increase fetch.min.bytes and fetch.max.wait.ms to get larger batches
• Process records in parallel (within a consumer, multi-thread by key)
• Use max.poll.records to control batch size
• Optimise processing code (avoid blocking I/O per record)

35. What is the impact of increasing partition count?

Benefits:

• More parallelism (more consumers in group)
• Higher throughput

Costs:

• More file descriptors per broker
• Longer leader election time
• Rebalances are slower with more partitions
• Cross-partition ordering is not guaranteed

Note: You can only increase partition count; reducing requires recreating the topic (data migration needed).

Operations

36. How does Kafka handle log retention?

Two retention strategies (configurable per topic):

For compacted topics, add cleanup.policy=compact.

37. How do you monitor Kafka in production?

Key JMX metrics to watch:

Tools: Prometheus + Kafka Exporter + Grafana dashboards, Confluent Control Center, Burrow (lag monitoring).

38. What is preferred leader election?

Each partition has a preferred leader (the first replica in the replica assignment list). After a failure and recovery, the original leader may not be reinstated automatically.

# Trigger preferred leader election
kafka-leader-election.sh --bootstrap-server broker:9092 \
  --election-type PREFERRED --all-topic-partitions

auto.leader.rebalance.enable=true does this automatically every leader.imbalance.check.interval.seconds.

39. How do you safely delete a Kafka topic?

# Ensure delete.topic.enable=true in server.properties (default true)
kafka-topics.sh --bootstrap-server broker:9092 --delete --topic old-topic

Risk: consumers with active subscriptions will get errors. Ensure all consumers have stopped or switched topics first.

Kafka Design Patterns

40. What is the outbox pattern with Kafka?

Solves dual-write (write to DB + produce to Kafka atomically):

1. Write event to `outbox` table in SAME DB transaction as business data
2. Debezium CDC connector reads outbox table changes (WAL)
3. Debezium produces events to Kafka
4. Delete (or mark) processed outbox rows

This guarantees no events are lost even if Kafka is temporarily unavailable.

41. What is event sourcing with Kafka?

Event sourcing: the state of an entity is derived by replaying its event history.

Topic: account-events (compacted)
  key=account-1: [Created, Deposited(100), Withdrawn(30), Deposited(50)]
  
Current balance = replay all events: 100 - 30 + 50 = 120

Kafka's log compaction + replayability make it a natural event store. KTables can materialise current state.

42. How would you implement a dead letter queue (DLQ) in Kafka?

try {
    process(record);
    consumer.commitSync();
} catch (ProcessingException e) {
    // Send to DLQ topic with original headers + error metadata
    ProducerRecord<String, String> dlqRecord = new ProducerRecord<>(
        "orders-dlq", record.key(), record.value()
    );
    dlqRecord.headers().add("error-message", e.getMessage().getBytes());
    dlqRecord.headers().add("original-topic", record.topic().getBytes());
    dlqRecord.headers().add("original-offset",
        String.valueOf(record.offset()).getBytes());
    producer.send(dlqRecord);
    consumer.commitSync(); // Commit to move past the bad record
}

DLQ records can be replayed after fixing the processing bug.

43. What is the CQRS + Kafka pattern?

CQRS (Command Query Responsibility Segregation) with Kafka:

Write side:  REST API → Command → Kafka topic → Event Handler → DB
Read side:   Kafka topic → Kafka Streams / Consumer → Read Model (Redis/ES)

Commands produce events to Kafka; consumers update materialised views (read models) optimised for queries. Decouples write and read scalability.

Kafka Streams vs Kafka Consumer vs Flink

44. When to use Kafka Streams vs plain consumer vs Apache Flink?

45. What are Kafka's limitations?

Common Interview Scenarios

46. How would you design a real-time fraud detection system with Kafka?

Transactions → [transactions topic]
                    ↓
           Kafka Streams app
           - Tumbling window (per card, 1 min)
           - Aggregate: count, total amount, distinct merchants
           - Join with KTable: card risk profile
           - Filter: anomaly score > threshold
                    ↓
           [fraud-alerts topic]
                    ↓
           Alert service → block card, notify user

Key design decisions:

• Partition by card_id for ordered processing per card
• KTable for card profiles (compacted topic from CRM)
• Low linger.ms for real-time latency
• exactly_once_v2 to avoid double-alerts

47. How do you handle schema evolution in Kafka?

Using Avro + Schema Registry with compatibility modes:

Strategy:

1. Set BACKWARD compatibility (most common)
2. Always add fields with defaults
3. Never remove required fields; deprecate first, remove in later version
4. Numeric widening (int → long) is safe

48. How would you migrate from one Kafka cluster to another with zero downtime?

Mirror Maker 2 (MirrorMaker2) approach:

Source Cluster → MirrorMaker2 → Target Cluster
                 (replicates topics, offsets, consumer groups)

1. Start MirrorMaker2 to replicate all topics to target cluster
2. Wait for consumer group offset sync (MirrorCheckpointConnector)
3. Redirect consumers to target cluster (update bootstrap.servers)
4. Monitor lag on target until stable
5. Redirect producers to target cluster
6. Stop MirrorMaker2, decommission source

49. What happens when a broker goes down?

1. Leader partitions on the failed broker: controller detects via ZooKeeper/KRaft heartbeat timeout
2. Controller elects new leaders from the ISR of each affected partition
3. Metadata update propagated to all brokers and clients
4. Clients (producer/consumer) refresh metadata and reconnect to new leaders
5. When broker comes back: follower replicas catch up → rejoin ISR → can be re-elected as preferred leader

RPO (Recovery Point Objective): 0 if acks=all + min.insync.replicas=2 (no data loss). RTO (Recovery Time Objective): seconds (metadata refresh + leader election).

50. What is tiered storage in Kafka?

Tiered storage (Kafka 3.6 GA) offloads older log segments to object storage (S3, GCS, Azure Blob) while keeping recent data on local disks.

Broker local disk: hot segments (last 6 hours)
Object storage:    cold segments (older than 6 hours)

Benefits: dramatically reduce broker disk costs; consumers can read historical data from object storage transparently; decouple compute from storage.

Config: remote.log.storage.system.enable=true + configure RemoteStorageManager plugin.

Common mistakes

Kafka vs other messaging systems

FAQ

Q: How many partitions should a topic have? Start with max(target_throughput_MB_s / per_consumer_throughput_MB_s, num_brokers). A common starting point is 6–12 partitions. Avoid over-partitioning (>10k partitions per broker).

Q: Can consumers read from follower replicas? Yes, since Kafka 2.4 with replica.selector.class=RackAwareReplicaSelector. Useful to read from replicas in the same availability zone, reducing cross-AZ data transfer costs.

Q: What is ksqlDB? ksqlDB is a streaming SQL engine built on Kafka Streams. It lets you run SQL-like queries on Kafka topics without writing Java/Scala: CREATE STREAM filtered AS SELECT * FROM orders WHERE amount > 100;

Q: What is the difference between log.retention.hours and log.segment.delete.delay.ms? log.retention.hours controls when a segment is eligible for deletion. log.segment.delete.delay.ms is how long to wait after marking it eligible before actually deleting it (default 60s) — gives other threads time to finish reading.

Q: How do you reset a consumer group offset?

# Reset to earliest (must stop consumers first)
kafka-consumer-groups.sh --bootstrap-server broker:9092 \
  --group my-group --topic orders --reset-offsets \
  --to-earliest --execute

Q: What is the difference between Kafka and a traditional message broker? Traditional brokers (RabbitMQ, ActiveMQ) use a push model and delete messages after consumption. Kafka uses a pull model and retains messages on disk for a configurable period. This enables replay, multiple independent consumer groups, and decoupled producer/consumer scaling.