Kafka Consumer Groups and Partitions

By Kokil Thapa | Last reviewed: September 2026

You cannot scale a Kafka pipeline by adding pods alone. A kafka consumer group only parallelises work up to the partition count of the topic it reads. Run eight consumers against four partitions and four instances sit idle. They still heartbeat and join rebalances, but they process zero records. This guide explains partition assignment, idle-consumer math, rebalance behaviour, offset commits, and lag diagnosis so you can design streams that actually scale in production.

If you are wiring PHP backends into event pipelines, start with solid API boundaries via RESTful API design in Laravel. The distributed-systems rules stay the same regardless of language. For broader context on when Kafka fits your stack, see RabbitMQ vs Kafka compared and the Apache Kafka fundamentals overview.

What is a kafka consumer group and how does it map to partitions?

Every consumer instance joins a group identified by group.id. The group coordinator tracks membership and assigns partitions. The core rule is strict: within one group, each partition belongs to exactly one consumer. Two consumers never read the same partition at the same time.

That rule preserves message order for a given key. Records with the same key hash to the same partition. A single consumer processes them sequentially. Scale out by adding partitions and consumers, not by duplicating readers on one partition.

Groups are independent. Two groups subscribed to the same topic each receive every message. Offsets are stored per group in the internal __consumer_offsets topic. This pattern feeds analytics and search indexing from one stream without coupling throughput.

Partition assignment uses pluggable strategies. The default range assignor groups partitions by topic. Round-robin spreads them evenly. Sticky assignors minimise movement during rebalances. Pick a strategy that matches your deployment pattern and stick with it across all group members.

What happens when a kafka consumer group has more consumers than partitions?

This is the most searched pitfall around consumer groups, and the answer is blunt. Idle consumers are expected behaviour, not a bug. If a topic has P partitions and your group runs C consumers where C > P, then C − P consumers do no fetch work.

Those idle instances still join the group. They send heartbeats to the coordinator. They participate in rebalances when membership changes. They consume CPU and memory on your cluster. They just never poll records from any partition.

Teams often discover this after scaling Kubernetes replicas to match traffic that never arrives. HPA adds pods because CPU looks low. Each new pod joins the consumer group. Nothing speeds up because partition count is unchanged. The fix is never "add more consumers." The fix is "add more partitions" or "accept current parallelism."

The idle consumer formula

Use this before every scale-out decision:

• Active consumers = min(consumer instances, partition count)
• Idle consumers = max(0, consumer instances − partition count)
• Max throughput ceiling = partition count × per-partition processing rate

Example: six partitions, ten pods. Four pods sit idle forever unless you repartition. Repartitioning is possible but operationally heavy. It triggers rebalances, may reorder keys across partitions, and cannot be reversed without recreating the topic.

On production booking systems I have maintained, matching replica count to partition count eliminated wasted cloud spend. A three-partition topic runs three consumers for HA, not six "just in case." The fourth replica handles a different role or waits as a standby outside the group.

Verify assignment at runtime with the CLI shipped in Kafka distributions:

bin/kafka-consumer-groups.sh \
  --bootstrap-server broker:9092 \
  --group order-processor \
  --describe

# Look at PARTITION, CURRENT-OFFSET, LOG-END-OFFSET, LAG, CONSUMER-ID
# Unassigned consumers show no partition rows

For event-driven microservice design patterns, see event-driven microservices with Kafka. If you are comparing monolith extraction paths, monolith-to-microservices migration strategy covers when streaming pays off.

What triggers a kafka consumer group rebalance and how do you reduce disruption?

A rebalance reassigns partition ownership across group members. It fires when a consumer joins, leaves gracefully, fails a heartbeat, exceeds max.poll.interval.ms, or when partition count changes. During a classic eager rebalance, all consumers stop fetching. They revoke assignments and wait for the coordinator.

That stop-the-world window causes latency spikes during rolling deploys. On a legal-tech portal processing document events, naive deploys created 15–30 second gaps every release. Cooperative sticky assignment, available since Kafka 2.4, fixes most of this pain.

Eager vs cooperative sticky rebalancing

Eager mode revokes every partition from every consumer before reassignment. Cooperative sticky mode only revokes partitions that must move. Stable partitions keep fetching throughout the rebalance.

# Java consumer
spring.kafka.consumer.properties.partition.assignment.strategy=\
org.apache.kafka.clients.consumer.CooperativeStickyAssignor

# librdkafka / PHP-RDKafka
$conf->set('partition.assignment.strategy', 'cooperative-sticky');

# Tune timeouts to avoid false evictions
session.timeout.ms=45000
heartbeat.interval.ms=15000
max.poll.interval.ms=300000

Handle the revocation callback correctly. Flush uncommitted offsets for revoked partitions before returning from onPartitionsRevoked. The next owner may otherwise reprocess records or skip ahead depending on commit timing.

Official protocol details live in the Apache Kafka consumer configuration documentation. For Kubernetes-hosted clusters, running Kafka on Kubernetes with Strimzi covers operator-level partition management.

How should you choose partition count for a kafka consumer group?

Partition count sets the hard ceiling on parallelism. You cannot scale a single group beyond it without repartitioning. Choose the number upfront using measured throughput, not guesswork.

A practical starting point for domain events—orders, signups, booking updates—is 6–12 partitions. That matches typical three-to-six pod deployments with headroom. High-throughput clickstreams may need 24–64. Avoid hundreds of partitions unless benchmarks prove you need them. Excess partitions inflate broker metadata, slow leader election, and lengthen consumer startup.

Producer-side key choice matters as much as partition count. Use a stable business key—order ID, user ID, tenant ID—when order within that key is required. Use null keys or round-robin when order does not matter and you want even spread.

Align worker topology with partitions as covered in scaling Laravel background jobs for high-traffic applications. For enterprise pipeline design help, see enterprise application development services.

How do offsets and delivery semantics work within a consumer group?

Offsets are tracked per partition per group in __consumer_offsets. A consumer crash only affects replay scope for partitions it owned. Commit strategy determines whether you get at-most-once, at-least-once, or effectively exactly-once behaviour.

• Auto-commit (default): Timer-based commits regardless of processing state. Crashes between poll and commit cause duplicates. Avoid for money or legal records.
• Manual sync commit: Call commitSync() after each batch. At-least-once with added broker round-trip latency.
• Manual async commit: Higher throughput via commitAsync(). Commits may arrive out of order; pair with idempotent handlers.
• Transactional EOS: Idempotent producer plus consume-transform-produce within Kafka boundaries. Requires careful isolation.level config.

For Nepal payment integrations—eSewa, Khalti, ConnectIPS—I mandate manual commits with idempotency keys in PostgreSQL. Network drops make auto-commit unsafe for confirmation events. The same rule applies to duplicate legal filings or double webhook charges.

# Safe at-least-once pattern (pseudo-flow)
records = consumer.poll(timeout)
db.transaction(function () use ($records) {
    foreach ($records as $record) {
        if (IdempotencyStore::seen($record->key())) continue;
        processBusinessLogic($record);
        IdempotencyStore::mark($record->key());
    }
    consumer.commitSync(); // after DB commit succeeds
});

Webhook reliability patterns from webhook design for reliability and third-party API retry and backoff apply directly to consumer error handling. Test payload shapes with the JSON formatter tool before wiring handlers.

How do you diagnose consumer lag and partition skew?

Consumer lag is the gap between log-end offset and committed offset per partition. Rising lag means processing cannot keep pace with production. Root causes vary, so measure per partition—not just group totals.

1. Hot partition skew: One key dominates traffic. Seven consumers idle while one drowns. Audit key selection; split hot keys or accept weaker ordering.
2. Slow downstream I/O: Database calls or HTTP inside the poll loop bottleneck throughput. Profile batch duration; offload heavy work to downstream stages.
3. Rebalance storms: Short session.timeout.ms or GC pauses trigger churn. Raise timeout to 45s+ and fix pause sources.
4. Small fetch batches: Default fetch.min.bytes=1 wastes round trips. Try 10KB–100KB with fetch.max.wait.ms=500.
5. Ignored backpressure: Unbounded in-memory queues hide lag until OOM. Pause polling when internal buffers exceed thresholds.

Export per-partition lag to Prometheus via the Kafka exporter or Burrow. Alert on sustained p95 lag over five-minute windows, not single spikes. Legitimate catch-up after downtime creates temporary spikes—that is normal.

Correlate lag with application logs using approaches from AI-assisted debugging workflows. Full-stack observability guidance lives in observability for microservices and Prometheus and Grafana monitoring setup.

Circuit breaking on downstream failures, as described in circuit breakers and resilience patterns, prevents one slow dependency from stalling an entire partition consumer indefinitely.

Real-world event pipelines appear in projects like Adventure Third Pole Trek booking platform, where async order processing must stay reliable under seasonal traffic spikes.

Key Takeaways

• A kafka consumer group assigns each partition to exactly one consumer; max parallelism equals partition count, never consumer count.
• When consumers exceed partitions, the surplus stays idle—scale partitions first, not pods alone.
• Use cooperative-sticky assignment and static membership to cut deploy-time rebalance stalls.
• Disable auto-commit for financial, legal, or payment events; commit offsets only after idempotent processing succeeds.
• Monitor lag per partition to catch hot-key skew before one consumer becomes a bottleneck.
• Match Kubernetes replica counts to partition topology to avoid wasted compute and false confidence in throughput.

People Also Ask

Can multiple consumer groups read the same Kafka topic?

Yes. Each consumer group maintains independent offsets. Ten groups on one topic each receive every message. This is how you fan out to analytics, search indexing, and operational dashboards from a single event stream without coupling their processing speed.

Why do I have idle consumers in my Kafka consumer group?

Idle consumers appear when group membership exceeds partition count. Kafka assigns at most one consumer per partition within a group. Extra instances join heartbeats and rebalances but fetch no records. Add partitions or reduce replica count to match.

Does increasing partitions affect message ordering?

Ordering is guaranteed only within a single partition. Adding partitions redistributes key hashes. Messages with the same key stay ordered only if they still land on the same partition after repartitioning. Plan partition growth before strict ordering requirements harden.

What is the difference between consumer lag and consumer group lag?

Consumer lag is measured per partition—the offset gap for that shard. Group lag is the sum or max across all assigned partitions. A healthy group can show zero total lag while one hot partition lags badly, which is why per-partition metrics matter.

Build Event Pipelines That Scale Predictably

A kafka consumer group is not magic scaling dust. It is a contract: partitions define the ceiling, assignment defines ownership, and commits define delivery guarantees. Size partitions for realistic growth, keep consumer count aligned, adopt cooperative rebalancing, and treat lag as a first-class metric. Those choices separate prototypes from systems that survive production traffic.

Need help designing partition topology, integrating Kafka with Laravel, or hardening payment event pipelines for Nepal deployments? Contact us to discuss your architecture. For direct technical questions, you can also reach out about your specific requirements. Explore API development services or read more on the engineering blog.