Apache Kafka Fundamentals

By Kokil Thapa | Last reviewed: September 2026

Apache Kafka Fundamentals start with a simple idea: treat business events as an append-only log, not a transient message that disappears after delivery. Orders placed, payments confirmed, documents uploaded, and inventory adjusted all become durable records that multiple services can read at their own pace. If you build REST APIs and webhook-driven integrations, Kafka sits one layer below that surface—it is the backbone when volume, replay, and fan-out outgrow a single database or a basic queue. This guide walks through the core concepts a working engineer needs before wiring Kafka into a Laravel app, a microservice fleet, or a data pipeline.

What are the core components of Apache Kafka Fundamentals?

Kafka is a distributed commit log marketed as an event streaming platform. Unlike a traditional message broker that deletes a message once a consumer acknowledges it, Kafka retains records for a configurable period. That retention window is what makes replay, audit trails, and multiple independent consumers practical.

The five primitives you must internalize are:

• Topic — a named category of events, such as orders.created or payments.settled.
• Partition — an ordered, immutable sequence of records inside a topic; parallelism lives here.
• Broker — a Kafka server that stores partition data and serves read/write requests.
• Producer — a client that publishes records to a topic partition.
• Consumer — a client that reads records and tracks its position via an offset.

A Kafka cluster is two or more brokers working together. Cluster metadata— which broker leads which partition—lives in an internal topic called __consumer_offsets and, in modern deployments, is managed by KRaft (Kafka Raft) rather than an external ZooKeeper ensemble. For greenfield clusters in 2026, plan on KRaft mode; ZooKeeper-based installs are legacy.

Official reference material lives in the Apache Kafka documentation. Read the "Introduction" and "Design" sections before you touch production sizing.

How do Kafka topics and partitions work?

A topic is a logical stream. Partitions are the unit of storage, ordering, and parallelism. Records in a single partition are strictly ordered. Records across partitions have no global order unless you design for it.

Choosing a partition key

Producers assign each record to a partition. If you supply a key—such as customer_id—Kafka hashes the key and routes all events for that customer to the same partition. That gives you per-customer ordering. If you omit a key, records round-robin across partitions, which maximizes throughput but sacrifices ordering guarantees.

// Conceptual producer record (Java client style)
ProducerRecord<String, String> record =
    new ProducerRecord<>("orders", order.getCustomerId(), order.toJson());

Replication and the leader-follower model

Each partition has one leader broker that handles reads and writes. Follower brokers replicate the log for fault tolerance. The replication factor—often three in production—defines how many copies exist. Writes succeed when the leader and enough followers acknowledge according to your acks setting.

Partition count is chosen at topic creation. You can add partitions later, but you cannot reduce them without rebuilding the topic. A common mistake is creating too few partitions and hitting a throughput ceiling six months later.

For deeper coverage of how consumer groups map to partitions, see the companion piece on Kafka consumer groups and partitions.

How does a Kafka producer publish events reliably?

Producers batch records for efficiency. They retry on transient failures. They can enforce ordering within a partition by setting max.in.flight.requests.per.connection=1 when idempotence matters, though the idempotent producer (enabled via enable.idempotence=true) handles most duplicate concerns automatically in modern clients.

The acks ladder

1. acks=0 — fire-and-forget; fastest, no durability guarantee.
2. acks=1 — leader acknowledges; risk of loss if leader dies before replication.
3. acks=all — wait for all in-sync replicas; standard for money-moving or compliance events.

# docker-compose excerpt — local Kafka 3.x with KRaft (illustrative)
services:
  kafka:
    image: apache/kafka:3.9.0
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1

Create a topic and produce a test message from the CLI shipped with Kafka:

bin/kafka-topics.sh --create --topic orders \
  --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

bin/kafka-console-producer.sh --topic orders --bootstrap-server localhost:9092
> {"order_id": 1001, "status": "created"}

In Laravel-centric architectures, producers often live in the app layer or a sidecar service. The app validates business rules, writes to MySQL inside a transaction, then publishes an outbox event—a pattern that avoids "database committed but Kafka message lost" races. That design aligns with how enterprise application development teams treat transactional boundaries.

How do Kafka consumer groups and offsets keep readers in sync?

Consumers never delete data from the log. They commit an offset—a pointer to the next record they will read. Offsets are stored in the internal __consumer_offsets topic (or externally via Kafka Connect offset stores for specialized setups).

Consumer group rules

Consumers sharing the same group.id coordinate partition assignment. Each partition is consumed by at most one consumer in the group at a time. If you run three consumers in a group against a topic with six partitions, each consumer typically gets two partitions. Add a seventh consumer and one sits idle—partitions are the upper bound on parallelism.

Different consumer groups are independent. A fraud-detection service and a warehouse analytics job can both read the same topic from the beginning or from "now," each tracking its own offsets. That fan-out model is why Kafka beats a point-to-point queue when many downstream systems need the same event stream.

Commit strategies

Auto-commit is convenient for prototypes. It commits offsets on an interval and can double-process records after a crash. Manual commit after successful side effects—database write, email send, payment capture—is the production default. The rule is simple: commit only after your downstream work succeeds, or design consumers to be idempotent.

Compare this mental model with RabbitMQ in the RabbitMQ vs Kafka guide. RabbitMQ excels at task queues and competing consumers with acknowledgment-based removal. Kafka excels at high-volume logs, replay, and many subscribers.

When should you choose Kafka over Redis queues or Laravel Horizon?

Not every project needs Kafka. On a typical Laravel 12 or 13 deployment, Redis-backed queues with Horizon handle order emails, image resizing, and webhook retries cleanly. I've used that stack on production apps including booking and eCommerce systems where throughput stays in the hundreds of jobs per minute.

Kafka earns its operational cost when you hit one or more of these triggers:

• Event volume exceeds comfortable Redis memory or single-broker limits.
• Multiple teams need the same event stream with independent consumption speeds.
• You must replay history after a bug fix or a new analytics pipeline launch.
• Stream processing—aggregations, joins, windowed counts—is a first-class requirement.
• Audit and compliance require an immutable event trail beyond database binlogs.

For a digital grocery platform with delivery-zone logic, Redis queues often suffice until a dedicated analytics or partner-integration layer appears. At that point, publishing order.placed events to Kafka while keeping Laravel Horizon for operational jobs is a sane split.

How do you operate and monitor Kafka in production?

Kafka's durability does not remove the need for observability. Treat broker disk, under-replicated partitions, and consumer lag as paging-worthy signals. Consumer lag is the gap between the log end offset and a group's committed offset. Rising lag means downstream processing cannot keep pace.

Metrics that matter

• Under-replicated partitions — replication is unhealthy; investigate broker loss or network partitions.
• Offline partitions — no leader available; producers and consumers block for those shards.
• Request handler idle ratio — sustained low idle on brokers signals CPU or disk saturation.
• Consumer lag by group — business-visible backlog; tie alerts to SLA thresholds.

Pair Kafka with a metrics stack described in Prometheus monitoring fundamentals. Export JMX metrics via the Kafka exporter or Confluent-compatible agents. Log correlation through request IDs in event payloads simplifies tracing from an HTTP request to a published record.

Schema governance

Raw JSON in topics works until two services disagree on field types. Avro, Protobuf, or JSON Schema with a registry—Confluent Schema Registry or an open-source equivalent—enforces compatibility rules. Set BACKWARD or FULL compatibility before multiple producers share a topic. Breaking schema changes without a migration plan are a common source of silent consumer failures.

Security basics for production clusters include TLS encryption on the wire, SASL authentication, and ACLs limiting which principals can read or write each topic. Never expose plaintext broker ports to the public internet. On Ubuntu servers I administer, Kafka listeners sit behind private network interfaces with firewall rules similar to those used for MySQL and Redis—patterns covered under Linux system administration.

Payload debugging during integration work often starts with structured JSON. A JSON formatter helps validate sample events before you register a schema.

How do web applications integrate with Kafka safely?

Most PHP and Laravel teams do not embed a Kafka client inside every request cycle. Long-lived consumers run as separate worker processes—Supervisor, systemd, or Kubernetes Deployments—while HTTP controllers publish asynchronously.

Transactional outbox pattern

1. Begin a database transaction.
2. Insert or update business rows.
3. Insert a row into an outbox_events table with the serialized payload.
4. Commit the transaction.
5. A relay process reads the outbox and publishes to Kafka, marking rows as sent.

This mirrors reliable webhook delivery patterns: persist first, deliver second, retry with backoff on failure. The difference is the transport. Webhooks push to one URL; Kafka retains for many readers.

Rate limiting and abuse prevention on the HTTP edge still matter. See API rate limiting and abuse prevention for the ingress layer while Kafka handles asynchronous fan-out behind it.

For a shipped example of high-throughput order flows in Laravel, review the Quick And Easy Nepalese Grocery portfolio case. Event streaming was not the first tool chosen there; queues and solid database design carried the load until scale demanded more.

Stream processing frameworks—Kafka Streams, Apache Flink, ksqlDB—sit adjacent to core Kafka. They consume topics, compute aggregates, and write derived topics. Treat them as phase-two architecture unless you have a concrete windowed metric requirement on day one.

Related infrastructure topics on this site include Envoy proxy fundamentals for ingress routing and Ceph storage fundamentals when broker disks need a shared block layer across nodes.

Key Takeaways

• Kafka is a durable, partitioned commit log—not a traditional delete-on-ack queue.
• Partition count sets your parallelism ceiling; choose keys to preserve ordering where it matters.
• Use acks=all and replication factor 3 for production events you cannot afford to lose.
• Consumer groups scale readers; offsets must commit only after successful side effects.
• Start with Redis or RabbitMQ for modest Laravel workloads; adopt Kafka when replay and multi-subscriber fan-out become requirements.
• Monitor consumer lag and under-replicated partitions; pair brokers with Prometheus-style metrics and schema registry governance.

People Also Ask

What is the difference between a Kafka topic and a partition?

A topic is the named stream—such as user.signups. Partitions are the physical shards inside that topic, each an ordered log. Producers write to specific partitions; consumers read from assigned partitions within their group. More partitions mean more parallel readers, up to the number of consumers in a group.

Does Kafka delete messages after consumers read them?

No. Consumers advance an offset pointer; records remain until retention policy deletes them by time or size. That design enables replay, late-joining consumers, and audit use cases. If you need automatic removal after processing, a message queue like RabbitMQ is often a better fit.

What is KRaft in Apache Kafka?

KRaft (Kafka Raft) is Kafka's built-in metadata quorum that replaces Apache ZooKeeper for controller duties. New clusters in 2026 should deploy in KRaft mode for simpler operations and faster metadata recovery. Legacy ZooKeeper-based clusters still exist but are on a deprecation path—check the version-specific migration guide on kafka.apache.org before upgrading.

How many brokers do you need for a production Kafka cluster?

Three brokers is the common minimum when replication factor is three and you want to survive a single broker loss without losing write availability. Smaller teams sometimes run two brokers for non-critical staging environments. Production also demands separate disks for log directories, monitored free space, and a clear backup strategy for topic data you cannot rebuild from upstream sources.

Build event-driven systems with the right foundation

Apache Kafka Fundamentals boil down to a durable log, smart partitioning, and consumer groups that scale independently. Master those ideas before you provision a three-broker cluster or wire schema registry policies. Match the tool to the workload—Redis queues for operational jobs, Kafka when history, fan-out, or stream analytics become non-negotiable.

If you are planning an event-driven platform, API layer, or migration from synchronous webhooks to a stream backbone, contact us or explore custom software development services to scope architecture that fits your team size and budget. For ongoing broker tuning and observability, support and maintenance keeps production lag dashboards green after launch.