Stream Processing with Kafka and Flink

By Kokil Thapa | Last reviewed: September 2026

High-volume applications generate events faster than batch jobs can keep up. Stream processing with Kafka and Flink solves that gap by ingesting live data through Apache Kafka and transforming it in Apache Flink before results land in databases, dashboards, or downstream APIs. If you already run a Kafka fundamentals pipeline for logs or webhooks, Flink is the compute layer that turns those topics into business logic. This guide walks through architecture, setup, semantics, and production patterns a working team can deploy in 2026.

What is stream processing with Kafka and Flink?

Kafka holds ordered, partitioned event streams. Flink executes continuous queries over those streams with low latency. Together they form a decoupled pipeline: producers never wait on consumers, and Flink workers scale independently of Kafka brokers.

Think of Kafka as the tape reel and Flink as the factory line reading off it. Web clicks, payment callbacks, IoT readings, and inventory updates all become immutable records in topics. Flink operators map, filter, aggregate, and join those records in real time instead of waiting for nightly ETL.

Teams building event-driven microservices with Kafka often start with simple consumers. When requirements grow—session windows, stream-to-stream joins, complex CEP—Flink replaces hand-rolled consumer loops. That shift mirrors moving from cron scripts to enterprise application job queues, but at millions of events per hour.

How do you set up Kafka and Flink for stream processing?

Start with a three-broker Kafka cluster and a Flink session or application cluster. Docker Compose works for local dev; production teams typically deploy via Kubernetes using operators like Strimzi for Kafka, as covered in our Kafka on Kubernetes with Strimzi guide.

Create Kafka topics with sensible partitioning

Partition count sets your maximum Flink parallelism for a single Kafka source. Match partitions to expected throughput. A rule of thumb: one partition handles roughly 10–30 MB/s depending on message size.

# Create a topic for order events
kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic orders \
  --partitions 12 --replication-factor 3

# Verify
kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders

Configure the Flink Kafka connector

Flink 2.x ships a unified Kafka source in the flink-connector-kafka module. Point it at your bootstrap servers and define a deserialization schema. The official Flink Kafka connector documentation covers version compatibility matrices—verify your Flink and Kafka client versions before upgrading.

// Java DataStream API — OrderEvent from Kafka topic "orders"
KafkaSource<OrderEvent> source = KafkaSource.<OrderEvent>builder()
    .setBootstrapServers("kafka-1:9092,kafka-2:9092")
    .setTopics("orders")
    .setGroupId("flink-order-aggregator")
    .setStartingOffsets(OffsetsInitializer.latest())
    .setValueOnlyDeserializer(new OrderEventDeserializer())
    .build();

DataStream<OrderEvent> orders = env.fromSource(
    source,
    WatermarkStrategy.<OrderEvent>forBoundedOutOfOrderness(Duration.ofSeconds(5))
        .withTimestampAssigner((event, ts) -> event.getEventTime()),
    "kafka-orders-source"
);

Enable checkpointing before any stateful logic

Checkpointing is non-negotiable for production stream processing with Kafka and Flink. Set an interval of 30–120 seconds depending on recovery tolerance and state size.

env.enableCheckpointing(60_000); // every 60 seconds
env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
env.getCheckpointConfig().setMinPauseBetweenCheckpoints(30_000);
env.getCheckpointConfig().setExternalizedCheckpointCleanup(
    ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION
);

Store checkpoints on S3, HDFS, or a shared NFS volume. Without durable checkpoint storage, a JobManager failure forces a full replay from Kafka offsets you may no longer hold.

Run the Flink job

1. Package your job as a fat JAR with all connector dependencies shaded correctly.
2. Submit via flink run -c com.example.OrderJob target/order-job.jar.
3. Confirm the job appears in the Flink Web UI at port 8081.
4. Watch the Kafka consumer lag metric—lag should stay flat under steady load.
5. Validate sink output against a known test event you publish manually.

For PHP or Laravel backends that already enqueue work asynchronously, the mental model is similar to Laravel queues and background jobs—except Kafka retains every message and Flink processes unbounded input with managed state.

How does Flink windowing and state work in Kafka pipelines?

Windowing groups events by time or count so you can compute aggregates like "revenue per minute" or "clicks per session." Event time—when something actually happened—is preferred over processing time, which reflects when Flink read the record.

Watermarks tell Flink how far event time has progressed. A bounded-out-of-orderness strategy of five seconds means Flink waits up to five seconds for late events before closing a window. Tune this against your SLA; payment systems often need tighter bounds than click analytics.

DataStream<OrderEvent> keyed = orders.keyBy(OrderEvent::getCustomerId);

DataStream<OrderSummary> perMinute = keyed
    .window(TumblingEventTimeWindows.of(Time.minutes(1)))
    .allowedLateness(Time.seconds(5))
    .aggregate(new OrderAggregateFunction());

perMinute.addSink(new JdbcSink(...));

Keyed state grows with cardinality. A keyed aggregation on customer ID with ten million active customers demands RocksDB incremental checkpoints and enough TaskManager heap for network buffers—not for holding full state in memory.

Understanding Kafka consumer groups and partitions helps here. Each Flink subtask typically reads a subset of partitions. Rebalancing Kafka partitions without resetting offsets can redistributed load unevenly until the next checkpoint.

Kafka plus Flink vs batch ETL vs message queues — which fits?

Not every workload needs Flink. Small teams processing hundreds of events per minute often succeed with Symfony Messenger, RabbitMQ, or Laravel queues. Flink earns its operational cost above roughly 50k events per minute or when sub-second analytics matter.

Read our RabbitMQ vs Kafka comparison before committing. Kafka optimises for replay and retention; RabbitMQ optimises for task delivery with acknowledgements. Flink sits on top of Kafka, not RabbitMQ, though Flink has RabbitMQ connectors for niche cases.

Redpanda offers a Kafka-compatible API with lower JVM footprint. Flink jobs often run unchanged against Redpanda brokers—useful when broker RAM is constrained on a single-region deployment.

How do you run Kafka and Flink stream processing in production?

Production means monitoring, capacity planning, schema governance, and a rollback path. Treat Flink jobs like database migrations—version them, test on staging topics, and blue-green deploy with savepoints.

Schema management with Avro or Protobuf

JSON is fine for prototypes. Production pipelines use Confluent Schema Registry or Apicurio with Avro schemas. Flink deserializers reference schema IDs embedded in each message. The Apache Kafka documentation explains wire format conventions that registry clients rely on.

Breaking schema changes—removing a required field—will crash deserializers mid-job. Use compatibility modes (backward, forward) and CI checks that validate producer and consumer schemas before deploy.

Monitoring essentials

• Kafka: under-replicated partitions, broker disk usage, request latency p99.
• Flink: checkpoint duration, failed checkpoints, backpressure ratio, records-lag-max per operator.
• Business: sink row counts versus source event counts over sliding windows.

Alert on checkpoint failures twice in a row. One failure might be transient S3 latency; consecutive failures mean state corruption or skewed keys overloading one subtask.

Savepoints for upgrades and rescaling

# Trigger savepoint
flink savepoint <job-id> s3://checkpoints/savepoints/

# Cancel with savepoint and restore with new parallelism
flink run -s s3://checkpoints/savepoints/savepoint-abc \
  -p 24 target/order-job-v2.jar

Rescaling changes how keys map to subtasks. Stateful jobs may need `maxParallelism` set at first deployment—changing it later invalidates savepoint compatibility.

Integrating with Laravel or PHP APIs

Most PHP applications remain request-driven. A common hybrid pattern: Laravel writes domain events to Kafka through a REST proxy or native php-rdkafka extension; Flink aggregates; results sync back via a thin read API or materialised MySQL table the app queries. That keeps your web tier on PHP 8.5 and Laravel 13 while analytics run on the JVM.

For teams needing API development around event streams, expose aggregated results—not raw Kafka topics—to external clients. Apply the same rate limiting and abuse prevention patterns you would on any public REST surface.

On a marketplace-style platform like Gulfbizlist, real-time listing impressions might flow through Kafka while Flink computes trending scores displayed on the homepage within seconds.

Cost and sizing for Nepal and global teams

A minimal production cluster—three Kafka brokers, three Flink TaskManagers, managed storage—runs roughly Rs 45,000–90,000 per month on cloud VMs (~USD 335–670). That excludes engineer time for on-call. Smaller Nepal startups often share a multi-tenant cluster or start with managed Confluent Cloud / Aiven until event volume justifies dedicated hardware.

Linux system administration skills matter because both systems expose JVM tuning knobs, disk I/O limits, and network buffer settings. Misconfigured vm.max_map_count on Ubuntu breaks Elasticsearch sinks; misconfigured Kafka log.retention.hours drops data Flink still needs for replay.

What are common mistakes in stream processing with Kafka and Flink?

The failures are predictable. Teams underestimate state size, skip idempotent sinks, or deploy without staging topics that mirror production partition counts.

Unbounded state without TTL. A keyed process function that stores every user session forever will fill RocksDB until TaskManagers OOM. Add expiration or move cold data to object storage.

Ignoring backpressure. When sink throughput lags source throughput, Flink buffers until checkpoints balloon. Fix the sink first—add JDBC batching, widen MySQL write IOPS, or scale sink parallelism.

Using processing time for billing. Processing-time windows close when wall-clock ticks, not when events arrive. Out-of-order payment events get assigned to wrong windows. Always prefer event time with explicit watermarks for money-adjacent logic.

Non-transactional sinks with exactly-once enabled. Flink's exactly-once guarantee covers Kafka source and transactional Kafka sink. Writing to plain JDBC without upsert semantics still duplicates rows on failure. Use idempotent keys or two-phase commit capable sinks.

Validate JSON payloads during development with a JSON formatter and schema linter before they hit production topics. One malformed field should route to a dead-letter topic, not crash the job.

When async needs are modest, Symfony Messenger or database-backed queues remain simpler. Adopt Flink when replay, windows, or joins become first-class requirements—not because Kafka appeared on a architecture diagram.

Key Takeaways

• Kafka is the durable log; Flink is the stateful compute engine—design topics and partition counts before writing operators.
• Enable checkpointing and choose delivery semantics deliberately: exactly-once for financial data, at-least-once with idempotent sinks for many analytics cases.
• Use event time and watermarks for windowed aggregations; processing time is only acceptable for best-effort metrics.
• Manage schemas with Avro and a registry—JSON breaks silently at scale.
• Monitor checkpoint health and consumer lag; alert on consecutive checkpoint failures.
• Upgrade and rescale via savepoints; set maxParallelism on day one to avoid incompatible state later.

People Also Ask

Can Flink work without Kafka?

Yes. Flink reads from Pulsar, Kinesis, RabbitMQ, files, and sockets. Kafka is the most common pairing because of retention, replay, and ecosystem tooling. Teams already on Kafka should use the native Flink Kafka source rather than bridging through an intermediate service.

Is Apache Flink hard to learn for PHP or Laravel developers?

The stream processing concepts—event time, state, checkpoints—are new. Java or Python APIs add syntax overhead. Laravel developers comfortable with queues grasp the producer-consumer split quickly. Windowing and keyed state take longer. Start with a stateless filter job, then add tumbling windows before attempting joins.

How is Flink different from Kafka Streams?

Kafka Streams is a Java library embedded in your application—no separate cluster. Flink is a dedicated runtime with its own scheduler and Web UI. Kafka Streams suits simpler transformations colocated with microservices. Flink suits heavy state, complex CEP, and mixed sources. You can run both, but operational ownership differs.

What hardware do Kafka and Flink need in production?

Kafka brokers want fast SSD, 32 GB+ RAM, and 10 Gbps network for high throughput. Flink TaskManagers need RAM for network buffers plus local SSD for RocksDB. A three-node starter cluster often uses 8 vCPU and 32 GB per node. Scale horizontally by adding brokers and TaskManagers before vertically maxing single machines.

Ship real-time pipelines with confidence

Stream processing with Kafka and Flink gives you replayable events, sub-second analytics, and fault-tolerant state at scale. Start small: one topic, one stateless job, checkpoints to S3, and lag dashboards. Add windows and joins only when product requirements demand them. If you need help designing event pipelines, API surfaces for stream output, or hybrid Laravel-plus-Kafka architectures, contact us or explore custom software development and support and maintenance options. Related reading: active-active vs active-passive multi-cloud, about the author, and the blog archive for more streaming and backend guides.