Distributed Systems

Designing Event-Driven Systems with Kafka

A practical checklist for thinking about event contracts, ordering, retries, idempotency, and observability.

Event-driven architecture is attractive because it separates producers from consumers and makes asynchronous work natural. But the event stream is also a contract, and contracts need careful design.

Start with the event, not the topic

An event should describe something that happened, rather than expose an internal implementation detail. Consumers should be able to understand it without knowing the producer’s private state.

Ordering is local

Ordering guarantees depend on the partitioning model. If a business invariant requires ordered processing, the partition key needs to reflect that invariant.

Retries require idempotency

A retry means the consumer may see the same event again. Consumers therefore need a deliberate strategy for duplicate delivery rather than assuming exactly-once behavior everywhere.

Observability is part of the design

Correlation identifiers, consumer lag, processing latency, retry counts, and dead-letter handling should be designed alongside the message flow. Otherwise the system can be asynchronous and distributed—and impossible to explain when something goes wrong.