Designing Evolvable Kafka Event Architectures: Schema Evolution Without Breaking Consumers

Project Context

Project Context: This is Issue #13 of my ongoing implementation series for a Kafka-based distributed telemetry ingestion platform. The complete implementation and future iterations can be tracked here: Repository.

This issue focuses on schema evolution, version-aware event processing, and designing stable event contracts for changing telemetry requirements.

While building this Kafka-based distributed telemetry ingestion platform, I reached a point where the problem was no longer about moving data from one place to another. The pipeline already had the fundamental flow:

Sensor Producers → Kafka Topics → Consumers → Processing Layer → Persistence Layer → Database

The Challenge of Evolving Events

The challenge was: How do we evolve telemetry events when the data model itself changes over time?

Imagine a sensor device today sends:

• Temperature
• Humidity
• Timestamp

Tomorrow, a newer generation of the device sends:

• Temperature
• Humidity
• Timestamp
• Battery Level

Current file structure:

Figure 1: Current telemetry package structure showing the organization of domain models, versioned event contracts, processing services, Kafka consumers, and persistence components.

The easiest approach is to keep modifying one event class and keep adding fields. But that creates a growing contract:

telemetry/domain/TelemetryEvent.java: Initial telemetry event model before introducing version-specific representations.

The model started accumulating fields that belonged only to certain device versions. Over time, the original event becomes tightly coupled to every future requirement. Older versions start carrying fields they never understood, and every new change risks affecting existing consumers.

The question became: How do we allow event versions to evolve independently while keeping the processing pipeline stable?

Figure 2: The initial single-event approach where new requirements continuously expand the telemetry model, increasing coupling between event versions and consumers.

Separating Telemetry Versions

The first design decision was to separate telemetry versions. Instead of one expanding model, I introduced:

TelemetryEventV1 -> Temperature + Humidity

telemetry/domain/event/TelemetryEventV1.java: TelemetryEventV1 implementation representing the first telemetry schema version with only the common sensor fields defined by the VersionedTelemetryEvent contract.

TelemetryEventV2 -> Temperature + Humidity + Battery Level

telemetry/domain/event/TelemetryEventV2.java: TelemetryEventV2 implementation extending the telemetry schema with battery support through BatteryAwareTelemetryEvent while maintaining the common VersionedTelemetryEvent contract.

Now the versions could evolve independently. But this introduced another question: Who decides which version of the event should be created?

The answer was a dedicated version mapping layer. The flow became:

Figure 3: Version-specific telemetry models allowing each event version to evolve independently while keeping version selection centralized through TelemetryEventVersionMapper.

The mapper correctly identified the version and created the corresponding event object.

telemetry/service/TelemetryEventVersionMapper.java: TelemetryEventVersionMapper centralizing version selection and transformation logic to convert incoming telemetry events into their corresponding version-specific implementations.

The Java Type System Problem

Then I encountered a Java type system problem. Initially, the mapper returned:

Object

This was because both versions were objects. The mapping worked. But the consumer received only:

Object event;

At that point, Java only knew: "This is an Object." It did not know whether it was:

• TelemetryEventV1
• TelemetryEventV2

The version information was lost at the contract level and the consumer could not safely access version-specific behavior.

Introducing a Common Contract

This led to introducing a common contract:

VersionedTelemetryEvent

Now both event versions implemented the same interface. The flow evolved into:

Figure 4: Common event contract introduced through VersionedTelemetryEvent, allowing multiple telemetry versions to share a stable abstraction while preserving their version-specific fields.

The consumer no longer cared about the exact implementation. It only depended on the contract. This was the first application of the Dependency Inversion Principle: High-level processing logic depended on an abstraction rather than concrete event classes.

telemetry/domain/event/VersionedTelemetryEvent.java: VersionedTelemetryEvent defining the stable contract shared across all telemetry event versions while keeping version-specific capabilities outside the core interface.

But another design question appeared. If the processor receives:

VersionedTelemetryEvent event;

what happens when V2 has:

getBatteryLevel()

but V1 does not? Should battery become part of the common interface? Something like:

interface VersionedTelemetryEvent {
  getTemperature();
  getHumidity();
  getBatteryLevel();
}

That would solve V2, but it would force V1 to support functionality it does not have. The contract would become polluted.

Capability-Based Design

The solution was capability-based design. Instead of making every telemetry event support every possible feature:

• VersionedTelemetryEvent
• BatteryAwareTelemetryEvent
• LocationAwareTelemetryEvent
• FirmwareAwareTelemetryEvent

Only events that support a capability implement it. So:

TelemetryEventV1 -> VersionedTelemetryEvent

TelemetryEventV2 -> VersionedTelemetryEvent + BatteryAwareTelemetryEvent

telemetry/domain/event/capability/BatteryAwareTelemetryEvent.java: BatteryAwareTelemetryEvent capability interface isolating battery-specific behavior so only telemetry versions supporting battery data implement this additional contract.

telemetry/domain/event/capability/LocationAwareTelemetryEvent.java: LocationAwareTelemetryEvent capability interface designed to support future location-enabled telemetry versions without modifying the core event contract.

Now the processor can handle optional capabilities safely:

telemetry/service/TelemetryEventProcessor.java: TelemetryEventProcessor consuming the common VersionedTelemetryEvent contract, transforming events into persistence records, and handling optional capabilities through capability-based interfaces.

This followed the Interface Segregation Principle: Classes should not be forced to depend on methods they do not use.

Separation of Processing and Persistence

The next responsibility boundary was processing. The event object should not directly become a database entity. So the final flow became:

Figure 5: Final telemetry processing flow separating event version handling, business transformation, and persistence responsibilities across independent layers.

The processor became responsible for transformation. The persistence layer became responsible only for persistence. This followed the Single Responsibility Principle:

• Mapper handles version selection.
• Processor handles transformation.
• Persistence handles storage.

Each component has one clear reason to change.

Final Consumer Service - TelemetryConsumerService.java: TelemetryConsumerService orchestrating the version-aware Kafka consumption flow by mapping incoming events, processing versioned telemetry contracts, and delegating persistence responsibilities.

The final architecture allowed:

• New telemetry versions without modifying existing ones.
• Optional capabilities without bloating the base interface.
• Cleaner separation between ingestion, processing, and storage.

Lessons on Schema Evolution

The important realization was: Schema evolution is not solved by adding a version number. The real challenge is designing the boundaries correctly:

• What belongs to every event?
• What belongs to a specific version?
• What should become a capability?
• Which layer should know about version details?

One more architectural gap became clear during this process. The consumer side is now version-aware:

Kafka Event -> Version Mapper -> Versioned Processing

But the producer side is still using the generic event contract. The complete evolution will eventually become:

Producer -> Versioned Event DTO -> Kafka -> Versioned Consumer -> Processing -> Persistence

That introduces the next design area:

• Polymorphic Kafka serialization
• Producer-side versioned DTOs
• Event envelope design
• Schema compatibility strategies

The most valuable part of this implementation was not creating multiple event classes. It was learning that good architecture often comes from repeatedly asking: "Does this responsibility really belong here?" Every uncomfortable question exposed a better boundary.

I would love to hear your thoughts and perspectives on designing evolvable event-driven systems. How do you approach schema evolution in Kafka-based architectures? What patterns or trade-offs have worked well in your production systems? Open to discussions, feedback, and different architectural viewpoints. Thank you.