The compiler was never what you wanted

You have an orders topic on a Kafka cluster, its values encoded with Avro against a schema in the Schema Registry. You want the orders worth more than fifty euros on a topic of their own, and you have decided to do it with Kafka Streams - a JVM library, your code, your deployment.

The schema has five fields:

{
  "type": "record",
  "name": "Order",
  "namespace": "com.alginte.demo",
  "fields": [
    {"name": "orderId", "type": "string"},
    {"name": "customerId", "type": "string"},
    {"name": "item", "type": "string"},
    {"name": "quantity", "type": "int"},
    {"name": "priceEur", "type": "double"}
  ]
}

You want one line of logic over them: quantity * priceEur > 50. Here is everything standing between that line and a topic of big orders.

Seven Steps

The route Confluent's own examples take, and many projects with them:

1. Get the schema out of the registry and into your repository as an .avsc - or, if your team owns the schema in the repository and publishes it to the registry, the other way round. Whichever copy you call the source, there are now two that can disagree.
2. Add the code generator to your build.
3. Configure it - source and output directories, and the string type.
4. Build, producing Order.java under target/generated-sources.
5. Write the topology against the generated class.
6. Package the application, with the schema, the class and the serde.
7. Deploy it somewhere that runs a JVM.

Steps 2 and 3 are this, once - in Maven, though Gradle's equivalent has the same shape:

<plugin>
  <groupId>org.apache.avro</groupId>
  <artifactId>avro-maven-plugin</artifactId>
  <version>1.12.1</version>
  <executions>
    <execution>
      <phase>generate-sources</phase>
      <goals><goal>schema</goal></goals>
      <configuration>
        <sourceDirectory>${project.basedir}/src/main/avro</sourceDirectory>
        <!-- without this, string fields generate as CharSequence, not String; Confluent's own examples set it for the same reason -->
        <stringType>String</stringType>
      </configuration>
    </execution>
  </executions>
</plugin>

And step 5 is the part you actually wanted to write:

builder.stream("orders", Consumed.with(Serdes.Void(), orderSerde))
       .filter((key, order) -> order.getQuantity() * order.getPriceEur() > 50)
       .to("big-orders", Produced.with(Serdes.Void(), orderSerde));

Seven steps, one of them the predicate. Two of them - adding the plugin and configuring it - you do that once. The filter is deliberately trivial - a real topology joins, aggregates and branches - but the seven steps are identical for twenty operators, because they are charged per project rather than per line of logic.

And application.id, bootstrap.servers, the registry URL and your cluster's authentication are missing from the list because they are the price of running against a real cluster and registry, not the price of generating classes. Those two are not the cost.

Schema Evolution

Then you want to use a new field. Somebody adds region to the schema. While it sits there unused you are fine - Avro resolves the writer's schema against yours, your generated class does not know the field exists, and nothing needs rebuilding. Schema evolution is doing its job.

Then somebody asks for EU orders only. The change to your logic is one term: && order.getRegion().equals("EU"). The change to your project is steps 1, 4, 5, 6 and 7 - pull the new schema, regenerate, rewrite, repackage, redeploy.

That is the actual price, and it is charged not per schema change but per schema change you need. Which, over the life of a pipeline, is many of them: fields get added because somebody intends to use them.

What the Seven Steps Buy

They buy the compiler. order.getQuantiy() does not compile. Rename a field, regenerate, and every stale use site turns red before anything runs. The IDE completes field names. Refactoring works. That is worth having.

It is also answering a narrower question than the one you actually have. The compiler can tell you that getQuantiy() is not a method. It cannot tell you whether quantity * priceEur > 50 is the predicate you meant, whether it matches any record on the topic, or whether the field you are multiplying holds what you think it holds. For that, the seven steps have one answer: deploy it and look.

So the loop you are really in is not edit, compile. It is edit, compile, package, deploy, produce a record, read the output - and it costs the same whether the expression was right or wrong.

A Tighter Loop

There is another shape for this, and it is the one we build: Alginte, a browser-based topology builder that assembles it at runtime from topics you pick, instead of compiling it into an application you ship.

Point it at orders. The schema comes from the registry at runtime, the five field names arrive as completions, and the predicate is a string - written in SpEL, Spring's expression language, evaluated once per record:

value.get('quantity') * value.get('priceEur') > 50

That expression is evaluated against a real record from the topic while it is being typed - a record off the partition rather than a mock or a fixture, with the answer beside it. When it is right, it deploys as a Kafka Streams topology: the same library, the same KafkaStreams client, the same rebalances, state stores and changelog topics you would have got from the seven steps. No separate engine is involved. The only thing that changed is how the topology was written.

No .avsc in a repository, no plugin, no target/generated-sources. The record on the in line is a real one off orders - quantity 3, priceEur 17. The out line is whatever the expression currently returns, and it follows the expression as it is edited: 51, which is the number the > 50 filter is about to judge. Nothing has been deployed.

Mistakes surface in the same place. A method that does not exist on the type the record actually carries is reported while it is being typed, naming the type - the post about building that editor has that one on camera.

That is not a compiler. It is the question the compiler could not answer, asked against real data, answered in seconds.

And when somebody adds a sixth field and you want to use it, you type its name.

What You Give Up

The compiler. value.get('quantiy') is a valid expression - javac never sees it, so nothing rejects it before it runs. Rename across the project goes too, and the unit tests that constructed Order objects.

What you do not lose is the typo itself. On Avro a misspelt field throws at access - Not a valid schema field: quantiy - so it fails in the editor and fails deployed, identically. That is not the compiler's guarantee, which is made before anything runs. It is the same failure arriving in both places at the same moment: a weaker promise, and a real one.

Some of it returns in a different form - field names completed from the registry, so the typo is never offered; the expression checked against a real record as you type; the same failure surfacing in the editor that would surface in the deployed topology. A different guarantee, weaker in some places, stronger in one.

What Does Not Go

The review. A topology drawn here exports as JSON and imports back, so the thing a reviewer reads and the thing git keeps is a file, not a browser session - the build step is gone, the artifact is not.

You can have the generic half of this without any of the rest. Nothing stops you writing a Java topology that reads GenericRecord and never generates a class: steps 1 to 4 disappear, and so does the compiler, since record.get("quantiy") is a string lookup that javac will not check either.

What you are left with is the worst of both - no type safety, and steps 6 and 7 still in front of you. The generic types are not the point. The loop around them is: the completions, the record on screen, and the deploy that is a click rather than a pipeline.

What That Buys

A faster answer to the question you actually had. And if you would rather write the Java regardless - the stream is one part of a larger application, or your team works that way, or any of the other good reasons - the loop is still worth having first.

Draw it here, get the expressions right against real records, then spend the seven steps on logic you already know works. That is a better use of an afternoon than finding out after the deploy.

This came out of building Alginte, a visual Kafka Streams builder - self-hosted, free to run.