Java's new AOT cache feature is in a rough state

Introduction

The AOT (ahead-of-time) cache is the JDK's newest approach to improving startup and warmup times of Java applications. The idea is simple: you perform a training run of your application, during which the JVM records what your application does, e.g. which classes it loads and links and which methods it calls.

When your application is later started, the JVM can skip a lot of the initialization work as it is already done and immediately has all the classes available in a loaded and linked state.

In theory, this is a free startup performance boost without having to change anything in your application. In practice, using it for a real-world application that is distributed to users, like we do at XPipe, comes with a lot of pitfalls as we had to learn.

In the best case, the AOT cache is just not applied due to incompatibilities and you gain nothing. In the worst case, if you are not careful, your JVM crashes unexpectedly. So essentially a free startup boost, but only under certain conditions, and with free crashes included.

Hopefully this post will prevent you from running into the same issues as we did while rolling out this feature. At the end, you can find the build scripts we use for our applications based on all the things we learned over time.

Note that this post does not contain any performance measurements or comparisons, as that is a whole another story. This post is already too long for what it covers.

The concepts

If you are familiar with the JDK AOT concepts, you can skip this section. If not, you can get a short overview over all the relevant parts that go into the topic of AOT optimizations.

Timeline

The AOT cache itself has been extended with every JDK release since its introduction, with each step specified in a JEP (JDK Enhancement Proposal):

• JDK 24: JEP 483: Ahead-of-Time Class Loading & Linking introduced the AOT cache itself
• JDK 25: JEP 514: Ahead-of-Time Command-Line Ergonomics simplified the cache creation to a single step with -XX:AOTCacheOutput, and JEP 515: Ahead-of-Time Method Profiling added method profiles to the cache so that the JIT compiler can warm up faster
• JDK 26: JEP 516: Ahead-of-Time Object Caching with Any GC made cached objects usable with any garbage collector
• JDK 28: JEP 544: Ahead-of-Time Code Compilation will also add compiled code to the cache

Class-data sharing (CDS)

The AOT cache is not built from scratch. It is a natural evolution of class-data sharing (CDS), a HotSpot feature that has existed since JDK 5. CDS stores parsed class metadata in an archive file that the JVM can directly map into memory on startup.

It was extended over time with application class-data sharing in JDK 10, a default CDS archive for JDK classes that is shipped with every JDK since JDK 12, and dynamic CDS archives in JDK 13.

The AOT cache uses the same infrastructure, which is why you will still see CDS terms like "shared archive" or the cds log tag in many places when working with the AOT cache.

AOT class linking

AOT cache and AOT class linking are often used interchangeably, but they are not the same thing. The AOT cache is the file, which can contain different kinds of data: the loaded class data (the part inherited from CDS), method profiles, cached Java objects, some native code, and pre-linked classes.

AOT class linking is just one of these optimizations. It is turned on automatically when creating an AOT cache (-XX:+AOTClassLinking) and stores classes in an already linked state, so that the JVM can skip that work on startup. You want to enable class linking, otherwise you lose out on startup performance gains.

GraalVM Native Image

The AOT cache should also not be confused with GraalVM Native Image. Native Image compiles your whole application ahead of time into a standalone native executable that runs without a JVM. To do that, it has to make a closed-world assumption: all code that can be executed at runtime has to be known at build time, so dynamic features like reflection, dynamic class loading, or resources have to be explicitly configured. In return, you get a very fast startup and low memory usage.

The AOT cache takes a different approach. Your application still runs on a normal JVM with the JIT compiler and has the full dynamic capabilities of Java, so you don't have to change or configure anything in your application. The cache only speeds up the work the JVM has to do anyway. If it can't be used, the JVM just falls back to doing all of that at runtime as usual.

The startup improvements are therefore smaller than with Native Image, but you also don't have to deal with any of its restrictions. If you ever had to troubleshoot GraalVM native image issues, you are probably familiar with those.

CPU incompatibilities

In theory, the whole AOT cache idea sounds like something that should only affect the class data and not perform any native compilation. Because if it did, you would run into all sorts of issues if the individual CPU features of the training system and actual system would differ. Even if the CPU architecture is the same.

Anyone who used GraalVM native image and ran into crashes on systems with missing CPU features is probably familiar with this and the GraalVM build argument -march=compatibility.

In practice, it turns out the AOT cache with class linking does generate small parts of native code. You will run into the consequences of this if your training system supports something like AVX, and the running system does not (or not to the same level, like no support for AVX-512, which corresponds to the JDK's -XX:UseAVX=3 level).

Because then, your JVM will crash with:

A fatal error has been detected by the Java Runtime Environment: SIGILL (0x4) at pc=0x00007fcc85ae2613, pid=4054, tid=4147 JRE version: OpenJDK Runtime Environment Zulu25.32+21-CA (25.0.2+10) (build 25.0.2+10-LTS) Java VM: OpenJDK 64-Bit Server VM Zulu25.32+21-CA (25.0.2+10-LTS, mixed mode, sharing, tiered, compressed oops, compact obj headers, g1 gc, linux-amd64) Problematic frame: v ~AdapterBlob 0x00007fcc85ae2613

or

# # A fatal error has been detected by the Java Runtime Environment: # # EXCEPTION_ILLEGAL_INSTRUCTION (0xc000001d) at pc=0x000001d00d9f6c2b, pid=39700, tid=37844 # # JRE version: OpenJDK Runtime Environment Zulu25.30+17-CA (25.0.1+8) (build 25.0.1+8-LTS) # Java VM: OpenJDK 64-Bit Server VM Zulu25.30+17-CA (25.0.1+8-LTS, mixed mode, sharing, tiered, compressed oops, compact obj headers, g1 gc, windows-amd64) # Problematic frame: # v ~AdapterBlob 0x000001d00d9f6c2b

This means that you for example can't run an application with an AOT cache which was trained on an 13th gen Intel Core CPU on a target system that has a 12th gen Intel Core CPU as those have different levels of CPU feature support for AVX. If you only have newer gen CPUs and ship your app to a customer who still runs older gen hardware, you will get an unhappy customer who thinks that you don't even do basic testing for their software.

Adapter caches

As it turns out, it generates adapter caches, which were added in JDK 25 with JDK-8350209: Preserve adapters in AOT cache. Adapters are small pieces of machine code that switch between interpreted and compiled Java calls (see the HotSpot source for a more detailed explanation). They're generated for the CPU features of the machine that builds the cache without taking into account that the application might be run on another system later on. As soon as a system that lacks one of the CPU features used in the cached adapters calls one of them, the JVM crashes.

This is not only a problem for desktop applications that are distributed to users. Even on backend servers, the AOT cache is often created in a CI pipeline or a container image build, and then deployed to machines whose CPUs differ from the build machine, e.g. when a cluster contains different instance types or the cloud provider moves your workload to other hardware.

How on earth is that the default behaviour for the AOT cache?

Disabling adapter caching

It turns out you can disable the AOT adapter caches by passing the arguments -XX:+UnlockDiagnosticVMOptions -XX:-AOTAdapterCaching to your AOT cache generation run. This option is not explained or documented anywhere, so you have to read the JDK source for that one.

The even more confusing part was that this option is declared as false by default. It gets turned on automatically, however, by the AOT cache generation implementation if the cache contains AOT-linked classes, see here for JDK 25. You can see this with -XX:+PrintFlagsFinal, which shows the value as true {ergonomic} during the assembly run. So it is disabled by default, except in the default case.

Passing these arguments is a basically a requirement if you use the AOT cache. Otherwise, your users will run into crashes. This issue was eventually reported as JDK-8388703. JDK 27 will instead stores the CPU features of the training machine for the native code part of the AOT cache, which currently contains the adapters, and only uses the cached adapters on an exact CPU feature match.

What the AOT code cache actually is

To make things even more confusing, the term "AOT code cache" means two different things. They say there are only two hard things in computer science, cache invalidation and naming things, and this one manages to involve both.

In the JDK source code and the logs, the AOT code cache (AOTCodeCache) is the part of the AOT cache that contains native code. This has existed since JDK 25 and currently contains the adapters described above. So when you see log messages like AOT Code Cache is not used: AOT Class Linking is not used or AOT Code Cache disabled: cpu features are incompatible, or read bug reports about the AOT code cache, they are mostly about the adapters, not about compiled Java code. It is also not the same as the regular code cache in which the JIT compiler stores compiled code at runtime.

The CPU feature checks in JDK 27 mentioned above are also part of this. The actual compilation of Java code ahead of time, which is what most people would think of when hearing about an AOT code cache, will only come with JDK 28, and will store the compiled code in this same AOT code cache.

AOT code compilation in JDK 28

JDK 28 will release an additional feature, JEP 544: Ahead-of-Time Code Compilation, which, as the name suggests, will also do a native code compilation for your cache, with more code than just the adapters. According to the JEP, the AOT code will not be loaded if the CPU features have a mismatch, which at least will avoid crashes. This uses the same CPU feature check that already exists in JDK 27, which requires an exact match: on x86, all CPU features except hyper-threading have to be identical. So even a newer CPU that supports more features than the training machine will not load the AOT code.

However, that will also make this feature quite useless for applications that are distributed to users, as the chance of the CPU of a user exactly matching the CPU of your build machine is quite low. What would be needed is some sort of compatibility switch that disables all optional CPU features. Sure, you will get some reduced performance of the compiled code, but in exchange, a faster startup time.

Technically, the compared CPU features are the ones that are enabled for the JVM, which can be reduced via options like -XX:UseAVX (see here). But you would have to pass the same options to the training run and the actual application run, and not every CPU feature has such an option, so that is not a real solution either. There needs to be a proper compatibility mode switch for this.

Scraping logs

When dealing with the AOT cache, the JVM argument -Xlog:aot is your best friend. Granted, the kind of friend who answers every question with a riddle. It will print the details when generating an AOT cache and will also show whether the AOT cache is loaded properly when you run your built application. However, there are several pitfalls as the output is quite tricky to understand and easy to misinterpret.

Two runs walk into a log

JEP 514: Ahead-of-Time Command-Line Ergonomics introduced an easier workflow for building the AOT cache. Instead of having to perform two steps, you now only need one invocation of your application to build the AOT cache. However, technically it still performs two separate JVM runs, it just hides that fact from you. As a result, you will get logs from two invocations as one, while only the second one, the assembly run that actually writes the AOT cache, tells you what ends up in the cache. Both look very similar.

If you are unaware of that, you will spend a lot of time debugging the wrong logs until you figure out that there are two separate logs printed one after another, and only the latter is relevant for the cache contents. So be careful when using CTRL+F and check whether you are looking at the correct version of the log.

Confusing log messages

Furthermore, the log messages are missing basically all the relevant information, and there are almost no docs to back them up. For example, the run that assembles the AOT cache will log something like this:

[0.024s][info][aot] full module graph: disabled [0.024s][info][aot] initial optimized module handling: enabled [0.024s][info][aot] initial full module graph: disabled [0.024s][info][aot] Using AOT-linked classes: false (static archive: no aot-linked classes) [0.032s][info][aot] JVM_StartThread() ignored: java.lang.ref.Reference$ReferenceHandler [0.032s][info][aot] JVM_StartThread() ignored: java.lang.ref.Finalizer$FinalizerThread [0.050s][info][aot] Not starting management agent during creation of AOT cache.

Now, your first questions are probably: What is a full module graph, why is it disabled, how can I enable it, does that influence the performance of the AOT cache? For all the answers, you will have to consult the JDK source code as that is barely explained. Why a certain feature is disabled is also not explained, so you will have t