Native Reference Guide

The garbage collectors available for Quarkus users are currently Serial GC and Epsilon GC.

For information about options to control maximum heap size, young space, and other typical use cases found in the JVM, see the GraalVM Memory Management guide. Setting the maximum heap size, either as a percentage or an explicit value, is generally recommended.

Multiple options exist to print information about garbage collection cycles, depending on the level of detail required. The minimum detail is provided -XX:+PrintGC, which prints a message for each GC cycle that occurs:

When you combine this option with -XX:+VerboseGC you still get a message per GC cycle, but it contains extra information. Also, adding this option shows the sizing decisions made by the GC algorithm at startup:

Beyond these two options, -XX:+PrintHeapShape and -XX:+TraceHeapChunks provide even lower level details about memory chunks on top of which the different memory regions are constructed.

The most up-to-date information on GC logging flags can be obtained by printing the list of flags that can be passed to native executables:

As described in the Measuring Performance guide, the footprint of Quarkus applications is measured using the resident set size (RSS). This is also applicable to native applications, but the runtime engine that manages the footprint in this case is built in the native executable itself rather than the JVM.

The reporting techniques specified in the Measuring Performance guide are applicable to native applications too, but what causes the RSS to be higher or lower is specific to how the generated native executables work.

When the RSS is higher in one native version of the application versus another, the following checks should be carried out first:

Often profiling, instrumenting or tracing applications is the best way to figure out how things work. In the case of RSS and native applications, the techniques that Brendan Gregg explains in the "Memory Leak (and Growth) Flame Graphs" guide are particularly useful. This section will apply the information in that article to show how to use perf and bcc/eBPF to understand what causes Quarkus native executables to consume memory on startup.

This debugging guide has the following requirements:

This guide builds and executes Quarkus native executables within a Linux environment. To offer a homogeneous experience across all environments, the guide relies on a container runtime environment to build and run the native executables. The instructions below use Docker as example, but very similar commands should work on alternative container runtimes, e.g. podman.

Finally, this guide assumes the use of the Mandrel distribution of GraalVM for building native executables, and these are built within a container so there is no need for installing Mandrel on the host.

Start by creating a new Quarkus project. Open a terminal and run the following command:

针对Linux和MacOS用户

For Windows users:

对于Windows用户

Some Quarkus configuration options will be used constantly throughout this debugging guide, so to help declutter command line invocations, it’s recommended to add these options to the application.properties file. So, go ahead and add the following options to that file:

As a first step, change to the project directory and build the native executable for the application:

Run the application to verify it works as expected. In one terminal:

In another:

The rest of this section explores ways to build the native executable with extra information, but first, stop the running application. We can obtain this information while building the native executable by adding additional native-image build options using -Dquarkus.native.additional-build-args, e.g.

Executing that will produce additional output lines like this:

The target info file contains information such as the target platform, the toolchain used to compile the executable, and the C library in use:

The native library info file contains information on the static libraries added to the binary and the other libraries dynamically linked to the executable:

Even more detail can be obtained by passing in --verbose as an additional native-image build argument. This option can be very useful in detecting whether the options that you pass at a high level via Quarkus are being passed down to the native executable production, or whether some third party jar has some native-image configuration embedded in it that is reaching the native-image invocation:

Running with --verbose demonstrates how the native-image building process is two sequential java processes:

One may also combine multiple native build options by separating with a comma, e.g.:

Given a native executable, various Linux tools can be used to inspect it. To allow supporting a variety of environments, inspections will be done from within a Linux container. Let’s create a Linux container image with all the tools required for this guide:

Using docker in the non-Linux environment, you can create an image using this Dockerfile via:

Then, go to the root of the project and run the Docker container we have just created as:

ldd shows the shared library dependencies of an executable:

strings can be used to look for text messages inside the binary:

Using strings you can also get Mandrel information given the binary:

Finally, using readelf we can inspect different sections of the binary. For example, we can see how the heap and text sections take most of the binary:

Optionally, the native build process can generate reports that show what goes into the binary:

The reports will be created under target/debugging-native-1.0.0-SNAPSHOT-native-image-source-jar/reports/. These reports are some of the most useful resources when encountering issues with missing methods/classes, or encountering forbidden methods by Mandrel.

Quarkus instructs Mandrel to initialize as much as possible at build time, so that runtime startup can be as fast as possible. This is important in containerized environments where the startup speed has a big impact on how quickly an application is ready to do work. Build time initialization also minimizes the risk of runtime failures due to unsupported features becoming reachable through runtime initialization, thus making Quarkus more reliable.

The most common examples of build-time initialized code are static variables and blocks. Although Mandrel executes those at run-time by default, Quarkus instructs Mandrel to run them at build-time for the reasons given.

This means that any static variables initialized inline, or initialized in a static block, will keep the same value even if the application is restarted. This is a different behaviour compared to what would happen if executed as Java.

To see this in action with a very basic example, add a new TimestampResource to the application that looks like this:

Rebuild the binary using:

Run the application in one terminal (make sure you stop any other native executable container runs before executing this):

Send a GET request multiple times from another terminal:

to see how the current time has been baked into the binary. This time was calculated when the binary was being built, hence application restarts have no effect.

In some situations, built time initializations can lead to errors when building native executables. One example is when a value gets computed at build time which is forbidden to reside in the heap of the JVM that gets baked into the binary. To see this in action, add this REST resource:

When trying to rebuild the application, you’ll encounter an error:

So, what the message above is telling us is that our application caches a value that is supposed to be random as a constant. This is not desirable because something that’s supposed to be random is no longer so, because the seed is baked in the image. The message above makes it quite clear what is causing this, but in other situations the cause might be more obfuscated. As a next step, we’ll add some extra flags to the native executable generation to get more information.

As suggested by the message, let’s start by adding an option to track object instantiation:

The error messages point to the code in the example, but it can be surprising that a reference to DnsClient appears. Why is that? The key is in what happens inside KeyPairGenerator.initialize() method call. It uses JCAUtil.getSecureRandom() which is why this is problematic, but sometimes the tracing options can show some stack traces that do not represent what happens in reality. The best option is to dig through the source code and use tracing output for guidance but not as full truth.

Moving the KEY_PAIR_GEN.initialize(1024); call to the run-time executed method encryptDecrypt is enough to solve this particular issue. Rebuild the application and verify that encrypt/decrypt endpoint works as expected by sending any message and check if the reply is the same as the incoming message:

Additional information on which classes are initialized and why can be obtained by passing in the -H:+PrintClassInitialization flag via -Dquarkus.native.additional-build-args.

One of the drawbacks of using native executables is that they cannot be debugged using the standard Java debuggers, instead we need to debug them using gdb, the GNU Project debugger. To demonstrate how to do this, we are going to generate a native Quarkus application that crashes due to a Segmentation Fault when accessing http://localhost:8080/crash. To achieve this, add the following REST resource to the project:

This code will try to copy 256 bytes from address 0x0 to 0x80 resulting in a Segmentation Fault. To verify this, compile and run the example application:

This will result in the following output:

The omitted output above contains clues to what caused the issue, but in this exercise we are going to assume that no information was provided. Let’s try to debug the segmentation fault using gdb. To do that, go to the root of the project and enter the tools container:

Then start the application in gdb and execute run.

Next, try to access http://localhost:8080/crash:

This will result in the following message in gdb:

If we try to get more info about the backtrace that led to this crash we will see that there is not enough information available.

This is because we didn’t compile the Quarkus application with -Dquarkus.native.debug.enabled, so gdb cannot find debugging symbols for our native executable, as indicated by the "No debugging symbols found in ./target/debugging-native-1.0.0-SNAPSHOT-runner" message in the beginning of gdb.

Recompiling the Quarkus application with -Dquarkus.native.debug.enabled and rerunning it through gdb we are now able to get a backtrace making clear what caused the crash. On top of that, add -H:-OmitInlinedMethodDebugLineInfo option to avoid inlined methods being omitted from the backtrace:

This will result in the following message in gdb:

We already see that gdb is able to tell us which method caused the crash and where it’s located in the source code. We can also get a backtrace of the call graph that led us to this state:

Similarly, we can get a backtrace of the call graph of other threads.

Alternatively, we can list the backtraces of all threads with a single command:

Note, however, that despite being able to get a backtrace we can still not list the source code at point with the list command.

This is because gdb is not aware of the location of the source files. We are running the executable outside the target directory. To fix this we can either rerun gdb from the target directory or, run directory target/debugging-native-1.0.0-SNAPSHOT-native-image-source-jar/sources e.g.:

We can now examine line 169 and get a first hint of what might be wrong (in this case we see that it fails at the first read from src which contains the address 0x0000), or walk up the stack using gdb’s up command to see what part of our code led to this situation. For more information about using gdb to debug native executables, see the GraalVM Debug Info Feature guide.

Native executable generation is a multi-step process. The analysis and compile steps are the most expensive of all and hence the ones that dominate the time spent generating the native executable.

In the analysis phase, a static points-to analysis starts from the main method of the program to find out what is reachable. As new classes are discovered, some of them will be initialized during this process depending on the configuration. In the next step, the heap is snapshotted and checks are made to see which types need to be available at runtime. The initialization and heap snapshotting can cause new types to be discovered, in which case the process is repeated. The process stops when a fixed point is reached, that is when the reachable program grows no more.

The compilation step is pretty straightforward, it simply compiles all the reachable code.

The time spent in analysis and compilation phases depends on how big the application is. The bigger the application, the longer it takes to compile it. However, there are certain features that can have an exponential effect. For example, when registering types and methods for reflection access, the analysis can’t easily see what’s behind those types or methods, so it has to do more work to complete the analysis step.

Starting with Mandrel 23.1 and GraalVM for JDK 21, the native executable generation process will warn about the use of experimental options with a message like this:

If the mentioned option is added by a third party library like in the example above, you should consider opening an issue in the library’s repository to ask for the option to be removed. If the option is added by your application, you should consider either removing it (if it’s not necessary) or wrapping it between -H:+UnlockExperimentalVMOptions and -H:-UnlockExperimentalVMOptions.

When building a native executable Quarkus requires all classes being referenced by the code, no matter if they are build-time or run-time initialized, to be present in the classpath. This way it ensures that there will be no crashes at runtime due to potential NoClassDefFoundError exceptions. To achieve this it makes use of GraalVM’s --link-at-build-time parameter:

This, however, may result in an AnalysisError\$ParsingError due to an UnresolvedElementException at build time. This is often caused because the application references a class from an optional dependency.

If you have access to the source code responsible for the reference to the missing dependency and can alter it, you should consider one of the following:

In the unfortunate case where the reference causing the issue is made by a 3rd party library, that you cannot modify, you should consider one of the following:

Note that although option (1) is the best choice performance wise, as it minimizes the applications footprint,it might not be trivial to implement. To make matters worse, it’s also not easy to maintain as it is tightly coupled to the 3rd party library implementation. Option (2) is a straight forward alternative to work around the issue, but comes at the cost of including possibly never invoked code in the resulting native executable.

Building native executables is not only time consuming, but it also takes a fair amount of memory. For example, building a sample native Quarkus Jakarta Persistence application such as the Hibernate ORM quickstart, may use 6GB to 8GB resident set size in memory. A big chunk of this memory is Java heap, but extra memory is required for other aspects of the JVM that runs the native building process. It is still possible to build such applications in environments that have total memory close to the limits, but to do that it is necessary to shrink the maximum heap size of the GraalVM native image process. To do that, set a maximum heap size using the quarkus.native.native-image-xmx property. For example, we can instruct GraalVM to use 5GB of maximum heap size by passing in -Dquarkus.native.native-image-xmx=5g in the command line.

Building native executables this way might have the side effect of requiring more time to complete. This is due to garbage collection having to work harder for native image generation to have free space to do its job.

Note that typical applications are likely bigger than quickstarts, so the memory requirements will also likely be higher.

As with most things in life there are some trade-offs involved when choosing native compilation over JVM mode. So depending on the application the runtime performance of a native application might be slower compared to JVM mode, though that’s not always the case.

JVM execution of an application includes runtime optimization of the code that profits from profile information built up during execution. That includes the opportunities to inline a lot more of the code, locate hot code on direct paths (i.e. ensure better instruction cache locality) and cut out a lot of the code on cold paths (on the JVM a lot of code does not get compiled until something tries to execute it — it is replaced with a trap that causes deoptimization and recompilation). Removal of cold paths provides many more optimization opportunities than are available for ahead of time compilation because it significantly reduces the branch complexity and combinatorial logic of the smaller amount of hot code that is compiled.

By contrast, native executable compilation has to cater for all possible execution paths when it compiles code offline since it does not know which are the hot or cold paths and cannot use the trick of planting a trap and recompiling if it is hit. For the same reason it cannot load the dice to ensure that code cache conflicts are minimized by co-locating hot paths adjacent. Native executable generation is able to remove some code because of the closed world hypothesis but that is often not enough to make up for all the benefits that profiling and runtime deopt & recompile provides to the JVM JIT compiler.

Note, however, that there is a price you pay for that potentially higher JVM speed, and that price is in increased resource usage (both CPU and memory) and startup time because:

The reason for 1) is that code needs to be run interpreted for some time and, possibly, to be compiled several times before all potential optimizations are realized to ensure that:

An implication of 1) is that for small, short-lived applications a native executable may well be a better bet. Although the compiled code is not as well optimized it is available straight away.

The reason for 2) is that the JVM is essentially running the compiler at runtime in parallel with the application itself. In the case of native executables the compiler is run ahead of time removing the need to run the compiler in parallel with the application.

There are several reasons for 3). The JVM does not have a closed world assumption. So, it has to be able to recompile code if loading of new classes implies that it needs to revise optimistic assumptions made at compile time. For example, if an interface has only one implementation it can make a call jump directly to that code. However, in the case where a second implementation class is loaded the call site needs to be patched to test the type of the receiver instance and jump to the code that belongs to its class. Supporting optimizations like this one requires keeping track of a lot more details of the class base than a native executable, including recording the full class and interface hierarchy, details of which methods override other methods, all method bytecode etc. In a native executable most of the details of class structure and bytecode can be ignored at run time.

The JVM also has to cope with changes to the class base or execution profiles that result in a thread going down a previously cold path. At that point the JVM has to jump out of the compiled code into the interpreter and recompile the code to cater for a new execution profile that includes the previously cold path. That requires keeping runtime info that allow a compiled stack frame to be replaced with one or more interpreter frames. It also requires runtime extensible profile counters to be allocated and updated to track what has or has not been executed.

This can be attributed to a number of different reasons:

There is a GraalVM upstream issue with some interesting discussions about that topic.

One can see which Mandrel version was used to generate a binary by inspecting the binary as follows:

See Native Memory Management GC Logging section for details.

Starting with GraalVM 22.2.0 it is possible to create heap dumps upon request, e.g. kill -SIGUSR1 <pid>. Support for dumping the heap dump upon an out of memory error will follow up.

Yes you can. In fact, debugging native executables on a Linux bare metal box offers the best possible experience. In this kind of environments, root access is not needed except to install packages required to run some debug steps, or to enable perf to gather events at the kernel.

These are the packages you’ll need on your Linux environment to run through the different debugging sections:

There are multiple ways in which a native executable produced by Mandrel can be profiled. All the methods require you to pass in the -H:-DeleteLocalSymbols option.

The method shown in this reference guide generates a binary with DWARF debug information, runs it via perf record and then uses perf script and flame graph tooling to generate the flamegraphs. However, the perf script post-processing step done on this binary can appear to be slow or can show some DWARF errors.

An alternative method to generate flame graphs is to pass in -H:+PreserveFramePointer when generating the native executable instead of generating the DWARF debug information. It instructs the binary to use an extra register for the frame pointer. This enables perf to do stack walking to profile the runtime behaviour. To generate the native executable using these flags, do the following:

To get runtime profiling information out of the native executable, simply do:

The recommended method for generating runtime profiling information is using the debug information rather than generating a binary that preserves the frame pointer. This is because adding debug information to the native executable build process has no negative runtime performance whereas preserving the frame pointer does.

DWARF debug info is generated in a separate file and can even be omitted in the default deployment and only be transferred and used on demand, for profiling or debugging purposes. Furthermore, the presence of debug info enables perf to show us the relevant source code lines as well, hence it does not bloat the native executable itself. To do that, simply call perf report with an extra parameter to show source code lines:

The performance penalty of preserving the frame pointer is due to using the extra register for stack walking, particularly in x86_64 compared to aarch64 where there are fewer registers available. Using this extra register reduces the number of registers that are available for other work, which can lead to performance penalties.

Although it is possible to remote debug processes within containers, it might be easier to step-by-step debug native-image by installing Mandrel locally and adding it to the path of the shell process.

Native executable generation is the result of two Java processes that are executed sequentially. The first process is very short and its main job is to set things up for the second process. The second process is the one that takes care of most of the work. The steps to debug one process or the other vary slightly.

Let’s discuss first how to debug the second process, which is the one you most likely to want to debug. The starting point for the second process is the com.oracle.svm.hosted.NativeImageGeneratorRunner class. To debug this process, simply add --debug-attach=*:8000 as an additional build time argument:

The starting point for the first process is the com.oracle.svm.driver.NativeImages class. In GraalVM CE distributions, this first process is a binary, so debugging it in the traditional way with a Java IDE is not possible. However, Mandrel distributions (or locally built GraalVM CE instances) keep this as a normal Java process, so you can remote debug this process by adding the --vm.agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:8000 as an additional build argument, e.g.

Java Flight Recorder (JFR) and JDK Mission Control (JMC) can be used to profile native binaries since GraalVM CE 21.2.0. However, JFR in GraalVM is currently limited in capabilities compared to HotSpot. The custom event API is fully supported, but some VM level features are unavailable. More events and JFR features will continue to be added in later releases. The following table outlines Native Image JFR support and limitations by version.

To add JFR support to your Quarkus executable, add the application property: -Dquarkus.native.monitoring=jfr. E.g.

Once the image is compiled, enable and start JFR via runtime flags: -XX:+FlightRecorder and -XX:StartFlightRecording. For example:

For more information about using JFR, see the GraalVM JDK Flight Recorder (JFR) with Native Image guide.

In this situation, switching to JVM mode would be the best thing to try first. If the performance issues continue after switching to JVM mode, you can use more established and mature tooling to figure out the root cause. If the performance issue is limited to native mode only, you might not be able to use perf, so JFR is the only way to gather any information in this situation. As JFR support for native expands, the ability to detect root causes of performance issues directly in production will improve.

To fix classpath, class initialization or forbidden API errors at build time it’s best to use build time reports to understand the closed world universe. A complete picture of the universe, along with the relationships between the different classes and methods will help uncover and fix most of the issues.

To fix runtime native specific errors, it’s best to have debug info builds of the native executables around, so that gdb can be hooked up quickly to debug the issue. If you also add local symbols to the debug info builds, you will obtain precise profiling information as well.

It might so happen that the build gets stalled and even ends up with:

One of the possible explanations could be a lack of entropy, e.g. on an entropy constrained VM, if such a source is needed as it is the case with Bouncycastle at build time.

One can check the available entropy on a Linux system with:

If the amount is not in hundreds, it could be a problem. A possible workaround is to compromise, acceptable for testing, and set:

The proper solution is to increase the entropy available for the system. That is specific for each OS vendor and virtualization solution though.

When building on recent machines and running your native executable on older machines, you may see the following failure when starting the application:

This error message means that the native compilation used more advanced instruction sets, not supported by the CPU running the application. To work around that issue, add the following line to the application.properties:

Then, rebuild your native executable. This setting forces the native compilation to use an older instruction set, increasing the chance of compatibility.

To explicitly define the target architecture run native-image -march=list to get the supported configurations and then set -march to one of them, e.g., quarkus.native.additional-build-args=-march=x86-64-v4. If you are targeting an AMD64 host, -march=x86-64-v2 would work in most cases.