The English version of quarkus.io is the official project site. Translated sites are community supported on a best-effort basis.

Using OpenTelemetry Tracing

This guide explains how your Quarkus application can utilize OpenTelemetry (OTel) to provide distributed tracing for interactive web applications.

先决条件

完成这个指南,你需要:

  • 大概15分钟

  • 编辑器

  • JDK 17+ installed with JAVA_HOME configured appropriately

  • Apache Maven 3.9.8

  • Docker and Docker Compose or Podman, and Docker Compose

  • 如果你愿意的话,还可以选择使用Quarkus CLI

  • 如果你想构建原生可执行程序,可以选择安装Mandrel或者GraalVM,并正确配置(或者使用Docker在容器中进行构建)

应用结构

在本指南中,我们创建了一个简单的REST应用程序来演示分布式跟踪服务。

解决方案

我们建议您按照下面几节的说明,一步一步地创建应用程序。不过,您可以直接跳到已完成的例子。

克隆 Git 仓库: git clone https://github.com/quarkusio/quarkus-quickstarts.git ,或下载一个 存档

The solution is located in the opentelemetry-quickstart directory.

创建Maven项目

首先,我们需要一个新的项目。使用以下命令创建一个新的项目:

CLI
quarkus create app org.acme:opentelemetry-quickstart \
    --extension='rest,quarkus-opentelemetry' \
    --no-code
cd opentelemetry-quickstart

创建Grade项目,请添加 --gradle 或者 --gradle-kotlin-dsl 参数。

For more information about how to install and use the Quarkus CLI, see the Quarkus CLI guide.

Maven
mvn io.quarkus.platform:quarkus-maven-plugin:3.14.4:create \
    -DprojectGroupId=org.acme \
    -DprojectArtifactId=opentelemetry-quickstart \
    -Dextensions='rest,quarkus-opentelemetry' \
    -DnoCode
cd opentelemetry-quickstart

创建Grade项目,请添加 -DbuildTool=gradle 或者 -DbuildTool=gradle-kotlin-dsl 参数。

For Windows users:

  • If using cmd, (don’t use backward slash \ and put everything on the same line)

  • If using Powershell, wrap -D parameters in double quotes e.g. "-DprojectArtifactId=opentelemetry-quickstart"

This command generates the Maven project and imports the quarkus-opentelemetry extension, which includes the default OpenTelemetry support, and a gRPC span exporter for OTLP.

If you already have your Quarkus project configured, you can add the quarkus-opentelemetry extension to your project by running the following command in your project base directory:

CLI
quarkus extension add opentelemetry
Maven
./mvnw quarkus:add-extension -Dextensions='opentelemetry'
Gradle
./gradlew addExtension --extensions='opentelemetry'

这将在您的构建文件中添加以下内容:

pom.xml
<dependency>
    <groupId>io.quarkus</groupId>
    <artifactId>quarkus-opentelemetry</artifactId>
</dependency>
build.gradle
implementation("io.quarkus:quarkus-opentelemetry")

Examine the Jakarta REST resource

创建一个 src/main/java/org/acme/opentelemetry/TracedResource.java 文件,内容如下:

package org.acme.opentelemetry;

import jakarta.ws.rs.GET;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;
import org.jboss.logging.Logger;

@Path("/hello")
public class TracedResource {

    private static final Logger LOG = Logger.getLogger(TracedResource.class);

    @GET
    @Produces(MediaType.TEXT_PLAIN)
    public String hello() {
        LOG.info("hello");
        return "hello";
    }
}

请注意,应用程序中没有包含任何关于追踪的代码。默认情况下,不需要修改任何必要的代码就可以实现对发送到这个节点的请求进行追踪。

创建配置

There are no mandatory configurations for the extension to work.

By default, the exporters will send out data in batches, using the gRPC protocol and endpoint http://localhost:4317.

If you need to change any of the default property values, here is an example on how to configure the default OTLP gRPC Exporter within the application, using the src/main/resources/application.properties file:

quarkus.application.name=myservice (1)
quarkus.otel.exporter.otlp.endpoint=http://localhost:4317 (2)
quarkus.otel.exporter.otlp.headers=authorization=Bearer my_secret (3)
quarkus.log.console.format=%d{HH:mm:ss} %-5p traceId=%X{traceId}, parentId=%X{parentId}, spanId=%X{spanId}, sampled=%X{sampled} [%c{2.}] (%t) %s%e%n  (4)

# Alternative to the console log
quarkus.http.access-log.pattern="...traceId=%{X,traceId} spanId=%{X,spanId}" (5)
1 All telemetry created from the application will include an OpenTelemetry Resource attribute indicating the telemetry was created by the myservice application. If not set, it will default to the artifact id.
2 gRPC endpoint to send the telemetry. If not set, it will default to http://localhost:4317.
3 可选的gRPC消息头,通常用于认证
4 Add tracing information into log messages.
5 You can also only put the trace info into the access log. In this case you must omit the info in the console log format.

We provide signal agnostic configurations for the connection related properties, meaning that you can use the same properties for both tracing and metrics when you set:

quarkus.otel.exporter.otlp.endpoint=http://localhost:4317

If you need different configurations for each signal, you can use the specific properties:

quarkus.otel.exporter.otlp.traces.endpoint=http://trace-uri:4317 (1)
quarkus.otel.exporter.otlp.metrics.endpoint=http://metrics-uri:4317 (2)
1 The endpoint for the traces exporter.
2 The endpoint for the metrics exporter.

运行应用程序

First we need to start a system to visualise the OpenTelemetry data. We have 2 options:

  • Start an all-in-one Grafana OTel LGTM system for traces and metrics.

  • Jaeger system just for traces.

Grafana OTel LGTM option

This features a Quarkus Dev service including a Grafana for visualizing data, Loki to store logs, Tempo to store traces and Prometheus to store metrics. Also provides and OTel collector to receive the data.

Jaeger to see traces option

Configure and start the OpenTelemetry Collector to receive, process and export telemetry data to Jaeger that will display the captured traces.

Jaeger-all-in-one includes the Jaeger agent, an OTel collector, and the query service/UI. You do not need to install a separated collector. You can directly send the trace data to Jaeger (after enabling OTLP receivers there, see e.g. this blog entry for details).

您可以通过 docker-compose up -d 命令和下面的 docker-compose.yml 文件来启动OpenTelemetry Collector和Jaeger系统:

version: "2"
services:

  # Jaeger
  jaeger-all-in-one:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686" # Jaeger UI
      - "14268:14268" # Receive legacy OpenTracing traces, optional
      - "4317:4317"   # OTLP gRPC receiver
      - "4318:4318"   # OTLP HTTP receiver, not yet used by Quarkus, optional
      - "14250:14250" # Receive from external otel-collector, optional
    environment:
      - COLLECTOR_OTLP_ENABLED=true

You should remove the optional ports you don’t need them.

Start the application

现在我们准备运行我们的应用程序。如果使用 application.properties 来配置tracer:

CLI
quarkus dev
Maven
./mvnw quarkus:dev
Gradle
./gradlew --console=plain quarkusDev

或者如果通过JVM参数配置OTLP gRPC节点:

CLI
quarkus dev -Djvm.args="-Dquarkus.otel.exporter.otlp.traces.endpoint=http://localhost:4317"
Maven
./mvnw quarkus:dev -Djvm.args="-Dquarkus.otel.exporter.otlp.traces.endpoint=http://localhost:4317"
Gradle
./gradlew --console=plain quarkusDev -Djvm.args="-Dquarkus.otel.exporter.otlp.traces.endpoint=http://localhost:4317"

With the OpenTelemetry Collector, the Jaeger system and the application running, you can make a request to the provided endpoint:

$ curl http://localhost:8080/hello
hello

When the first request has been submitted, you will be able to see the tracing information in the logs:

10:49:02 INFO  traceId=, parentId=, spanId=, sampled= [io.quarkus] (main) Installed features: [cdi, opentelemetry, resteasy-client, resteasy, smallrye-context-propagation, vertx]
10:49:03 INFO  traceId=17ceb8429b9f25b0b879fa1503259456, parentId=3125c8bee75b7ad6, spanId=58ce77c86dd23457, sampled=true [or.ac.op.TracedResource] (executor-thread-1) hello
10:49:03 INFO  traceId=ad23acd6d9a4ed3d1de07866a52fa2df, parentId=, spanId=df13f5b45cf4d1e2, sampled=true [or.ac.op.TracedResource] (executor-thread-0) hello

然后访问 Jaeger界面 来查看追踪信息。

Hit CTRL+C or type q to stop the application.

JDBC

通过JDBC instrumentation 可以为您的应用程序的每个JDBC查询添加一个span。要启用它,请在您的构建文件中添加以下依赖:

pom.xml
<dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-jdbc</artifactId>
</dependency>
build.gradle
implementation("io.opentelemetry.instrumentation:opentelemetry-jdbc")

As it uses a dedicated JDBC datasource wrapper, you must enable telemetry for your datasource:

# enable tracing
quarkus.datasource.jdbc.telemetry=true

# configure datasource
quarkus.datasource.db-kind=postgresql
quarkus.datasource.jdbc.url=jdbc:postgresql://localhost:5432/mydatabase

其他配置

有些情况下需要对OpenTelemetry进行自定义配置。以下章节将展示如何对一些必要的属性进行正确配置。

ID生成器(ID Generator)

OpenTelemetry扩展将默认使用一个随机的 ID发生器 来创建trace和span标识符。

一些供应商特定协议需要一个自定义的ID生成器,您可以通过创建一个自定义生成器来覆盖默认的ID。OpenTelemetry扩展将检测 IdGenerator CDI Bean,并在配置tracer生成器时使用它。

@Singleton
public class CustomConfiguration {

    /** Creates a custom IdGenerator for OpenTelemetry */
    @Produces
    @Singleton
    public IdGenerator idGenerator() {
        return AwsXrayIdGenerator.getInstance();
    }
}

Propagators

OpenTelemetry propagates cross-cutting concerns through propagators that will share an underlying Context for storing state and accessing data across the lifespan of a distributed transaction.

默认情况下,OpenTelemetry扩展启用了 W3C Trace ContextW3C Baggage propagators,但是您可以通过设置 [参考配置] 中描述的 propagators 配置来选择任何支持的OpenTelemetry propagators。

Additional Propagators

  • b3 , b3multi , jaegerottrace propatagors 需要将 trace-propagators 扩展作为一个依赖添加到您的项目中。

pom.xml
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-extension-trace-propagators</artifactId>
</dependency>
build.gradle
implementation("io.opentelemetry:opentelemetry-extension-trace-propagators")
  • The xray propagator will need the aws extension to be added as a dependency to your project.

pom.xml
<dependency>
    <groupId>io.opentelemetry.contrib</groupId>
    <artifactId>opentelemetry-aws-xray-propagator</artifactId>
</dependency>
build.gradle
implementation("io.opentelemetry.contrib:opentelemetry-aws-xray-propagator")

Customise Propagator

To customise the propagation header you can implement the TextMapPropagatorCustomizer interface. This can be used, as an example, to restrict propagation of OpenTelemetry trace headers and prevent potentially sensitive data to be sent to third party systems.

/**
 * /**
 * Meant to be implemented by a CDI bean that provides arbitrary customization for the TextMapPropagator
 * that are to be registered with OpenTelemetry
 */
public interface TextMapPropagatorCustomizer {

    TextMapPropagator customize(Context context);

    interface Context {
        TextMapPropagator propagator();

        ConfigProperties otelConfigProperties();
    }
}

资源

See the main OpenTelemetry Guide resources section.

End User attributes

When enabled, Quarkus adds OpenTelemetry End User attributes as Span attributes. Before you enable this feature, verify that Quarkus Security extension is present and configured. More information about the Quarkus Security can be found in the Quarkus Security overview.

The attributes are only added when authentication has already happened on a best-efforts basis. Whether the End User attributes are added as Span attributes depends on authentication and authorization configuration of your Quarkus application. If you create custom Spans prior to the authentication, Quarkus cannot add the End User attributes to them. Quarkus is only able to add the attributes to the Span that is current after the authentication has been finished. Another important consideration regarding custom Spans is active CDI request context that is used to propagate Quarkus SecurityIdentity. In principle, Quarkus is able to add the End User attributes when the CDI request context has been activated for you before the custom Spans are created.

quarkus.otel.traces.eusp.enabled=true (1)
quarkus.http.auth.proactive=true (2)
1 Enable the End User Attributes feature so that the SecurityIdentity principal and roles are added as Span attributes. The End User attributes are personally identifiable information, therefore make sure you want to export them before you enable this feature.
2 Optionally enable proactive authentication. The best possible results are achieved when proactive authentication is enabled because the authentication happens sooner. A good way to determine whether proactive authentication should be enabled in your Quarkus application is to read the Quarkus Proactive authentication guide.
This feature is not supported when a custom Jakarta REST SecurityContexts is used.

采样器(Sampler)

A sampler decides whether a trace should be discarded or forwarded, effectively managing noise and reducing overhead by limiting the number of collected traces sent to the collector.

Quarkus comes equipped with a built-in sampler, and you also have the option to create your custom sampler.

To use the built-in sampler, you can configure it by setting the desired sampler parameters as detailed in the OpenTelemetry参考配置. As an example, you can configure the sampler to retain 50% of the traces:

# build time property only:
quarkus.otel.traces.sampler=traceidratio
# Runtime property:
quarkus.otel.traces.sampler.arg=0.5

An interesting use case for the sampler is to activate and deactivate tracing export at runtime, acording to this example:

# build time property only:
quarkus.otel.traces.sampler=traceidratio
# On (default). All traces are exported:
quarkus.otel.traces.sampler.arg=1.0
# Off. No traces are exported:
quarkus.otel.traces.sampler.arg=0.0

Quarkus 3.0 introduced breaking changes on the configuration.

Sampler related property names and values change to comply with the latest Java OpenTelemetry SDK. During a transition period it will be possible to set the new configuration values in the old property because we are mapping quarkus.opentelemetry.tracer.samplerquarkus.otel.traces.sampler.

If the sampler is parent based, there is no need to set, the now dropped property, quarkus.opentelemetry.tracer.sampler.parent-based.

The values you need to set on quarkus.opentelemetry.tracer.sampler are now:

Old Sampler config value New Sampler config value New Sampler config value (Parent based)

on

always_on

parentbased_always_on

off

always_off

parentbased_always_off

ratio

traceidratio

parentbased_traceidratio

If you need to use a custom sampler there are now 2 different ways:

Sampler CDI Producer

You can create a sampler CDI producer. The Quarkus OpenTelemetry extension will detect the Sampler CDI bean and will use it when configuring the Tracer.

@Singleton
public class CustomConfiguration {

    /** Creates a custom sampler for OpenTelemetry */
    @Produces
    @Singleton
    public Sampler sampler() {
        return JaegerRemoteSampler.builder()
        .setServiceName("my-service")
        .build();
    }
}

OTel Sampler SPI

This will use the SPI hooks available with the OTel Autoconfiguration. You can create a simple Sampler class:

public class CustomSPISampler implements Sampler {
    @Override
    public SamplingResult shouldSample(Context context,
            String s,
            String s1,
            SpanKind spanKind,
            Attributes attributes,
            List<LinkData> list) {
        // Do some sampling here
        return Sampler.alwaysOn().shouldSample(context, s, s1, spanKind, attributes, list);
    }

    @Override
    public String getDescription() {
        return "custom-spi-sampler-description";
    }
}

Then a Sampler Provider:

public class CustomSPISamplerProvider implements ConfigurableSamplerProvider {
    @Override
    public Sampler createSampler(ConfigProperties configProperties) {
        return new CustomSPISampler();
    }

    @Override
    public String getName() {
        return "custom-spi-sampler";
    }
}

Write the SPI loader text file at resources/META-INF/services with name io.opentelemetry.sdk.autoconfigure.spi.traces.ConfigurableSamplerProvider containing the full qualified name of the CustomSPISamplerProvider class.

Then activate on the configuration:

quarkus.otel.traces.sampler=custom-spi-sampler

As you can see, CDI is much simpler to work with.

其他的植入(instrumentation)

一些Quarkus扩展需要额外的代码来确保traces被传播到后续执行中。以下章节将展示跨越进程边界传播traces的必要条件。

本节中用到的植入方式已经过Quarkus测试,并且在标准和本地模式下都能工作。

CDI

Annotating a method in any CDI aware bean with the io.opentelemetry.instrumentation.annotations.WithSpan annotation will create a new Span and establish any required relationships with the current Trace context.

Annotating a method in any CDI aware bean with the io.opentelemetry.instrumentation.annotations.AddingSpanAttributes will not create a new span but will add annotated method parameters to attributes in the current span.

If a method is annotated by mistake with @AddingSpanAttributes and @WithSpan annotations, the @WithSpan annotation will take precedence.

Method parameters can be annotated with the io.opentelemetry.instrumentation.annotations.SpanAttribute annotation to indicate which method parameters should be part of the span. The parameter name can be customized as well.

例如:

@ApplicationScoped
class SpanBean {
    @WithSpan
    void span() {

    }

    @WithSpan("name")
    void spanName() {

    }

    @WithSpan(kind = SERVER)
    void spanKind() {

    }

    @WithSpan
    void spanArgs(@SpanAttribute(value = "arg") String arg) {

    }

    @AddingSpanAttributes
    void addArgumentToExistingSpan(@SpanAttribute(value = "arg") String arg) {

    }
}

Available OpenTelemetry CDI injections

As per MicroProfile Telemetry Tracing specification, Quarkus supports the CDI injections of the following classes:

  • io.opentelemetry.api.OpenTelemetry

  • io.opentelemetry.api.trace.Tracer

  • io.opentelemetry.api.trace.Span

  • io.opentelemetry.api.baggage.Baggage

You can inject these classes in any CDI enabled bean. For instance, the Tracer is particularly useful to start custom spans:

@Inject
Tracer tracer;

...

public void tracedWork() {
    Span span = tracer.spanBuilder("My custom span")
        .setAttribute("attr", "attr.value")
        .setParent(Context.current().with(Span.current()))
        .setSpanKind(SpanKind.INTERNAL)
        .startSpan();

    // traced work

    span.end();
}

Quarkus Messaging - Kafka

When using the Quarkus Messaging extension for Kafka, we are able to propagate the span into the Kafka Record with:

TracingMetadata tm = TracingMetadata.withPrevious(Context.current());
Message out = Message.of(...).withMetadata(tm);

The above creates a TracingMetadata object we can add to the Message being produced, which retrieves the OpenTelemetry Context to extract the current span for propagation.

Quarkus Security Events

Quarkus supports exporting of the Security events as OpenTelemetry Span events.

quarkus.otel.security-events.enabled=true (1)
1 Export Quarkus Security events as OpenTelemetry Span events.

Exporters

See the main OpenTelemetry Guide exporters section.

Quarkus core extensions instrumented with OpenTelemetry tracing

Disable parts of the automatic tracing

Automatic tracing instrumentation parts can be disabled by setting quarkus.otel.instrument.* properties to false.

Examples:

quarkus.otel.instrument.grpc=false
quarkus.otel.instrument.messaging=false
quarkus.otel.instrument.resteasy-client=false
quarkus.otel.instrument.rest=false
quarkus.otel.instrument.resteasy=false

OpenTelemetry参考配置

See the main OpenTelemetry Guide configuration reference.

Related content