The English version of quarkus.io is the official project site. Translated sites are community supported on a best-effort basis.

Kafka开发服务(Dev Services)

If any Kafka-related extension is present (e.g. quarkus-messaging-kafka), Dev Services for Kafka automatically starts a Kafka broker in dev mode and when running tests. So, you don’t have to start a broker manually. The application is configured automatically.

Because starting a Kafka broker can be long, Dev Services for Kafka uses Redpanda, a Kafka compatible broker which starts in ~1 second.

启用/禁用Kafka开发服务

Kafka的开发服务是自动启用的,除非:

  • quarkus.kafka.devservices.enabled 被设置为 false

  • 配置了 kafka.bootstrap.servers

  • 所有的Reactive Messaging Kafka通道都设置了 bootstrap.servers 属性

Kafka的开发服务依赖于Docker来启动broker。如果您的环境不支持Docker,您需要手动启动broker,或者连接到一个已经运行的broker。您可以使用 kafka.bootstrap.servers 来配置broker地址。

共享的broker

大多数情况下,您需要在应用程序之间共享broker。Kafka的开发服务实现了一个 服务发现(service discovery) 机制,可以让您的多个在 开发 模式下运行的Quarkus应用程序共享一个broker。

Kafka开发服务用 quarkus-dev-service-kafka 标签来启动容器,该标签用于识别容器。

如果您需要多个(共享的)broker,您可以配置 quarkus.kafka.devservices.service-name 属性并指明broker的名称。它会查询一个具有相同名称的容器,如果找不到的话就启动一个新的容器。默认的服务名称是 kafka

在开发模式下,共享是默认启用的,但在测试模式下是禁用的。您可以用 quarkus.kafka.devservices.shared=false 停用共享。

设置端口

默认情况下,Kafka开发服务会随机挑选一个端口并配置应用程序。您可以通过配置 quarkus.kafka.devservices.port 属性来设置端口。

注意,Kafka的广告地址(advertised address)会自动配置为所选择的端口。

配置镜像

Dev Services for Kafka supports Redpanda, kafka-native and Strimzi (in Kraft mode) images.

Redpanda is a Kafka compatible event streaming platform. Because it provides a fast startup times, Dev Services defaults to Redpanda images from redpandadata/redpanda. You can select any version from https://hub.docker.com/r/redpandadata/redpanda.

kafka-native provides images of standard Apache Kafka distribution compiled to native binary using Quarkus and GraalVM. While still being experimental, it provides very fast startup times with small footprint.

Image type can be configured using

quarkus.kafka.devservices.provider=kafka-native

Strimzi provides container images and Operators for running Apache Kafka on Kubernetes. While Strimzi is optimized for Kubernetes, the images work perfectly in classic container environments. Strimzi container images run "genuine" Kafka broker on JVM, which is slower to start.

quarkus.kafka.devservices.provider=strimzi

对于Strimzi,您可以从 https://quay.io/repository/strimzi-test-container/test-container?tab=tags ,选择任何可以获得Kraft支持的Kafka版本(2.8.1及以上)的镜像

quarkus.kafka.devservices.image-name=quay.io/strimzi-test-container/test-container:0.106.0-kafka-3.7.0

配置Kafka主题

您可以配置Kafka开发服务来在broker启动后创建主题。主题以给定的分区数量以及1个副本创建。

下面的例子创建了一个名为 test ,有3个分区的主题,以及另一个名为 messages ,有2个分区的主题。

quarkus.kafka.devservices.topic-partitions.test=3
quarkus.kafka.devservices.topic-partitions.messages=2

如果具有某个指定名称的主题已经存在,那么创建会被跳过,并且不会尝试对已经存在的主题重新分区到不同的数量。

您可以通过 quarkus.kafka.devservices.topic-partitions-timeout 来配置在主题创建中所使用的Kafka管理员客户端调用的超时时间,默认为2秒。

Transactional and Idempotent producers support

By default, the Redpanda broker is configured to enable transactions and idempotence features. You can disable those using:

quarkus.kafka.devservices.redpanda.transaction-enabled=false
Redpanda transactions does not support exactly-once processing.

Configuration reference

Configuration property fixed at build time - All other configuration properties are overridable at runtime

Configuration property

类型

默认

If Dev Services for Kafka has been explicitly enabled or disabled. Dev Services are generally enabled by default, unless there is an existing configuration present. For Kafka, Dev Services starts a broker unless kafka.bootstrap.servers is set or if all the Reactive Messaging Kafka channel are configured with a bootstrap.servers.

Environment variable: QUARKUS_KAFKA_DEVSERVICES_ENABLED

Show more

boolean

Optional fixed port the dev service will listen to.

If not defined, the port will be chosen randomly.

Environment variable: QUARKUS_KAFKA_DEVSERVICES_PORT

Show more

int

Kafka dev service container type.

Redpanda, Strimzi and kafka-native container providers are supported. Default is redpanda.

Note that Strimzi and Kafka Native images are launched in Kraft mode.

Environment variable: QUARKUS_KAFKA_DEVSERVICES_PROVIDER

Show more

redpanda, strimzi, kafka-native

redpanda

The Kafka container image to use.

Dependent on the provider.

Environment variable: QUARKUS_KAFKA_DEVSERVICES_IMAGE_NAME

Show more

string

Indicates if the Kafka broker managed by Quarkus Dev Services is shared. When shared, Quarkus looks for running containers using label-based service discovery. If a matching container is found, it is used, and so a second one is not started. Otherwise, Dev Services for Kafka starts a new container.

The discovery uses the quarkus-dev-service-kafka label. The value is configured using the service-name property.

Container sharing is only used in dev mode.

Environment variable: QUARKUS_KAFKA_DEVSERVICES_SHARED

Show more

boolean

true

The value of the quarkus-dev-service-kafka label attached to the started container. This property is used when shared is set to true. In this case, before starting a container, Dev Services for Kafka looks for a container with the quarkus-dev-service-kafka label set to the configured value. If found, it will use this container instead of starting a new one. Otherwise, it starts a new container with the quarkus-dev-service-kafka label set to the specified value.

This property is used when you need multiple shared Kafka brokers.

Environment variable: QUARKUS_KAFKA_DEVSERVICES_SERVICE_NAME

Show more

string

kafka

The topic-partition pairs to create in the Dev Services Kafka broker. After the broker is started, given topics with partitions are created, skipping already existing topics. For example, quarkus.kafka.devservices.topic-partitions.test=2 will create a topic named test with 2 partitions.

The topic creation will not try to re-partition existing topics with different number of partitions.

Environment variable: QUARKUS_KAFKA_DEVSERVICES_TOPIC_PARTITIONS__TOPIC_NAME_

Show more

Map<String,Integer>

Timeout for admin client calls used in topic creation.

Defaults to 2 seconds.

Environment variable: QUARKUS_KAFKA_DEVSERVICES_TOPIC_PARTITIONS_TIMEOUT

Show more

Duration 

2S

Environment variables that are passed to the container.

Environment variable: QUARKUS_KAFKA_DEVSERVICES_CONTAINER_ENV__ENVIRONMENT_VARIABLE_NAME_

Show more

Map<String,String>

Enables transaction support. Also enables the producer idempotence. Find more info about Redpanda transaction support on https://vectorized.io/blog/fast-transactions/. Notice that KIP-447 (producer scalability for exactly once semantic) and KIP-360 (Improve reliability of idempotent/transactional producer) are not supported.

Environment variable: QUARKUS_KAFKA_DEVSERVICES_REDPANDA_TRANSACTION_ENABLED

Show more

boolean

true

Port to access the Redpanda HTTP Proxy (pandaproxy).

If not defined, the port will be chosen randomly.

Environment variable: QUARKUS_KAFKA_DEVSERVICES_REDPANDA_PROXY_PORT

Show more

int

About the Duration format

To write duration values, use the standard java.time.Duration format. See the Duration#parse() Java API documentation for more information.

You can also use a simplified format, starting with a number:

  • If the value is only a number, it represents time in seconds.

  • If the value is a number followed by ms, it represents time in milliseconds.

In other cases, the simplified format is translated to the java.time.Duration format for parsing:

  • If the value is a number followed by h, m, or s, it is prefixed with PT.

  • If the value is a number followed by d, it is prefixed with P.

Related content