Connecting to an Elasticsearch cluster
Elasticsearch is a well known full text search engine and NoSQL datastore.
In this guide, we will see how you can get your REST services to interact with an Elasticsearch cluster.
Quarkus provides two ways of accessing Elasticsearch:
-
The lower level REST Client
-
The Elasticsearch Java client
A third Quarkus extension for the "high level REST Client" used to exist, but was removed as this client has been deprecated by Elastic and has some licensing issues. |
先决条件
完成这个指南,你需要:
-
大概15分钟
-
编辑器
-
JDK 17+ installed with
JAVA_HOME
configured appropriately -
Apache Maven 3.9.9
-
如果你愿意的话,还可以选择使用Quarkus CLI
-
如果你想构建原生可执行程序,可以选择安装Mandrel或者GraalVM,并正确配置(或者使用Docker在容器中进行构建)
-
Elasticsearch installed or Docker installed
应用结构
本指南中构建的应用程序非常简单:用户可以使用一个表单在列表中添加元素并更新列表。
浏览器和服务器之间的所有信息都被格式化为JSON。
The elements are stored in Elasticsearch.
创建Maven项目
首先,我们需要一个新的工程项目。用以下命令创建一个新项目:
For Windows users:
-
If using cmd, (don’t use backward slash
\
and put everything on the same line) -
If using Powershell, wrap
-D
parameters in double quotes e.g."-DprojectArtifactId=elasticsearch-quickstart"
This command generates a Maven structure importing the Quarkus REST (formerly RESTEasy Reactive), Jackson, and Elasticsearch low level REST client extensions.
The Elasticsearch low level REST client comes with the quarkus-elasticsearch-rest-client
extension
that has been added to your build file.
If you want to use the Elasticsearch Java client instead, replace the quarkus-elasticsearch-rest-client
extension by the quarkus-elasticsearch-java-client
extension.
We use the |
To add the extensions to an existing project, follow the instructions below.
For the Elasticsearch low level REST client, add the following dependency to your build file:
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-elasticsearch-rest-client</artifactId>
</dependency>
implementation("io.quarkus:quarkus-elasticsearch-rest-client")
For the Elasticsearch Java client, add the following dependency to your build file:
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-elasticsearch-java-client</artifactId>
</dependency>
implementation("io.quarkus:quarkus-elasticsearch-java-client")
创建你的第一个JSON REST服务
在这个例子中,我们将创建一个应用程序来管理fruit列表。
首先,让我们创建 Fruit
实体类,如下所示:
package org.acme.elasticsearch;
public class Fruit {
public String id;
public String name;
public String color;
}
这非常的简单。需要注意的一件事是, JSON 序列化层需要具有默认构造函数。
Now create a org.acme.elasticsearch.FruitService
that will be the business layer of our application
and will store/load the fruits from the Elasticsearch instance.
Here we use the low level REST client, if you want to use the Java API client instead,
follow the instructions in the Using the Elasticsearch Java Client paragraph instead.
package org.acme.elasticsearch;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.stream.Collectors;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.inject.Inject;
import org.apache.http.entity.ContentType;
import org.apache.http.entity.StringEntity;
import org.apache.http.util.EntityUtils;
import org.elasticsearch.client.Request;
import org.elasticsearch.client.Response;
import org.elasticsearch.client.RestClient;
import io.vertx.core.json.JsonArray;
import io.vertx.core.json.JsonObject;
@ApplicationScoped
public class FruitService {
@Inject
RestClient restClient; (1)
public void index(Fruit fruit) throws IOException {
Request request = new Request(
"PUT",
"/fruits/_doc/" + fruit.id); (2)
request.setJsonEntity(JsonObject.mapFrom(fruit).toString()); (3)
restClient.performRequest(request); (4)
}
public void index(List<Fruit> list) throws IOException {
var entityList = new ArrayList<JsonObject>();
for (var fruit : list) {
entityList.add(new JsonObject().put("index", new JsonObject()(5)
.put("_index", "fruits").put("_id", fruit.id)));
entityList.add(JsonObject.mapFrom(fruit));
}
Request request = new Request(
"POST", "fruits/_bulk?pretty");
request.setEntity(new StringEntity(
toNdJsonString(entityList),(6)
ContentType.create("application/x-ndjson")));(7)
restClient.performRequest(request);
}
public void delete(List<String> identityList) throws IOException {
var entityList = new ArrayList<JsonObject>();
for (var id : identityList) {
entityList.add(new JsonObject().put("delete",
new JsonObject().put("_index", "fruits").put("_id", id)));(8)
}
Request request = new Request(
"POST", "fruits/_bulk?pretty");
request.setEntity(new StringEntity(
toNdJsonString(entityList),
ContentType.create("application/x-ndjson")));
restClient.performRequest(request);
}
public Fruit get(String id) throws IOException {
Request request = new Request(
"GET",
"/fruits/_doc/" + id);
Response response = restClient.performRequest(request);
String responseBody = EntityUtils.toString(response.getEntity());
JsonObject json = new JsonObject(responseBody); (9)
return json.getJsonObject("_source").mapTo(Fruit.class);
}
public List<Fruit> searchByColor(String color) throws IOException {
return search("color", color);
}
public List<Fruit> searchByName(String name) throws IOException {
return search("name", name);
}
private List<Fruit> search(String term, String match) throws IOException {
Request request = new Request(
"GET",
"/fruits/_search");
//construct a JSON query like {"query": {"match": {"<term>": "<match"}}
JsonObject termJson = new JsonObject().put(term, match);
JsonObject matchJson = new JsonObject().put("match", termJson);
JsonObject queryJson = new JsonObject().put("query", matchJson);
request.setJsonEntity(queryJson.encode());
Response response = restClient.performRequest(request);
String responseBody = EntityUtils.toString(response.getEntity());
JsonObject json = new JsonObject(responseBody);
JsonArray hits = json.getJsonObject("hits").getJsonArray("hits");
List<Fruit> results = new ArrayList<>(hits.size());
for (int i = 0; i < hits.size(); i++) {
JsonObject hit = hits.getJsonObject(i);
Fruit fruit = hit.getJsonObject("_source").mapTo(Fruit.class);
results.add(fruit);
}
return results;
}
private static String toNdJsonString(List<JsonObject> objects) {
return objects.stream()
.map(JsonObject::encode)
.collect(Collectors.joining("\n", "", "\n"));
}
}
1 | We inject an Elasticsearch low level RestClient into our service. |
2 | We create an Elasticsearch request. |
3 | We use Vert.x JsonObject to serialize the object before sending it to Elasticsearch, you can use whatever you want to serialize your objects to JSON. |
4 | We send the request (indexing request here) to Elasticsearch. |
5 | As we index collection of objects we should use index , create or update action. |
6 | We use toNdJsonString(entityList) call to produce output like below
|
7 | Pass the content type that is expected by the search backend for bulk requests. |
8 | The bulk API’s delete operation JSON already contains all the required information; hence, there is no request body following this operation in the Bulk API request body.
|
9 | In order to deserialize the object from Elasticsearch, we again use Vert.x JsonObject. |
Now, create the org.acme.elasticsearch.FruitResource
class as follows:
package org.acme.elasticsearch;
import java.io.IOException;
import java.net.URI;
import java.util.List;
import java.util.UUID;
import jakarta.inject.Inject;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.POST;
import jakarta.ws.rs.DELETE;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.PathParam;
import jakarta.ws.rs.QueryParam;
import jakarta.ws.rs.core.Response;
import jakarta.ws.rs.BadRequestException;
@Path("/fruits")
public class FruitResource {
@Inject
FruitService fruitService;
@POST
public Response index(Fruit fruit) throws IOException {
if (fruit.id == null) {
fruit.id = UUID.randomUUID().toString();
}
fruitService.index(fruit);
return Response.created(URI.create("/fruits/" + fruit.id)).build();
}
@Path("bulk")
@DELETE
public Response delete(List<String> identityList) throws IOException {
fruitService.delete(identityList);
return Response.ok().build();
}
@Path("bulk")
@POST
public Response index(List<Fruit> list) throws IOException {
fruitService.index(list);
return Response.ok().build();
}
@GET
@Path("/{id}")
public Fruit get(String id) throws IOException {
return fruitService.get(id);
}
@GET
@Path("/search")
public List<Fruit> search(@RestQuery String name, @RestQuery String color) throws IOException {
if (name != null) {
return fruitService.searchByName(name);
} else if (color != null) {
return fruitService.searchByColor(color);
} else {
throw new BadRequestException("Should provide name or color query parameter");
}
}
}
The implementation is pretty straightforward and you just need to define your endpoints using the Jakarta REST annotations and use the FruitService
to list/add new fruits.
Configuring Elasticsearch
The main property to configure is the URL to connect to the Elasticsearch cluster.
For a typical clustered Elasticsearch service, a sample configuration would look like the following:
# configure the Elasticsearch client for a cluster of two nodes
quarkus.elasticsearch.hosts = elasticsearch1:9200,elasticsearch2:9200
In our case, we are using a single instance running on localhost:
# configure the Elasticsearch client for a single instance on localhost
quarkus.elasticsearch.hosts = localhost:9200
If you need a more advanced configuration, you can find the comprehensive list of supported configuration properties at the end of this guide.
开发服务
Quarkus supports a feature called Dev Services that allows you to start various containers without any config.
In the case of Elasticsearch, this support extends to the default Elasticsearch connection.
What that means practically is that, if you have not configured quarkus.elasticsearch.hosts
, Quarkus will automatically
start an Elasticsearch container when running tests or dev mode, and automatically configure the connection.
When running the production version of the application, the Elasticsearch connection needs to be configured as usual,
so if you want to include a production database config in your application.properties
and continue to use Dev Services
we recommend that you use the %prod.
profile to define your Elasticsearch settings.
For more information you can read the Dev Services for Elasticsearch guide.
Programmatically Configuring Elasticsearch
On top of the parametric configuration, you can also programmatically apply additional configuration to the client by implementing a RestClientBuilder.HttpClientConfigCallback
and annotating it with ElasticsearchClientConfig
. You may provide multiple implementations and configuration provided by each implementation will be applied in a randomly ordered cascading manner.
For example, when accessing an Elasticsearch cluster that is set up for TLS on the HTTP layer, the client needs to trust the certificate that Elasticsearch is using. The following is an example of setting up the client to trust the CA that has signed the certificate that Elasticsearch is using, when that CA certificate is available in a PKCS#12 keystore.
import io.quarkus.elasticsearch.restclient.lowlevel.ElasticsearchClientConfig;
import org.apache.http.impl.nio.client.HttpAsyncClientBuilder;
import org.apache.http.ssl.SSLContextBuilder;
import org.apache.http.ssl.SSLContexts;
import org.elasticsearch.client.RestClientBuilder;
import jakarta.enterprise.context.Dependent;
import javax.net.ssl.SSLContext;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.security.KeyStore;
@ElasticsearchClientConfig
public class SSLContextConfigurator implements RestClientBuilder.HttpClientConfigCallback {
@Override
public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
try {
String keyStorePass = "password-for-keystore";
Path trustStorePath = Paths.get("/path/to/truststore.p12");
KeyStore truststore = KeyStore.getInstance("pkcs12");
try (InputStream is = Files.newInputStream(trustStorePath)) {
truststore.load(is, keyStorePass.toCharArray());
}
SSLContextBuilder sslBuilder = SSLContexts.custom()
.loadTrustMaterial(truststore, null);
SSLContext sslContext = sslBuilder.build();
httpClientBuilder.setSSLContext(sslContext);
} catch (Exception e) {
throw new RuntimeException(e);
}
return httpClientBuilder;
}
}
See Elasticsearch documentation for more details on this particular example.
Classes marked with |
Running an Elasticsearch cluster
As by default, the Elasticsearch client is configured to access a local Elasticsearch cluster on port 9200 (the default Elasticsearch port), if you have a local running instance on this port, there is nothing more to do before being able to test it!
If you want to use Docker to run an Elasticsearch instance, you can use the following command to launch one:
docker run --name elasticsearch -e "discovery.type=single-node" -e "ES_JAVA_OPTS=-Xms512m -Xmx512m"\
-e "cluster.routing.allocation.disk.threshold_enabled=false" -e "xpack.security.enabled=false"\
--rm -p 9200:9200 docker.io/elastic/elasticsearch:8.15.0
运行应用程序
Let’s start our application in dev mode:
quarkus dev
./mvnw quarkus:dev
./gradlew --console=plain quarkusDev
You can add new fruits to the list via the following curl command:
curl localhost:8080/fruits -d '{"name": "bananas", "color": "yellow"}' -H "Content-Type: application/json"
And search for fruits by name or color via the following curl command:
curl localhost:8080/fruits/search?color=yellow
Using the Elasticsearch Java Client
Here is a version of the FruitService
using the Elasticsearch Java Client instead of the low level one:
import java.io.IOException;
import java.io.StringReader;
import java.util.List;
import java.util.stream.Collectors;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.inject.Inject;
import co.elastic.clients.elasticsearch.ElasticsearchClient;
import co.elastic.clients.elasticsearch.core.IndexRequest;
import co.elastic.clients.elasticsearch._types.FieldValue;
import co.elastic.clients.elasticsearch._types.query_dsl.QueryBuilders;
import co.elastic.clients.elasticsearch.core.search.Hit;
import co.elastic.clients.elasticsearch.core.search.HitsMetadata;
import co.elastic.clients.elasticsearch.core.GetRequest;
import co.elastic.clients.elasticsearch.core.GetResponse;
import co.elastic.clients.elasticsearch.core.SearchRequest;
import co.elastic.clients.elasticsearch.core.SearchResponse;
import co.elastic.clients.elasticsearch.core.BulkRequest;
import co.elastic.clients.elasticsearch.core.BulkResponse;
@ApplicationScoped
public class FruitService {
@Inject
ElasticsearchClient client; (1)
public void index(Fruit fruit) throws IOException {
IndexRequest<Fruit> request = IndexRequest.of( (2)
b -> b.index("fruits")
.id(fruit.id)
.document(fruit)); (3)
client.index(request); (4)
}
public void index(List<Fruit> list) throws IOException {
BulkRequest.Builder br = new BulkRequest.Builder();
for (var fruit : list) {
br.operations(op -> op
.index(idx -> idx.index("fruits").id(fruit.id).document(fruit)));
}
BulkResponse result = client.bulk(br.build());
if (result.errors()) {
throw new RuntimeException("The indexing operation encountered errors.");
}
}
public void delete(List<String> list) throws IOException {
BulkRequest.Builder br = new BulkRequest.Builder();
for (var id : list) {
br.operations(op -> op.delete(idx -> idx.index("fruits").id(id)));
}
BulkResponse result = client.bulk(br.build());
if (result.errors()) {
throw new RuntimeException("The indexing operation encountered errors.");
}
}
public Fruit get(String id) throws IOException {
GetRequest getRequest = GetRequest.of(
b -> b.index("fruits")
.id(id));
GetResponse<Fruit> getResponse = client.get(getRequest, Fruit.class);
if (getResponse.found()) {
return getResponse.source();
}
return null;
}
public List<Fruit> searchByColor(String color) throws IOException {
return search("color", color);
}
public List<Fruit> searchByName(String name) throws IOException {
return search("name", name);
}
private List<Fruit> search(String term, String match) throws IOException {
SearchRequest searchRequest = SearchRequest.of(
b -> b.index("fruits")
.query(QueryBuilders.match().field(term).query(FieldValue.of(match)).build()._toQuery()));
SearchResponse<Fruit> searchResponse = client.search(searchRequest, Fruit.class);
HitsMetadata<Fruit> hits = searchResponse.hits();
return hits.hits().stream().map(hit -> hit.source()).collect(Collectors.toList());
}
}
1 | We inject an ElasticsearchClient inside the service. |
2 | We create an Elasticsearch index request using a builder. |
3 | We directly pass the object to the request as the Java API client has a serialization layer. |
4 | We send the request to Elasticsearch. |
Hibernate Search Elasticsearch
Quarkus supports Hibernate Search with Elasticsearch via the quarkus-hibernate-search-orm-elasticsearch
extension.
Hibernate Search Elasticsearch allows to synchronize your Jakarta Persistence entities to an Elasticsearch cluster and offers a way to query your Elasticsearch cluster using the Hibernate Search API.
If you are interested in it, please consult the Hibernate Search with Elasticsearch guide.
Cluster Health Check
If you are using the quarkus-smallrye-health
extension, both extensions will automatically add a readiness health check
to validate the health of the cluster.
So when you access the /q/health/ready
endpoint of your application, you will have information about the cluster status.
It uses the cluster health endpoint, the check will be down if the status of the cluster is red, or the cluster is not available.
This behavior can be disabled by setting the quarkus.elasticsearch.health.enabled
property to false
in your application.properties
.
构建一个本地可执行文件
You can use both clients in a native executable.
你可以使用常用命令构建本机可执行文件:
quarkus build --native
./mvnw install -Dnative
./gradlew build -Dquarkus.native.enabled=true
Running it is as simple as executing ./target/elasticsearch-low-level-client-quickstart-1.0.0-SNAPSHOT-runner
.
然后你可以使用的浏览器访问 <a href="http://localhost:8080/fruits.html" class="bare">http://localhost:8080/fruits.html</a>
来使用你的应用程序。
解决方案
Accessing an Elasticsearch cluster from the low level REST client or the Elasticsearch Java client is easy with Quarkus as it provides easy configuration, CDI integration and native support for it.
配置参考
Configuration property fixed at build time - All other configuration properties are overridable at runtime
Configuration property |
类型 |
默认 |
||
---|---|---|---|---|
Whether a health check is published in case the smallrye-health extension is present. Environment variable: Show more |
boolean |
|
||
The list of hosts of the Elasticsearch servers. Environment variable: Show more |
list of host:port |
|
||
The protocol to use when contacting Elasticsearch servers. Set to "https" to enable SSL/TLS. Environment variable: Show more |
string |
|
||
The username for basic HTTP authentication. Environment variable: Show more |
string |
|||
The password for basic HTTP authentication. Environment variable: Show more |
string |
|||
The connection timeout. Environment variable: Show more |
|
|||
The socket timeout. Environment variable: Show more |
|
|||
The maximum number of connections to all the Elasticsearch servers. Environment variable: Show more |
int |
|
||
The maximum number of connections per Elasticsearch server. Environment variable: Show more |
int |
|
||
The number of IO thread. By default, this is the number of locally detected processors. Thread counts higher than the number of processors should not be necessary because the I/O threads rely on non-blocking operations, but you may want to use a thread count lower than the number of processors. Environment variable: Show more |
int |
|||
Defines if automatic discovery is enabled. Environment variable: Show more |
boolean |
|
||
Refresh interval of the node list. Environment variable: Show more |
|
|||
类型 |
默认 |
|||
Whether this Dev Service should start with the application in dev mode or tests. Dev Services are enabled by default
unless connection configuration (e.g. Environment variable: Show more |
boolean |
|||
Optional fixed port the dev service will listen to. If not defined, the port will be chosen randomly. Environment variable: Show more |
int |
|||
The Elasticsearch distribution to use. Defaults to a distribution inferred from the explicitly configured Environment variable: Show more |
|
|||
The Elasticsearch container image to use. Defaults depend on the configured
Environment variable: Show more |
string |
|||
The value for the ES_JAVA_OPTS env variable. Environment variable: Show more |
string |
|
||
Whether the Elasticsearch server managed by Quarkus Dev Services is shared. When shared, Quarkus looks for running containers using label-based service discovery. If a matching container is found, it is used, and so a second one is not started. Otherwise, Dev Services for Elasticsearch starts a new container. The discovery uses the Container sharing is only used in dev mode. Environment variable: Show more |
boolean |
|
||
The value of the This property is used when This property is used when you need multiple shared Elasticsearch servers. Environment variable: Show more |
string |
|
||
Environment variables that are passed to the container. Environment variable: Show more |
Map<String,String> |
|||
Whether to keep Dev Service containers running after a dev mode session or test suite execution to reuse them in the next dev mode session or test suite execution. Within a dev mode session or test suite execution, Quarkus will always reuse Dev Services as long as their configuration (username, password, environment, port bindings, …) did not change. This feature is specifically about keeping containers running when Quarkus is not running to reuse them across runs.
This configuration property is set to Environment variable: Show more |
boolean |
|
About the Duration format
To write duration values, use the standard You can also use a simplified format, starting with a number:
In other cases, the simplified format is translated to the
|