The English version of quarkus.io is the official project site. Translated sites are community supported on a best-effort basis.

Blog An Extension for Long Running Activities

September 01, 2021 #extension

An Extension for Long Running Activities

By Michael Musgrove

简介

The Quarkus LRA extension is useful for building JAX-RS services that wish to definitively agree when an interaction has finished, with either a successful outcome or an unsuccessful one. In the successful case, all participants can clean up in the knowledge that all other services will do so as well. Conversely, in the unsuccessful case, participants know that each other will compensate for any actions performed during the interaction. This feature means that participating services can reach a consensus and achieve an atomic outcome.

We call the service interaction an LRA, short for Long Running Action. An LRA has specific properties and guarantees that aid in the construction of reliable service interactions. Each action is assigned a unique identifier (referred to as the LRA context) so that it can be distinguished from other LRAs.

Services start an LRA (or join an existing one) by marking a JAX-RS method with an @LRA annotation. When such a method is invoked the system will start the action and make its identifier available as a JAX-RS header called "Long-Running-Action". If the body of the method performs any JAX-RS invocations the header is automatically added to outgoing requests. In this way the target services can join in with the interaction (if they are also annotated with the @LRA annotation).

Any work performed by a method annotated in this way should be "compensatable" in the sense that if some service "cancels" the LRA then the service will be reliably notified that it should compensate for any work that it did. Each service is responsible for interpreting the notion of what it means to compensate, the only requirement is that it responds appropriately when it is notified that it should compensate. The service indicates how it should be notified by annotating one of its method with an @Compensate annotation. Refer to the javadoc for the @LRA annotation for the details of how to control the outcome of an LRA.

The extension provides an implementation of the MicroProfile LRA specification and its associated annotations.

LRA coordinators

The narayana-lra extension requires the presence of a running coordinator in the environment. Coordinators can be obtained from https://quay.io/repository/jbosstm/lra-coordinator or you can build your own coordinator using a maven pom that includes the appropriate dependencies. For the purpose of this blog we’ll show how to create one from scratch using the quarkus-maven-plugin. There is some extra information about configuring the coordinator in one of the narayana blogs.

Since the coordinator is just a JAX-RS resource we can build one using quarkus, adding the resteasy-jackson and rest-client extensions:

$ mvn io.quarkus:quarkus-maven-plugin:2.2.1.Final:create \
      -DprojectGroupId=org.acme \
      -DprojectArtifactId=narayana-lra-coordinator \
      -Dextensions="resteasy-jackson,rest-client"
$ cd narayana-lra-coordinator/
$ rm -rf src

Note that we removed the generated src directory because we just need the quarkus framework for running the coordinator.

Update the pom.xml file to add a dependency on the narayana coordinator implementation:

    <dependency>
      <groupId>org.jboss.narayana.rts</groupId>
      <artifactId>lra-coordinator-jar</artifactId>
      <version>5.12.0.Final</version>
    </dependency>

Now build it and run it in the background:

$ ./mvnw clean package
$ java -Dquarkus.http.port=50000 -jar target/quarkus-app/quarkus-run.jar &

Here we are running the coordinator on the default port used by the narayana-lra quarkus extension, namely 50000. You can verify that the coordinator is running okay by listing the current LRAs:

$ curl http://localhost:50000/lra-coordinator
[]

This snippet shows the request returning an empty json array.

We shall leave the coordinator running (on the default port) while we develop and test services that use LRAs. Towards the end of article we will show how to embed coordinators with services (NOTE: you cannot use this approach to run coordinators in native mode, a future extension will be provided to support this requirement).

JVM mode

To build a JAX-RS application with LRA support add the dependency io.quarkus:quarkus-narayana-lra to your application’s pom. This will add JAX-RS support and will also make the LRA annotations available when developing your services, it also registers a JAX-RS filter that automatically manages the participation of your services in LRAs.

As noted above, the guarantees (of eventual consistency) required by the LRA specification rely on the presence of a JAX-RS application that coordinates the services participating in the LRA. This component must be present when starting the interaction, when joining the interaction and when ending the interaction. If the coordinator becomes unavailable it should be restarted. Similarly, services participating in the LRA must be available during the end phase; the system will continue retrying a service until it indicates that it is finished with the LRA, and once a service has indicated successful compensation (or completion) it no longer takes part in the interaction (although it can register for a reliable notification that all services have finished compensating or completing). Although there can be many coordinators, at the time of writing, only one can manage a particular LRA.

Step 1: Create the application:

$ mvn io.quarkus:quarkus-maven-plugin:2.2.1.Final:create \
      -DprojectGroupId=org.acme \
      -DprojectArtifactId=narayana-lra-quickstart \
      -Dextensions="narayana-lra"
$ cd lra-quickstart

Note that if the coordinator is running on a port different from the default, i.e. 50000, then you will need to update the application config file (src/main/resources/application.properties) and specify the host and port:

quarkus.lra.coordinator-url=http://localhost:<port>/lra-coordinator

Verify that the generated application still works after these changes:

$ ./mvnw clean package

Step 2: Add LRA support

Now update the generated application so that the hello method will execute in the context of a Long Running Action:

Open the file src/main/java/org/acme/GreetingResource.java in an editor and annotate the hello method with an @LRA annotation (also inject the LRA context into the method using the JAX-RS javax.ws.rs.HeaderParam annotation). In addition, add two callback methods which will be called when the LRA is closed or cancelled.

The end result, including the imports, should look like the following:

package org.acme;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;

// Step 2a: Add imports for reading the LRA context and for using LRA annotations
import javax.ws.rs.HeaderParam;
import javax.ws.rs.PUT;
import javax.ws.rs.core.Response;
import java.net.URI;
import org.eclipse.microprofile.lra.annotation.ws.rs.LRA;
import org.eclipse.microprofile.lra.annotation.Compensate;
import org.eclipse.microprofile.lra.annotation.Complete;
import static org.eclipse.microprofile.lra.annotation.ws.rs.LRA.LRA_HTTP_CONTEXT_HEADER;

@Path("/hello")
public class GreetingResource {

    @GET
    @LRA // Step 2b: The method should run within an LRA
    @Produces(MediaType.TEXT_PLAIN)
    public String hello(@HeaderParam(LRA_HTTP_CONTEXT_HEADER) URI lraId /* Step 2c the context is useful for associating compensation logic */) {
        System.out.printf("hello with context %s%n", lraId);
        return "Hello RESTEasy";
    }

    // Step 2d: There must be a method to compensate for the action if it's cancelled
    @PUT
    @Path("compensate")
    @Compensate
    public Response compensateWork(@HeaderParam(LRA_HTTP_CONTEXT_HEADER) URI lraId) {
        System.out.printf("compensating %s%n", lraId);
        return Response.ok(lraId.toASCIIString()).build();
    }

    // Step 2e: An optional callback notifying that the LRA is closing
    @PUT
    @Path("complete")
    @Complete
    public Response completeWork(@HeaderParam(LRA_HTTP_CONTEXT_HEADER) URI lraId) {
        System.out.printf("completing %s%n", lraId);
        return Response.ok(lraId.toASCIIString()).build();
    }
}

With these changes, if you build the application and then invoke the hello method then an LRA will be started before the method is entered and ended after it finishes:

$ ./mvnw clean package
$ java -jar target/quarkus-app/quarkus-run.jar &
[1] 2389948
$ curl http://localhost:8080/hello
hello with context http://localhost:50000/lra-coordinator/0_ffffc0a8000e_8c1f_612a6e9b_a
completing http://localhost:50000/lra-coordinator/0_ffffc0a8000e_8c1f_612a6e9b_a
Hello RESTEasy

Make sure that the coordinator is still running otherwise you will see an error message similar to the following:

2021-08-11 14:27:45,779 WARN  [io.nar.lra] (executor-thread-0) LRA025025: Unable to process LRA annotations: -3: StartFailed (start LRA client request timed out, try again later) ()'

Notice the System.out messages indicating that the @Complete callback was invoked. Now kill the java process in preparation for the next step (the process id was printed on the console, in my example the pid is 2389948, so I typed kill 2389948).

Step 3: Extending the LRA across two service methods

In this step we will start an LRA but not end it when the method finishes by using the end element of the LRA annotation.

Define a second business method to do this:

    @GET
    @Path("/start")
    @LRA(end = false) // Step 3a: The method should run within an LRA
    @Produces(MediaType.TEXT_PLAIN)
    public String start(@HeaderParam(LRA_HTTP_CONTEXT_HEADER) URI lraId) {
        System.out.printf("hello with context %s%n", lraId);
        return lraId.toASCIIString();
    }

The only difference from the hello method is the @Path and @LRA annotations and that it returns the LRA id back to the caller. We will need this to set the header when we send a request to the hello method to finish the LRA (note that this header is also available in one of the JAX-RS response headers).

Kill the existing instance (kill 2389948) and rebuild the application (./mvnw package -DskipTests) and start it running in the background:

$ java -jar target/quarkus-app/quarkus-run.jar &
[1] 2495275

Start an LRA using curl to send a request the new method we have just added:

$ LRA_URL=$(curl http://localhost:8080/hello/start)
hello with context http://localhost:50000/lra-coordinator/0_ffffc0a8000e_a909_611a92ea_2

The start method was coded to return the LRA id and I have used bash to save it into an environment variable called LRA_URL. The original hello method used the default value of the end element of the @LRA annotation so if we call that method with an LRA context then the LRA will automatically close when the method finishes:

$ curl --header "Long-Running-Action: $LRA_URL" http://localhost:8080/hello
hello with context http://localhost:50000/lra-coordinator/0_ffffc0a8000e_a909_611a92ea_2
completing http://localhost:50000/lra-coordinator/0_ffffc0a8000e_a909_611a92ea_2
Hello RESTEasy

Notice that the completeWork method was invoked.

Step 4: Start an LRA in one microservice and end it in a different microservice

This step shows how two different microservices can coordinate their activities even though they have no coupling. Start a second instance of the hello application on a different port:

$ java -Dquarkus.http.port=8081 -jar target/quarkus-app/quarkus-run.jar &
[2] 2495369

Since we are still using the same application resource file and external coordinator there is no need to update the config.

Again, start an LRA using curl to send a request to the start method of the first service:

$ LRA_URL=$(curl http://localhost:8080/hello/start)
hello with context http://localhost:50000/lra-coordinator/0_ffffc0a8000e_a355_6113dede_11

and now end it in the second service (the one running on port 8081):

$ curl --header "Long-Running-Action: $LRA_URL" http://localhost:8081/hello
hello with context http://localhost:50000/lra-coordinator/0_ffffc0a8000e_a355_6113dede_11
completing http://localhost:50000/lra-coordinator/0_ffffc0a8000e_a355_6113dede_11
completing http://localhost:50000/lra-coordinator/0_ffffc0a8000e_a355_6113dede_11
Hello RESTEasy

Notice that both microservices indicated that they received the completion callback.

Terminate both java processes (kill 2495275 2495369).

Optional Step: using the MANDATORY element

Instead of using an existing method to close the LRA you might prefer to write one which expects there to be a context. In this case you would want to set the LRA.Type element:

    @GET
    @Path("/end")
    @LRA(value = LRA.Type.MANDATORY) // Step 3a: The method MUST be invoked with an LRA
    @Produces(MediaType.TEXT_PLAIN)
    public String end(@HeaderParam(LRA_HTTP_CONTEXT_HEADER) URI lraId) {
        return lraId.toASCIIString();
    }

Because the end method is annotated with @LRA(value = LRA.Type.MANDATORY), the context header must be present otherwise the method will return an error response code:

$ ./mvnw clean package -DskipTests
$ java -Dquarkus.http.port=8081 -jar target/quarkus-app/quarkus-run.jar &
[1] 300189
$ LRA_URL=$(curl http://localhost:8081/hello/start)
$ curl -v http://localhost:8081/hello/end
...
HTTP/1.1 412 Precondition Failed
...

whereas providing the LRA context header will work:

$ curl --header "Long-Running-Action: $LRA_URL" -I http://localhost:8081/hello/end
HTTP/1.1 200 OK
Content-Type: application/octet-stream
connection: keep-alive
$ kill 300189

Other LRA.Type element values may be useful, depending upon what your application is trying to achieve. For those readers familiar with JTA it is worth remarking that it was loosely modelled on the Java Transactional TxType annotation.

Native mode

in native mode only external coordinators (i.e. not embedded with the application) are supported (we will provide a coordinator extension in a later release to address this deficiency).

First build a native executable:

$ ./mvnw package -DskipTests -Pnative

Check that the external coordinator started in the section on coordinators is still running on port 50000 and then start the service as a native executable in the background. Note that if the coordinator is not running on the default port you would need to either pass in the location of a running coordinator as a Java system property (-Dquarkus.lra.coordinator-url=http://localhost:50000/lra-coordinator) or you can update the application config and rebuild the native executable.

Start an instance of the native service:

$ ./target/narayana-lra-quickstart-1.0.0-SNAPSHOT-runner &
[1] 2434426

Take a note of the process id (i.e. 2434426) since we will need to kill the process later.

Start a new LRA:

$ LRA_URL=$(curl http://localhost:8080/hello/start)

and end it in a different method:

$ curl --header "Long-Running-Action: $LRA_URL" http://localhost:8080/hello
hello with context http://localhost:50000/lra-coordinator/0_ffffc0a8000e_8479_612e13fa_2
completing http://localhost:50000/lra-coordinator/0_ffffc0a8000e_8479_612e13fa_2
Hello RESTEasy

Kill the service in preparation for the next step (kill 2434426) or just leave it running.

Failure handling

In this step we will start an LRA running in one service and then kill the service before the LRA has finished. Then we’ll use a second service to end the LRA and note that second service completes but that the LRA will still be in the Closing state because the participant in the first, failed, service still needs to complete. If the LRA is to reach the Closed state then the failed service must be restarted so that it can can respond to the Complete request.

Restart the fist service on the default port 8080 (and note its process id):

$ ./target/narayana-lra-quickstart-1.0.0-SNAPSHOT-runner &
[1] 2434936

and start a second service instance (on port 8082):

$ ./target/narayana-lra-quickstart-1.0.0-SNAPSHOT-runner -Dquarkus.http.port=8082 &
[2] 2434984

Start an LRA at the first service:

$ LRA_URL=$(curl http://localhost:8080/hello/start)
hello with context http://localhost:50000/lra-coordinator/0_ffffc0a8000e_a355_6113dede_34

Kill the first service

$ kill 2434936
2021-08-11 16:02:24,542 INFO  [io.quarkus] (Shutdown thread) narayana-lra-quickstart stopped in 0.003s

Now, with only the second service running, try ending the LRA:

$ curl --header "Long-Running-Action: $LRA_URL" http://localhost:8082/hello
hello with context http://localhost:50000/lra-coordinator/0_ffffc0a8000e_a355_6113dede_34
completing http://localhost:50000/lra-coordinator/0_ffffc0a8000e_a355_6113dede_34
Hello RESTEasy

The LRA will still be running, as you may verify by querying the coordinator (curl http://localhost:50000/lra-coordinator).

To finish the LRA restart the failed service (which was listening on port 8080):

$ ./target/narayana-lra-quickstart-1.0.0-SNAPSHOT-runner &
[3] 2435130

Recovery processing is periodic (the default period between recovery passes is 2 minutes). If you cannot wait 2 minutes then you may manually trigger a recovery cycle via the coordinators recovery endpoint as follows:

$ curl http://localhost:50000/lra-coordinator/recovery
completing http://localhost:50000/lra-coordinator/0_ffffc0a8000e_a355_6113dede_34
[]

The item to note here is that the restarted service received the completion notification (completing …​). The result of the request to run a recovery cycle is a json array of recovering LRAs (in this example the list is empty because the last LRA has now finished as indicated by the empty json array []).

Clean up by stopping the two services (kill 2434984 2435130).

Appendix 1

Embedded Coordinators

Since coordinators are just JAX-RS applications they can be embedded with JAX-RS services quite easily by adding the LRA coordinator dependency the applications pom.xml file:

    <dependency>
      <groupId>org.jboss.narayana.rts</groupId>
      <artifactId>lra-coordinator-jar</artifactId>
      <version>5.12.0.Final</version>
    </dependency>

and since, by default, quarkus only allows one application per deployment you will need to add the the following property to the application config file (src/main/resources/application.properties):

quarkus.resteasy.ignore-application-classes=true

The same caveats as described in the the section on coordinators still apply:

  • no support for native executables.

  • each instance requires dedicated storage for the coordinators' transaction logs (since sharing transaction stores is not currently supported).

The embedded coordinator will be available on the same port as the application (with path lra-coordinator), but note that the default coordinator port is 50000 so you will need to configure its location in the application config to tell the application to use it:

quarkus.http.port=8080
quarkus.lra.coordinator-url=http://localhost:8080/lra-coordinator

The location of the transaction logs cannot be configured in this way and must be configured via a system property (ObjectStoreEnvironmentBean.objectStoreDir):

$ java -DObjectStoreEnvironmentBean.objectStoreDir=target/lra-logs -jar target/quarkus-app/quarkus-run.jar &
[1] 2443349
$ LRA_URL=$(curl http://localhost:8080/hello/start)
02021-08-11 17:42:30,464 INFO  [com.arj.ats.arjuna] (executor-thread-1) ARJUNA012170: TransactionStatusManager started on port 35827 and host 127.0.0.1 with service com.arjuna.ats.arjuna.recovery.ActionStatusService
hello with context http://localhost:8080/lra-coordinator/0_ffffc0a8000e_a985_6113fdf6_2
$ curl http://localhost:8080/lra-coordinator
[{"lraId":"http://localhost:8080/lra-coordinator/0_ffffc0a8000e_a985_6113fdf6_2","clientId":"org.acme.GreetingResource#start","status":"Active","startTime":1628700150466,"finishTime":0,"httpStatus":204,"topLevel":true,"recovering":false}]

Now end the LRA in a different method:

$ curl --header "Long-Running-Action: $LRA_URL" http://localhost:8080/hello
hello with context http://localhost:8080/lra-coordinator/0_ffffc0a8000e_a985_6113fdf6_2
completing http://localhost:8080/lra-coordinator/0_ffffc0a8000e_a985_6113fdf6_2
Hello RESTEasy

A later extension will provide better support for embedded coordinators (including configuring them using standard quarkus mechanisms).

Quarkus is open. All dependencies of this project are available under the Apache Software License 2.0 or compatible license.

This website was built with Jekyll, is hosted on GitHub Pages and is completely open source. If you want to make it better, fork the website and show us what you’ve got.

Navigation
Follow Us
Get Help
Languages
Quarkus is made of community projects
CC by 3.0 | Privacy Policy Sponsored by