In a previous post, we demonstrated client-side transaction retries by using meta-annotations and Aspect Oriented Programming (AOP) in Spring Boot.
In this post, we'll use a similar concept for a different Java stack: JavaEE (or JakartaEE as it's known today) and interceptors using the Contexts and Dependency Injection (CDI) framework.
Transaction Retries in JavaEE
This article demonstrates an AOP-driven retry strategy for JavaEE apps using the following stack:
Stateless session beans with bean-managed transactions
@AroundAdvice interceptor for retries
@TransactionBoundary meta-annotation with the interceptor binding
JAX-RS REST endpoint for testing
TomEE 8 as an embedded container with the web profile
JPA and Hibernate for data access
Source Code
The source code for examples of this article can be found on GitHub.
Solution
The Interceptor or proxy pattern is generally used to add cross-cutting functionality or logic in an application without code duplication. Transaction management is a typical cross-cutting concern which is provided out of the box in JavaEE. Public bean methods in stateless/stateful session beans are automatically transactional, given that container-managed transactions (CMTs) are used.
In this example, the service method acting as a boundary should always use REQUIRES_NEW propagation.
@Stateless
public class OrderService {
@PersistenceContext(unitName = "orderSystemPU")
private EntityManager entityManager;
@TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
public List<Order> findAllOrders() {
CriteriaQuery<Order> cq = entityManager.getCriteriaBuilder().createQuery(Order.class);
cq.select(cq.from(Order.class));
return entityManager.createQuery(cq).getResultList();
}
}
A natural extension to transparent transaction management is transaction retries. If a transaction fails with a transient serialization error, it should be automatically retried several times. This behaviour fits well into an interceptor that we'll use through an interceptor binding (annotation).
In the same example, like this:
@Stateless
public class OrderService {
@PersistenceContext(unitName = "orderSystemPU")
private EntityManager entityManager;
@TransactionBoundary
public Order updateOrder(Order order) {
entityManager.merge(order);
return order;
}
}
To mark classes and methods as repeatable transaction boundaries, let's first create the interceptor binding:
@Inherited
@InterceptorBinding
@Target({TYPE, METHOD})
@Retention(RUNTIME)
public @interface TransactionBoundary {
}
After creating the interceptor binding we need to create the actual interceptor implementation:
@TransactionBoundary
@Interceptor
@Priority(Interceptor.Priority.APPLICATION)
public class TransactionRetryInterceptor {
public static final int MAX_RETRY_ATTEMPTS = 10;
public static final int MAX_BACKOFF_TIME_MILLIS = 15000;
private static final ThreadLocalRandom RAND = ThreadLocalRandom.current();
@PersistenceContext(unitName = "orderSystemPU")
private EntityManager entityManager;
@Inject
private TransactionService transactionService;
@Inject
private Logger logger;
@AroundInvoke
public Object aroundTransactionBoundary(InvocationContext ctx) throws Exception {
Assert.isFalse(entityManager.isJoinedToTransaction(), "Expected no transaction!");
logger.info("Intercepting transactional method in retry loop: {}", ctx.getMethod().toGenericString());
for (int attempt = 1; attempt < MAX_RETRY_ATTEMPTS; attempt++) {
try {
Object rv = transactionService.executeWithinTransaction(ctx::proceed);
if (attempt > 1) {
logger.info("Recovered from transient error (attempt {}): {}",
attempt, ctx.getMethod().toGenericString());
} else {
logger.info("Transactional method completed (attempt {}): {}",
attempt, ctx.getMethod().toGenericString());
}
return rv;
} catch (Exception ex) {
Throwable t = ExceptionUtils.getMostSpecificCause(ex);
if (t instanceof SQLException) {
SQLException sqlException = (SQLException) t;
if (PSQLState.SERIALIZATION_FAILURE.getState().equals(sqlException.getSQLState())) {
long backoffMillis = Math.min((long) (Math.pow(2, attempt) + RAND.nextInt(0, 1000)),
MAX_BACKOFF_TIME_MILLIS);
logger.warn("Detected transient error (attempt {}) backoff for {}ms: {}",
attempt, backoffMillis, sqlException);
try {
Thread.sleep(backoffMillis);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
} else {
logger.info("Detected non-transient error (propagating): {}", t.getMessage());
throw ex;
}
} else {
logger.info("Detected non-transient error (propagating): {}", t.getMessage());
throw ex;
}
}
}
throw new SQLTransactionRollbackException("Too many serialization conflicts - giving up retries!");
}
}
There are a few things to highlight here. First is that beans need to use bean-managed transactions (BMTs) to defer the transaction marker to a dedicated service, invoked from the interceptor. Otherwise, when using container-managed transactions, the interceptor is called within the scope of the same transaction which may fail with a serialization error. A retry at that point would do no good since this logic must be applied before transactions are created.
Hence the purpose of `TransactionService` which makes a callback to the original business method within a transaction scope.
@Stateless
public class TransactionService {
@TransactionAttribute(REQUIRES_NEW)
public <T> T executeWithinTransaction(final Callable<T> task) throws Exception {
return task.call();
}
}
The actual stateless service bean declares its intent to use bean-managed transactions:
@Stateless
@TransactionManagement(TransactionManagementType.BEAN)
public class OrderService {
...
To try this out, we'll use a very unsophisticated order system designed to produce unrepeatable read (aka read/write) conflicts to activate the retry mechanism.
We will use multiple concurrent connections and read and write to the same key but with different values. Without client-side retries, one of the transactions would be rolled back with a serialization conflict. With retries, the failed transaction will be retried and eventually run to completion, transparently towards the application logic.
Building
Prerequisites
JDK8+ with 1.8 language level (OpenJDK compatible)
Maven 3+ (optional, embedded)
CockroachDB v22.1+ database
Install the JDK (Linux):
sudo apt-get -qq install -y openjdk-8-jdk
Clone the project
git clone git@github.com/kai-niemi/retry-demo.git
cd retry-demo
Build the project
chmod +x mvnw
./mvnw clean install
Setup
Create the database:
cockroach sql --insecure --host=localhost -e "CREATE database orders"
Create the schema:
cockroach sql --insecure --host=locahlost --database orders < src/resources/conf/create.sql
Start the app:
../mvnw clean install tomee:run
The default listen port is 8090 (can be changed in pom.xml):
Usage
Open another shell and check that the service is up and connected to the DB:
curl http://localhost:8090/api
This prints out an order form template that we will use to create new orders:
curl http://localhost:8090/api/order/template| jq
Alternatively, pipe it to a file:
curl http://localhost:8090/api/order/template > form.json
Create a new purchase order:
curl http://localhost:8090/api/order -i -X POST \
-H 'Content-Type: application/json' \
-d '{
"billAddress": {
"address1": "Street 1.1",
"address2": "Street 1.2",
"city": "City 1",
"country": "Country 1",
"postcode": "Code 1"
},
"customerId": -1,
"deliveryAddress": {
"address1": "Street 2.1",
"address2": "Street 2.2",
"city": "City 2",
"country": "Country 2",
"postcode": "Code 2"
},
"requestId": "bc3cba97-dee9-41b2-9110-2f5dfc2c5dae"
}'
Or using the file:
curl http://localhost:8090/api/order -H "Content-Type:application/json" -X POST \
-d "@form.json"
Produce a Read/Write Conflict
Assuming we have an order with ID 1 in status PLACED. We will now read that order and change the status to something else by using concurrent transactions. This is known as the unrepeatable read conflict, prevented by 1SR from happening.
To have a predictable outcome, we'll use two sessions with a controllable delay between the read and write operations.
Overview of SQL operations:
select * from purchase_order where id=1; -- T1
-- status is `PLACED`
wait 5s -- T1
select * from purchase_order where id=1; -- T2
wait 5s -- T2
update status='CONFIRMED' where id=1; -- T1
update status='PAID' where id=1; -- T2
commit; -- T1
commit; -- T2 ERROR!
Prepare to run the first command:
curl http://localhost:8090/api/order/1?status=CONFIRMED\&delay=5000 -i -X PUT
Open another session, and prepare to run a similar command in less than 5sec after the first one:
curl http://localhost:8090/api/order/1?status=PAID\&delay=5000 -i -X PUT
When both commands are executed serially, it will cause a serialization conflict like this:
ERROR: restart transaction: TransactionRetryWithProtoRefreshError: WriteTooOldError: write for key /Table/109/1/12/0 at timestamp 1669990868.355588000,0 too old; wrote at 1669990868.778375000,3: "sql txn" meta={id=92409d02 key=/Table/109/1/12/0 pri=0.03022202 epo=0 ts=1669990868.778375000,3 min=1669990868.355588000,0 seq=0} lock=true stat=PENDING rts=1669990868.355588000,0 wto=false gul=1669990868.855588000,0
The interceptor will however catch this error since it has a state code 40001, retry the business method and eventually succeed and deliver a 200 OK to the client.
Conclusion
In this article, we implemented a transaction retry strategy for JavaEE stateless session beans using bean-managed transactions and a custom interceptor with interceptor bindings. This reduces the amount of retry logic in ordinary service beans to simply add a @TransactionBoundary meta-annotation.
