Overview

Spring and Spring Boot provides a rich ecosystem for building modern applications and services. It's a powerful platform that provides easy-to-use infrastructure abstractions and seamless integration with most databases, messaging systems, cloud infrastructure and whatnot else.

In this tutorial, we will explore how annotations and AOP aspects can be used to leverage certain features in CockroachDB, such as follower reads and time travel queries. The same mechanism will also be used to implement a transparent transaction retry strategy.

The goal of this approach is to avoid obscuring business logic with data access logic and error handling, which can become quite verbose otherwise.

Following the AOP concept, these cross-cutting concerns can be concentrated in one single place and then weaved in where needed by annotations and pointcut expressions (see diagram below). Aspects in Spring are very useful for weaving in some specific advice, either around, before or after method calls.

Screenshot 2021-03-23 at 20.48.13.png

Spring Annotations

Let's take a look at how this can be applied to the following use cases:

Time travel queries - used for reading from a timestamp in the past.
Follower reads - used for reading from a follower replica, slightly in the past (~5s).
Adopting a design pattern for transactional robustness and clarity.
Transaction retries with exponential backoff.
Transaction priorities and other attributes.

Time Travel

To learn about time travel queries in CockroachDB, see as of system time. To use it in our Spring application, let's first create a method-level annotation for declaring the intention.

@Inherited
@Documented
@Target(ElementType.METHOD)
@Retention(RUNTIME)
public @interface TimeTravel {
    String interval() default "-30s";
}

Then we create a before advice that sets the time interval to read from the past (can also be an "around" advice).

@Aspect
@Order(AdvisorOrder.WITHIN_CONTEXT)
public class TimeTravelAspect {
    @Autowired
    private JdbcTemplate jdbcTemplate;

    @Before(value = "@annotation(timeTravel)", argNames = "timeTravel")
    public void beforeTimeTravelOperation(TimeTravel timeTravel) {
        Assert.isTrue(TransactionSynchronizationManager.isActualTransactionActive(),
                "TX not active - explicit transaction required");
        jdbcTemplate.update("SET TRANSACTION AS OF SYSTEM TIME INTERVAL '" + timeTravel.interval() + "'");
    }
}

The @Order annotation is used to control in what order the AOP proxies are invoked, which is important when multiple advices are used at the same time. Typically, we want the retry proxy to be called first, acting as the outer boundary, then the transaction proxy and finally the transaction attributes (session vars) proxy. Setting transaction attributes requires explicit transactions to work (disable auto-commit) so the inverse order wouldn't work.

Now, let's see how this is used in a web (API) controller:

    @GetMapping(value = "/{id}")
    @TransactionBoundary
    @TimeTravel
    public HttpEntity<AccountModel> getAccount(@PathVariable("id") Long accountId) {
        return new ResponseEntity<>(accountResourceAssembler
                .toModel(accountRepository.getOne(accountId)), HttpStatus.OK);
    }

All you do is add the @TimeTravel annotation, and that's it. After a transaction is created but before the service method gets called, the advice kicks in and sets the transaction attribute.

Follower Reads

A follower read pretty much follows the same pattern and technically, it is a time travel query with either time-bounded or exact staleness. To learn more about follower reads in CockroachDB, see here.

First, create the marker annotation:

@Inherited
@Documented
@Target(ElementType.METHOD)
@Retention(RUNTIME)
public @interface FollowerRead {
      String staleness() default "(exact)";
}

Then create a before advice that sets the follower read expression.

@Aspect
@Order(AdvisorOrder.WITHIN_CONTEXT)
public class FollowerReadAspect {
    @Autowired
    private JdbcTemplate jdbcTemplate;

   @Before(value = "@annotation(followerRead)", argNames = "followerRead")
   public void beforeFollowerReadOperation(FollowerRead followerRead) {
        Assert.isTrue(TransactionSynchronizationManager.isActualTransactionActive(),
                "TX not active - explicit transaction required");
        if ("(exact)".equals(followerRead.staleness())) {
            jdbcTemplate.execute(
                    "SET TRANSACTION AS OF SYSTEM TIME follower_read_timestamp()");
        } else {
            jdbcTemplate.execute(
                    "SET TRANSACTION AS OF SYSTEM TIME with_max_staleness('" + followerRead.staleness() + "')");
        }
    }
}

Transaction Retries

Now let's look at something a bit more interesting - transaction retries. Any database that runs in serializable isolation is subject to serialization errors in contending workloads.

To handle this gracefully when using explicit transactions (begin+commit), it's recommended to catch them server side and retry the transactions after an exponential backoff period. In Spring, there's an entire exception class hierarchy dedicated to classifying transient exceptions.

There is one classic stability pattern called Entity-Control-Boundary (ECB) that fits quite well for this purpose. This pattern brings clarity and robustness to transaction management in a typical Spring Boot application.

It works by using meta-annotations to assign ECB architectural roles to different elements, and then use aspects to apply transaction behaviour for these roles.

In a nutshell:

Use meta-annotations to drive transaction demarcation and propagation
Enable Spring's annotation-driven transaction manager
Use aspects to implement transparent transaction retries on transient errors

This annotation is optional and you can achieve the same thing by just using Springs @Transactional annotation, but it doesn't help to enforce architectural roles. Boundaries have a tendency to get blurry in organically growing code bases.

Elements

Boundary - Typically a web controller, service facade or JMS/Kafka service activator method that exposes the functionality of a service and interacts with clients.
Control - Typically a fine-grained service behind a boundary web that implements business logic.
Entity - Refers to a persistent domain object, typically mapped to a JPA entity.

Annotations

These are the (meta-)annotations applied to the above elements.

@TransactionBoundary - Annotation marking a transaction boundary. A boundary is allowed to start new transactions and suspend existing ones, hence it uses REQUIRES_NEW propagation.

@Inherited @Documented @Retention(RetentionPolicy.RUNTIME) @Target({ElementType.TYPE, ElementType.METHOD})

@Transactional(propagation = Propagation.REQUIRES_NEW)

public @interface TransactionBoundary { ... }
@TransactionService - Annotation marking a control service method that is NOT allowed to start new transactions and must be invoked from a transactional context. Hence it uses MANDATORY propagation.

@Inherited @Documented @Retention(RetentionPolicy.RUNTIME) @Target({ElementType.TYPE, ElementType.METHOD})

@Transactional(propagation = Propagation.MANDATORY)

public @interface TransactionService { }
@javax.persistence.Entity - Standard JPA entity annotation marking a managed persistent entity. Not strictly needed.

Aspects

So far we have just defined the semantic descriptors and transaction demarcations through annotations. Let's move on to the actual retry aspects. There are two flavours of retries, one executing the entire atomic/logical transaction again and the second rolling back to a savepoint and retrying from there.

@Aspect
@Order(AdvisorOrder.HIGHEST) // This advisor must be before the TX advisor in the call chain
public class RetryableAspect {
    protected final Logger logger = LoggerFactory.getLogger(getClass());

    @Around(value = "Pointcuts.anyTransactionBoundaryOperation(transactionBoundary)",
            argNames = "pjp,transactionBoundary")
    public Object aroundTransactionalMethod(ProceedingJoinPoint pjp, TransactionBoundary transactionBoundary)
            throws Throwable {
        // Grab from type if needed (for non-annotated methods)
        if (transactionBoundary == null) {
            transactionBoundary = AopSupport.findAnnotation(pjp, TransactionBoundary.class);
        }

        int numCalls = 0;

        final Instant callTime = Instant.now();

        do {
            try {
                numCalls++;
                Object rv = pjp.proceed();
                if (numCalls > 1) {
                    logger.info(
                            "Transient error recovered after " + numCalls + " of " + transactionBoundary
                                    .retryAttempts() + " retries ("
                                    + Duration.between(callTime, Instant.now()).toString() + ")");
                }
                return rv;
            } catch (TransientDataAccessException | TransactionSystemException ex) { // TX abort on commit's
                Throwable cause = NestedExceptionUtils.getMostSpecificCause(ex);
                if (cause instanceof SQLException) {
                    SQLException sqlException = (SQLException) cause;
                    if ("40001".equals(sqlException.getSQLState())) { // Transient error code
                        handleTransientException(sqlException, numCalls, pjp.getSignature().toShortString(),
                                transactionBoundary.maxBackoff());
                        continue;
                    }
                }

                throw ex;
            } catch (UndeclaredThrowableException ex) {
                Throwable t = ex.getUndeclaredThrowable();
                while (t instanceof UndeclaredThrowableException) {
                    t = ((UndeclaredThrowableException) t).getUndeclaredThrowable();
                }

                Throwable cause = NestedExceptionUtils.getMostSpecificCause(ex);
                if (cause instanceof SQLException) {
                    SQLException sqlException = (SQLException) cause;
                    if ("40001".equals(sqlException.getSQLState())) { // Transient error code
                        handleTransientException(sqlException, numCalls, pjp.getSignature().toShortString(),
                                transactionBoundary.maxBackoff());
                        continue;
                    }
                }
                throw ex;
            }
        } while (numCalls < transactionBoundary.retryAttempts());

        throw new ConcurrencyFailureException("Too many transient errors (" + numCalls + ") for method ["
                + pjp.getSignature().toShortString() + "]. Giving up!");
    }

    private void handleTransientException(SQLException ex, int numCalls, String method, long maxBackoff) {
        try {
            long backoffMillis = Math.min((long) (Math.pow(2, numCalls) + Math.random() * 1000), maxBackoff);
            if (numCalls <= 1 && logger.isWarnEnabled()) {
                logger.warn("Transient error (backoff {}ms) in call {} to '{}': {}",
                        backoffMillis, numCalls, method, ex.getMessage());
            }
            Thread.sleep(backoffMillis);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
    }
}

RetryableAspect - Catch transient errors and retry with exponential backoff
RetryableSavepointAspect - Catch transient errors and retry by rolling back to savepoint with exponential backoff

Notice that savepoints are not supported by JPA/Hibernate but work with JDBC.

The order in which these advices are weaved in between the source and target have significance. The ordering must be relative to Spring's transactional advice activated with @EnableTransactionManagement.

The typical call chain needs to be something like this:

   source (controller/service facade/service activator)
        |--> retryableAdvice (no transaction context allowed)  
        |--> transactionAdvice (Spring advice that starts a transaction)  
        |--> transactionHintsAdvice (only within a transaction context)  
    target (business service/repository expecting a transaction)

Usage Example

Using transparent retries in a web controller, acting as the transaction boundary:

    @PostMapping(value = "/transfer")
    @TransactionBoundary(retryAttempts = 20, maxBackoff = 45000)
    public HttpEntity<Void> transfer(@RequestBody TransferRequest request) {
        BigDecimal totalBalance = accountRepository.getBalance(request.getName());
        accountRepository.updateBalance(request.getName(), request.getAccountType(), request.getAmount());
        return new ResponseEntity<>(HttpStatus.OK);
    }

This particular use case, reading and writing to the same account concurrently is subject to the write skew anomaly that serializable isolation protects against. CockroachDB only runs in serializable. The demo shows how that manifests in transient retry errors.

The repository called using the @TransactionService marker annotation just to denote its role:

@Repository
@TransactionService
public interface AccountRepository extends JpaRepository<Account, Long> {
    @Modifying
    @Query("update Account a set a.balance = a.balance + ?3 where a.name = ?1 and a.type=?2")
    void updateBalance(String name, AccountType type, BigDecimal balance);
}

Conclusions

In this tutorial, we explore how annotations and AOP aspects can be used in Spring Boot to leverage certain features in CockroachDB, such as follower reads and time travel queries. We also implement a transparent transaction retry strategy using the same mechanism. This approach helps to avoid obscuring business logic with data access logic and error handling. We cover the use of time travel, follower reads, transaction retries, and the Entity-Control-Boundary (ECB) pattern for transactional robustness and clarity.

The annotation and aspect examples are available on GitHub here with a runnable demo.

Using Spring Annotations and AOP with CockroachDB

Comments (1)

More from this blog

Handling Null Values Efficiently with Spring Named Parameter Binding

Multi-active systems

User defined composite types

Transaction timeouts in CockroachDB

One-Phase Commit Transaction Strategy

Overview

Spring Annotations

Time Travel

Follower Reads

Transaction Retries

Elements

Annotations

Aspects

Usage Example

Conclusions

Command Palette

Comments (1)

More from this blog

Overview

Spring Annotations

Time Travel

Follower Reads

Transaction Retries

Elements

Annotations

Aspects

Usage Example

Conclusions