Using Spring’s NamedParameterJdbcTemplate and binding null values with a SQL type code and type name may lead to the pgJDBC driver performing costly metadata queries to infer internal types. This is by JDBC specification and pgJDBC design, but it’s good to know for performance reasons as it may go unnoticed. This post provides a few tips on how to avoid such queries entirely.

Problem

Any application or framework that use PreparedStatement.setNull(int,int,String) is candidate to these meta queries. The javadoc states:

void setNull(int parameterIndex, int sqlType, String typeName) throws SQLException

Sets the designated parameter to SQL NULL. This version of the method setNull should be used for user-defined types and REF type parameters. Examples of user-defined types include: STRUCT, DISTINCT, JAVA_OBJECT, and named array types.

Note: To be portable, applications must give the SQL type code and the fully-qualified SQL type name when specifying a NULL user-defined or REF parameter. In the case of a user-defined type the name is the type name of the parameter itself. For a REF parameter, the name is the type name of the referenced type. If a JDBC driver does not need the type code or type name information, it may ignore it. Although it is intended for user-defined and Ref parameters, this method may be used to set a null parameter of any JDBC type. If the parameter does not have a user-defined or REF type, the given typeName is ignored.

The SHOULD phrase means you can use it for any column type, including primitives and JSONB for example. Typically, the pgJDBC implementation lways attempts to resolve the internal type when both the sqlType and typeName is specified. In case of ARRAY, it also attempts to resolve the array element type, which is another even more involved meta data query. This repeats for every parameter bind operation, so it can really amplify write latency in the worst case.

A metadata query can look like:

SELECT pg_type.oid, typname
 FROM pg_catalog.pg_type LEFT
   JOIN (SELECT ns.oid AS nspoid, ns.nspname, r.r
 FROM pg_namespace AS ns
   JOIN (SELECT s.r, (current_schemas(_))[s.r] AS nspname
 FROM ROWS
 FROM (generate_series(_, array_upper(current_schemas(_), _))) AS s (r)) AS r USING (nspname)) AS sp
    ON sp.nspoid = typnamespace
   WHERE typname = _
 ORDER BY sp.r, pg_type.oid DESC
 LIMIT _

To demonstrate this, the following code sample passes null for the “price” and “description” columns. If null values are passed along with a column type constant and a type name, then the above query is sent by the driver to resolve the internal PG type.

MapSqlParameterSource namedParameters = new MapSqlParameterSource()
        .addValue("id", UUID.randomUUID())
        .addValue("inventory", 99)
        .addValue("name", "product-x99")
        .addValue("sku", "X-99")
        .addValue("price", null, Types.DECIMAL, "decimal")
        .addValue("description", null, Types.OTHER, "jsonb");

new NamedParameterJdbcTemplate(jdbcTemplate)
   .update("insert into product (id, inventory, name, description, price, sku) " +
           "values (:id,:inventory,:name,:description,:price,:sku)", namedParameters);

Solution

• Prefer indexed placeholders (?,?,..) over named parameter placeholders, thus avoiding the NamedParameterJdbcTemplate problem entirely

• Omit specifying the column type name. For example:

  • addValue("price", null)

  • addValue("description", null)

• If still binding a NULL value with a type name, use another type constant like Types.NULL:

  • addValue("price", null, Types.NULL, "decimal")

  • addValue("description", null, Types.NULL, "jsonb")

• Set the system property -Dspring.jdbc.getParameterType.ignore=true (source) to avoid similar situations with ARRAYs

How to verify

Set the pgJDBC and spring jdbc core namespace logger levels to TRACE.

<logger name="org.postgresql" level="TRACE"/>
<logger name="org.springframework.jdbc.core" level="TRACE"/>

The log output will be verbose but watch out for all pg_catalog queries before your actual INSERT or UPDATE statements:

2025-11-26 12:33:06.157 TRACE [o.postgresql.core.v3.QueryExecutorImpl]   simple execute, handler=org.postgresql.jdbc.PgStatement$StatementResultHandler@5c134052, maxRows=0, fetchSize=0, flags=17
2025-11-26 12:33:06.157 TRACE [o.postgresql.core.v3.QueryExecutorImpl]  FE=> Parse(stmt=null,query="SELECT pg_type.oid, typname   FROM pg_catalog.pg_type   LEFT   JOIN (select ns.oid as nspoid, ns.nspname, r.r           from pg_namespace as ns           join ( select s.r, (current_schemas(false))[s.r] as nspname                    from generate_series(1, array_upper(current_schemas(false), 1)) as s(r) ) as r          using ( nspname )        ) as sp     ON sp.nspoid = typnamespace  WHERE typname = $1  ORDER BY sp.r, pg_type.oid DESC LIMIT 1",oids={1043})
2025-11-26 12:33:06.157 TRACE [o.postgresql.core.v3.QueryExecutorImpl]  FE=> Bind(stmt=null,portal=null,$1=<('jsonb')>,type=VARCHAR)
2025-11-26 12:33:06.157 TRACE [o.postgresql.core.v3.QueryExecutorImpl]  FE=> Describe(portal=null)
2025-11-26 12:33:06.157 TRACE [o.postgresql.core.v3.QueryExecutorImpl]  FE=> Execute(portal=null,limit=0)
2025-11-26 12:33:06.157 TRACE [o.postgresql.core.v3.QueryExecutorImpl]  FE=> Sync

Conclusion

Efficiently handling null values in Spring's NamedParameterJdbcTemplate is essential for optimizing database interactions and minimizing unnecessary overhead. This involves preferring indexed placeholders, omitting column type names, and configuring a system property to avoid costly metadata queries.

• Securing business continuity in the event of regional data centre disruptions

• Deliver a good customer experience worldwide

• Deliver business adaptability and scalability to different market needs/volumes

• Compliance with data locality/placement regulations

• Sustainable operational costs as the business grows

Multi-active systems are a prerequisite to effectively adopting a multi-region strategy. Let's find out how.

Multi-Active Systems

A multi-active system is capable to operate and serve online traffic simultaneously from multiple active data centres and regions. With that comes characteristics that will satisfy the goals stated above without adding too much infrastructure/app complexity and cost overhead.

Multi-active systems run live in multiple datacenters in different regions all the time and workloads are dynamically shared across these datacenters. A multi-active system process requests simultaneously for either domestic or global markets without any assumptions on locality, traffic affinity or replication delays.

There are no actual concepts of traffic failover or failback. Instead, failures and disruptions are handled transparently through regional and/or global load balancing and traffic rerouting. If one failure domain (data centre or region) begins to fall behind due to disruptions, parts of its workload can move to other domains transparently. If one domain is completely offline, all its work is rebalanced to the remaining domains.

Using three failure domains allows one to go completely dark, while still allowing systems to make forward progress through consensus decisions. Using five failure domains allows two domains to go dark, and so on. This allows for systems to be both highly available and always consistent and correct.

A multi-active system is not without certain challenges:

• Needs a fit-for-purpose solution to manage state and replication (data distribution at a global scale is a difficult problem) such as:

  • Protecting rule invariants also during contention and failures

  • Ensure effectively once outcomes of event processing

• Higher service latency due to mandatory cross-datacenter coordination

• Existing design assumptions on single DC/region deployment

• Constraints around auxiliary system integrations

• Breaking current assumptions on system design

To contain complexity, most of these challenges are preferably pushed down to the resource tier - the database - instead of being managed in the app tier. CockroachDB is one such system with a wide range of multi-region deployment options and first-class support for crafting multi-active systems.

Failover-based Systems

To contrast multi-active systems, let's quickly look at the main predecessor: singly-homed, failover-based systems. A singly-homed system is crafted to operate and serve online traffic from a single data centre or at most a single region. Crafted in terms of design choices, technology selections, network assumptions and supporting infrastructure components.

In the event of a primary domain disruption or disaster, a singly-homed system may failover traffic to an alternative secondary data centre. After the disruption is cleared away, it may "fail back" traffic to the original primary domain.

This type of setup has many limitations:

• Unable to scale horizontally beyond a single data centre/region

• Unable to load-balance traffic freely across multiple active datacenters

• Dependent on standby, underutilized resources, increasing TCO (300% capacity for steady state)

• Must use asynchronous replication for availability and performance, with the risk of data loss

• Long recovery times after failures

• Complex and error-prone failover protocols with manual checkpoints/sign-offs

• Unclear when and if a standby system can resume traffic from a safe point

• Difficult and risky to test and verify that the protocol works

Disaster Recovery Spectrum

The main objective of a disaster recovery plan is to minimize the time it takes to recover from a severe disruption event and reduce the amount of data loss and other business impacts.

The spectrum of disaster recovery solutions typically ranges from offline backups to full-blown multi-region deployments, also with backups.

• Backups - Data is frequently backed up and sent off-side or to cloud storage.

  • The recovery time objective (RTO) is governed by the time it takes to restore the database to a new setup

  • The recovery point objective (RPO) is governed by the frequency of incremental and full backups

• Cold Standby - A minimally provisioned environment with the ability to take over core services from a failed primary data centre.

  • Higher TCO due to under-utilized standby capacity

  • RTO is governed by how fast a switchover can be made to the secondary

  • RPO is governed by the async replication delay from the primary to the secondary

• Warm Standby - A fully provisioned environment with the ability to take over a failed primary data centre.

  • Higher TCO due to excessive amounts of under-utilized standby capacity

  • Could serve certain read-only traffic at the same time as the primary

  • RTO and RPO are quite similar to a cold standby, only a bit lower due to higher readiness

• Multi-Active - Each deployment site serves production traffic simultaneously.

  • All data centres provide traffic at the same time for the entire keyspace

  • There is no actual notion of fail-over or fail-back, failures and recovery are handled transparently towards the app tier

  • RTO is governed by how quickly an isolated or crashed node can drop its authority over reads and writes to local data (typically a few seconds)

  • RPO is zero due to consensus-based replication

Multi-active systems stand out from most fail-over-based models in terms of cost and complexity reduction. It's far more resilient against different categories of disruptions, but not immune to disasters. If a multi-active system loses a majority of its failure domains (like 2 zones in a 3-zone region) or if some operator error corrupts a database, then the music stops. Therefore, backups are still commonly used alongside multi-active systems, which adds a safety harness for recovery.

Combined with multiple regions, the blast radius is extended to cover most conditions and you can also improve customer experience for a global market.

Multi-region deployments

One data centre is a single point of failure, similar to a single region. If that data centre/region goes offline for a longer period without any recovery option, it may have a severe impact on the business and the company's reputation.

Adding two or more data centres to a single region will increase the blast radius and decrease the likelihood of severe, long-lasting service disruptions due to a single DC outage.

Deploying a system (as in many services/components working in concert) across multiple, geo-separated regions extends the blast radius even further. Single-region assumptions cannot however be transferred to this new ecosystem due to how we traditionally manage state and consistency. Leveraging multi-region effectively requires a multi-active system architecture. Not exclusively, but it's very much a state/database undertaking that needs a fit-for-purpose solution like CockroachDB.

Summary

This article discusses the advantages of adopting a region-level deployment strategy for businesses, focusing on multi-active systems. These systems operate simultaneously across multiple data centres, providing increased resiliency and adaptability to market needs. The article also contrasts multi-active systems with traditional failover-based systems and examines the disaster recovery spectrum, including backups, cold standby, warm standby, and multi-active solutions. Ultimately, multi-active systems offer significant benefits in terms of cost and complexity reduction, while still requiring backups to ensure data safety.

Introduction

A composite type is simply a type composed of other types. In the following example, we are creating a composite money type. The money type is the combination of an amount, currency code and monetary type:

• The amount is a decimal with fractions matching the currency.

• The currency is a 3-letter ISO 4217 code.

• The monetary type is an arbitrary tag for denoting the type of money. For example:

  • RM for real money

  • FM for funny money

On top of the type, we'll also add a few UDFs for money arithmetics. Ideally, these functions should only be allowed when operands use the same currency and monetary type. For example, you want to prevent adding 10 USD with 15 SEK or real money with funny money. There's however no way to enforce these rules in the DB itself.

Let's begin with the money type:

CREATE TYPE money_type AS (amount decimal, currency_code char (3), monetary_type char (2));

Next, create a few UDFs for money operations:

CREATE FUNCTION money_amount(x money_type) RETURNS decimal IMMUTABLE LEAKPROOF LANGUAGE SQL AS $$
   select ((x).amount)::decimal
$$;

CREATE FUNCTION money_currency(x money_type) RETURNS char(3) IMMUTABLE LEAKPROOF LANGUAGE SQL AS $$
   select ((x).currency_code)::char(3)
$$;

CREATE FUNCTION money_monetary_type(x money_type) RETURNS char(3) IMMUTABLE LEAKPROOF LANGUAGE SQL AS $$
   select ((x).monetary_type)::char(3)
$$;

CREATE FUNCTION to_money(x string) RETURNS money_type IMMUTABLE LEAKPROOF LANGUAGE SQL AS $$
   select (
    split_part($1,' ',1)::decimal,
    split_part($1,' ',2),
    split_part($1,' ',3)
    )::money_type
$$;

Next, let's add a few money arithmetics UDFs:

CREATE FUNCTION money_add(IN m money_type, IN addend decimal) RETURNS money_type IMMUTABLE LEAKPROOF LANGUAGE SQL AS $$
select (
    (m).amount + addend,
    (m).currency_code,
    (m).monetary_type
    )
$$;

CREATE FUNCTION money_mult(IN m money_type, IN multiplier decimal) RETURNS money_type IMMUTABLE LEAKPROOF LANGUAGE SQL AS $$
select (
    (m).amount * multiplier,
    (m).currency_code,
    (m).monetary_type
    )
$$;

CREATE FUNCTION money_div(IN m money_type, IN dividend decimal) RETURNS money_type IMMUTABLE LEAKPROOF LANGUAGE SQL AS $$
select (
    (m).amount / dividend,
    (m).currency_code,
    (m).monetary_type
    )
$$;

That's about it. Now let's put the money type to use and see how things work. Create an account table holding a cached balance using the money type:

create table account
(
    id             uuid        not null default gen_random_uuid(),
    city           string      not null,
    balance        money_type  not null,
    name           string(128) not null,
    description    string(256) null,
    closed         boolean     not null default false,
    allow_negative integer     not null default 0,
    updated_at     timestamptz not null default clock_timestamp(),

    primary key (id)
);

Let's strengthen the integrity a bit with these CHECK constraints (totally optional):

alter table account
    add constraint check_account_allow_negative check (allow_negative between 0 and 1);
alter table account
    add constraint check_account_positive_balance check ((balance).amount * abs(allow_negative - 1) >= 0);
alter table account
    add constraint check_account_currency check ((balance).currency_code in ('SEK', 'USD', 'GBP', 'EUR'));

As you can see, different accounts may or may not accept a negative balance depending on the current flag. We could also have used an enum type for the currency.

Add some data:

INSERT INTO account (id, city, balance, name, allow_negative)
VALUES ('10000000-0000-0000-0000-000000000000', 'stockholm', to_money('100.00 SEK RM'), 'test:1', 0),
       ('20000000-0000-0000-0000-000000000000', 'stockholm', to_money('200.00 SEK RM'), 'test:2', 1),
       ('30000000-0000-0000-0000-000000000000', 'new york', to_money('300.00 USD PM'), 'test:3', 0),
       ('40000000-0000-0000-0000-000000000000', 'new york', to_money('400.00 USD PM'), 'test:4', 1);

Let's see how this looks:

select id,city,balance,
       money_amount(balance),
       money_currency(balance),
       money_monetary_type(balance)
from account;
                   id                  |   city    |     balance     | money_amount | money_currency | money_monetary_type
---------------------------------------+-----------+-----------------+--------------+----------------+----------------------
  10000000-0000-0000-0000-000000000000 | stockholm | (100.00,SEK,RM) |       100.00 | SEK            | RM
  20000000-0000-0000-0000-000000000000 | stockholm | (200.00,SEK,RM) |       200.00 | SEK            | RM
  30000000-0000-0000-0000-000000000000 | new york  | (300.00,USD,PM) |       300.00 | USD            | PM
  40000000-0000-0000-0000-000000000000 | new york  | (400.00,USD,PM) |       400.00 | USD            | PM
(4 rows)

Time: 14ms total (execution 14ms / network 0ms)

Let's execute an aggregation query:

select sum(money_amount(balance)) balance, 
       money_currency(balance) currency 
from account group by balance,currency;

  balance | currency
----------+-----------
   100.00 | SEK
   200.00 | SEK
   300.00 | USD
   400.00 | USD
(4 rows)

Time: 2ms total (execution 2ms / network 0ms)

Updating the money type can be done using one of the arithmetic functions:

UPDATE account set balance=money_add(balance,-90.00) where id='10000000-0000-0000-0000-000000000000';
UPDATE account set balance=money_add(balance,-100.00) where id='20000000-0000-0000-0000-000000000000';

Conclusion

This article provides an example of how to create a user-defined composite type in CockroachDB v23.1, specifically a money type composed of an amount, currency code, and monetary type. It also explains how to use UDFs for money operations, create a table to hold a cached balance, and add CHECK constraints to strengthen the integrity.

This has changed since CockroachDB v23.1 where a new session variable for transaction timeouts was introduced, unsurprisingly called transaction_timeout:

New in v23.1: Aborts an explicit transaction when it runs longer than the configured duration. Stored in milliseconds; can be expressed in milliseconds or as an INTERVAL.

Overview

Transaction timeouts are helpful if you need to set a fixed upper limit for how long to wait for an explicit transaction to complete. If a transaction is not completed within that timeframe it's aborted and then you that any provisional writes did not complete.

In contrast, if you just wait for an arbitrary amount of time and then interrupt the calling thread, then you have an ambiguous result where you can't tell if an operation took place or not since the commit could have been completed or rolled back just before the cancellation. Ambiguous results for non-idempotent operations are typically not a good thing for safety.

Now let's see how to hook up transaction timeouts in a fully transparent way using Spring's @Transactional annotation and AspectJ. Similar to how we can deal with transaction retries.

Source Code

The code for this article is available on GitHub.

AOP Timeout Solution

We are going to set the attributes using AOP and AspectJ, which is a core concept in Spring Boot.

A small recap on basic AOP terminology:

• Aspect - An orthogonal cross-cutting concern that you wrap in a contained module or aspect. Like retries, logging, security or in our case setting session variables.

• Joinpoint - Points in the application code where to plugin the aspect, such as method execution or the handling of an exception.

• Pointcut - One or more join points where advice should be executed, often using pointcut expressions.

• Advice - The action to be performed either before or after method execution, akin to an interceptor.

To set setting attributes, we create a TransactionAttributesAspect with an around-advice:

import org.aspectj.lang.ProceedingJoinPoint;
import org.aspectj.lang.annotation.Around;
import org.aspectj.lang.annotation.Aspect;
import org.aspectj.lang.annotation.Pointcut;
import org.springframework.core.Ordered;
import org.springframework.core.annotation.Order;
import org.springframework.transaction.TransactionDefinition;
import org.springframework.transaction.annotation.Transactional;

@Aspect
@Order(Ordered.LOWEST_PRECEDENCE - 2)
public class TransactionAttributesAspect {
    @Autowired
    private JdbcTemplate jdbcTemplate;

    @Pointcut("execution(public * *(..)) "
            + "&& @annotation(transactional)")
    public void anyTransactionalOperation(Transactional transactional) {
    }

    @Around(value = "anyTransactionalOperation(transactional)", argNames = "pjp,transactional")
    public Object doAroundTransactionalMethod(ProceedingJoinPoint pjp, Transactional transactional) throws Throwable {
        Assert.isTrue(TransactionSynchronizationManager.isActualTransactionActive(), "Explicit transaction required");
        applyVariables(transactional);
        return pjp.proceed();
    }

    private void applyVariables(Transactional transactional) {
        if (transactional.timeout() != TransactionDefinition.TIMEOUT_DEFAULT) {
            jdbcTemplate.update("SET transaction_timeout=?", transactional.timeout() * 1000);
        }

        if (transactional.readOnly()) {
            jdbcTemplate.execute("SET transaction_read_only=true");
        }
    }
}

This weaves in the doAroundTransactionalMethod advice at runtime on all public methods annotated with Spring's @Transactional annotation. This is pretty much what the pointcut expression says:

@Pointcut("execution(public * *(..)) && @annotation(transactional))

Lastly, we look at the annotation properties and use a JDBC template to set the appropriate variables while assuming there's an open transaction in scope.

if (transactional.timeout() != TransactionDefinition.TIMEOUT_DEFAULT) {
   jdbcTemplate.update("SET transaction_timeout=?", transactional.timeout() * 1000);
}

Testing Timeouts

To test this in action, let's create a simple service and a few repositories:

@Service
public class OrderService {
    @Autowired
    private OrderRepository orderRepository;

    @Autowired
    private ProductRepository productRepository;

    @Transactional(propagation = Propagation.REQUIRES_NEW, readOnly = true)
    public Product findProduct(String sku) {
        return productRepository.findBySku(sku)
                .orElseThrow(() -> new ObjectRetrievalFailureException(Product.class, sku));
    }

    @Transactional(propagation = Propagation.REQUIRES_NEW, timeout = 5)
    public void placeOrderWithTimeout(Order order, long delayMillis) {
        placeOrderAndUpdateInventory(order);

        try {
            logger.info("Entering sleep for " + delayMillis);
            Thread.sleep(delayMillis);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        } finally {
            logger.info("Exited sleep for " + delayMillis);
        }
    }

    @Transactional(propagation = Propagation.REQUIRES_NEW)
    public void placeOrderWithoutTimeout(Order order) {
        placeOrderAndUpdateInventory(order);
    }

    private void placeOrderAndUpdateInventory(Order order) {
        Assert.isTrue(!TransactionSynchronizationManager.isCurrentTransactionReadOnly(), "Read-only");
        Assert.isTrue(TransactionSynchronizationManager.isActualTransactionActive(), "No tx");

        // Update product inventories
        order.getOrderItems().forEach(orderItem -> {
            Product product = orderItem.getProduct();
            product.addInventoryQuantity(-orderItem.getQuantity());
            productRepository.save(product); // product is in detached state
        });
        order.setStatus(ShipmentStatus.confirmed);

        orderRepository.save(order);
    }
}

In the placeOrderWithTimeout method, there's a fake delay that can last longer than the configured timeout to trigger an abort. Let's verify this in an integration test:

public class TimeoutsTest extends AbstractIntegrationTest {
    @Autowired
    private OrderService orderService;

    @Autowired
    private TestSetup testSetup;

    @BeforeAll
    public void setupTest() {
        testSetup.setupTestData();
    }

    @org.junit.jupiter.api.Order(1)
    @Test
    public void whenCreatingOrderWithTimeoutThatExpires_thenExpectRollback() {
        Product p1 = orderService.findProduct("p1");
        int inventory = p1.getInventory();

        JpaSystemException ex = Assertions.assertThrows(JpaSystemException.class, () -> {
            orderService.placeOrderWithTimeout(Order.builder()
                            .andOrderItem()
                            .withProduct(p1)
                            .withQuantity(1)
                            .withUnitPrice(p1.getPrice())
                            .then()
                            .build(),
                    7000);
        });

        Assertions.assertEquals("transaction timeout expired", ex.getMessage());
        Assertions.assertEquals(inventory, orderService.findProduct("p1").getInventory());

        logger.info("Exception thrown", ex);
    }

    @org.junit.jupiter.api.Order(2)
    @Test
    public void whenCreatingOrderWithTimeout_thenExpectCommit() {
        Product p1 = orderService.findProduct("p1");
        int inventory = p1.getInventory();

        orderService.placeOrderWithTimeout(Order.builder()
                        .andOrderItem()
                        .withProduct(p1)
                        .withQuantity(1)
                        .withUnitPrice(p1.getPrice())
                        .then()
                        .build(),
                2000);

        Assertions.assertEquals(inventory - 1, orderService.findProduct("p1").getInventory());
    }

    @org.junit.jupiter.api.Order(3)
    @Test
    public void whenCreatingOrderWithoutTimeout_thenExpectCommit() {
        Product p1 = orderService.findProduct("p1");
        int inventory = p1.getInventory();

        orderService.placeOrderWithoutTimeout(Order.builder()
                .andOrderItem()
                .withProduct(p1)
                .withQuantity(1)
                .withUnitPrice(p1.getPrice())
                .then()
                .build());

        Assertions.assertEquals(inventory - 1, orderService.findProduct("p1").getInventory());
    }
}

In this example if the transaction time out, it throws JpaSystemException.

Conclusion

This article explains how to use the new transaction_timeout session variable in CockroachDB v23.1 to set a fixed upper limit for how long to wait for an explicit transaction to complete. It also provides an example of a service and repositories to test the timeout in an integration test, which verifies that when the timeout expires, the transaction is rolled back and the inventory remains unchanged.

A commonly adopted transaction strategy can be described as the best-efforts one-phase-commit (1PC) pattern. It's different from a global XA/2PC protocol where an external transaction manager ensures that all transaction properties are maintained across the involved transactional resources (database, queue, etc).

The basic idea behind 1PC is to delay the commit in a transaction as late as possible so that the only things that can go wrong are infrastructure failures (because they are rare). All business processing failures are caught before it happens.

It is a relaxation of ACID properties spanning multiple transactional resources. That also means there's a certain risk for system inconsistency in a worst-case scenario, which can be mitigated if the processing is idempotent. For this strategy to work as safely as possible, idempotency is key.

This concept is also described in full detail in Dr David Syer's article Distributed transactions in Spring, with and without XA from 2009.

Scenarios

To illustrate 1PC let's review a couple of examples.

Database and Message Broker

Consider a typical service activator scenario where there's a database and a JMS message queue involved. It includes a write to the database and an acknowledgement to the broker of receiving a message. Both these operations are independent, as in there's no atomicity.

This scenario also maps to Kafka which uses consumer offsets and message retention rather than ephemeral message acks (removed once ack:ed for queue).

1. Start messaging transaction (broker delivers a message)

2. Receive message

3. Start database transaction

4. Update database

5. Commit database transaction

6. Commit messaging transaction (ack is sent to broker upon which the message is removed from destination)

The order of the first four steps is not important. What is important is that the message must be received before updating the database and each transaction must start before its corresponding resource is used.

Dual-write problem (below):

@Component
public class RegistrationConsumer {
    @JmsListener(destination = "${active-mq.topic}", containerFactory = "jmsListenerContainerFactory")
    @Transactional(propagation = Propagation.REQUIRES_NEW)
    public void receiveMessage(RegistrationEvent event) {
        registrationRepository.save(toEntity(event));
    }
}

The following order is therefore just as valid:

1. Start messaging transaction

2. Start database transaction

3. Receive message

4. Update database

5. Commit database transaction

6. Commit messaging transaction

The last two steps (5 and 6) are important to be both in order and come last. It's better to surface business processing violations (bad input, rule violations, constraint violations etc) before sending things to be made permanent. When flushing database operations before acknowledging the message, there's less chance of both systems going out of sync.

An out-of-sync condition could be that the database transaction commits, but the message broker ack fails. Or the other way around, the commit fails and the ack succeeds. In either case, it will result in double-processing the same event and if the database writes are nonidempotent you end up with multiple side effects.

In the case of object-relational-mappers (ORM), most database actions take place during the commit phase due to the first-level cache. It's at that point where the JPA provider (Hibernate) performs update optimizations (collapsing) and determines what SQL statements to send to the database. It's also the phase where the database may raise data model constraint violations to preserve integrity.

Things that can go wrong in the messaging transaction are network and process failures with the broker, which are less likely to occur.

Database and Remote API Call

Consider a typical service boundary scenario where there's a database and a foreign API service involved. It includes a write to the database and an API call to the remote endpoint, which in turn creates some side-effect. Both these operations are independent, as in there's no atomicity.

1. Start database transaction

2. Update database

3. Send a POST request to the remote API

4. Commit database transaction

First off, it's not advisable to invoke remote calls from a transaction context. Therefore, the minimum ask would be to order the steps accordingly:

1. Send a POST request to the remote API

2. Start database transaction

3. Update database

4. Commit database transaction

Better, but still there's a potential issue here if the database transaction fails. In that case, when you retry the entire operation there will be another POST request sent to the endpoint. If that endpoint is nonidempotent you may end up with multiple side effects. Essentially this is the same problem as in the first example, called non-atomic dual-writes.

@Service
public class TransferService {
    @Transactional(propagation = Propagation.REQUIRES_NEW)
    public void createTransfer(TransferEntity entity) {
        ResponseEntity<String> response
                = new RestTemplate().postForEntity("https://api.bank.com",
toRquestPayload(entity), String.class);
        if (!response.getStatusCode().is2xxSuccessful()) {
            throw new IllegalStateException("Disturbance!");
        }
    }
}

However, in this scenario, if the endpoint is idempotent (invoking many times is the same as invoking once) then it doesn't matter how many times you retry it. It will only have one single side effect.

CockroachDB and XA

Currently CockroachDB doesn't support the XA protocol but there's a tracking issue for it. The good thing is that XA-distributed transactions are not strictly needed to support the above scenarios. There are plenty of alternative options, such as:

• Saga pattern:

  • A decomposed version of 2PC where involved services implement participation and compensation methods as part of an agreement protocol. Either using an orchestrated or choreographed approach.

  • The practical use is between disparate services (not between databases and/or brokers)

  • Quite complicated to implement and test and reduces understandability

• Outbox pattern:

  • Domain events are written to the database as part of the local transaction.

  • Domain events are published downstream after the commit point using CDC

  • Avoids the non-atomic dual write problem.

  • The practical use is between disparate services

• Inbox pattern:

  • Incoming messages are stored in the database and then CDC is used to publish or self-subscribe to the messages

  • Offloads the message broker and adds retention

  • The practical use is between disparate services

• 1PC with idempotency

  • As described in this article

Conclusion

This article outlined a commonly adopted transaction strategy described as the best-efforts one-phase-commit (1PC) pattern. It offers a simple, low-effort alternative to XA pre-conditioned that operations are idempotent.

Another great resource to illustrate the behaviour of serializable is Martin Kleppman's Hermitage project and the CockroachDB contribution. It goes through a rich set of anomalies ranging from dirty writes to write skew on disjoint and predicate reads.

Examples

These tests were executed using the following CockroachDB version:

$ cockroach version
Build Tag:        v22.2.8
Build Time:       2023/04/17 13:22:08
Distribution:     CCL
Platform:         linux amd64 (x86_64-pc-linux-gnu)
Go Version:       go1.19.6
C Compiler:       gcc 6.5.0
Build Commit ID:  9a7c644e565b21d29db26a0a82524a00809d0a8c
Build Type:       release

First, create a test database:

cockroach sql --insecure --host=localhost -e "CREATE database test"

Next, start three separate shell windows representing transactions T1, T2 and T3. Whenever there's a "-- T1" comment for a SQL statement, run that statement in the designated console session.

cockroach sql --insecure --host=localhost --database test // for T1
cockroach sql --insecure --host=localhost --database test // for T2
cockroach sql --insecure --host=localhost --database test // for T3

Black and White Marbles

This is a test for Write Skew (A5B), prevented by serializable. Write Skew is when two transactions overlap and one reads data that another is writing.

Schema setup:

create table if not exists marbles (
  id    bigint      not null primary key,
  color varchar(25) not null
);

delete from marbles where 1=1;

insert into marbles values
  (1,'black'),
  (2,'black'),
  (3,'black'),
  (4,'black'),
  (5,'black'),
  (6,'white'),
  (7,'white'),
  (8,'white'),
  (9,'white'),
  (10,'white');

Note: The set transaction isolation level serializable part is redundant for CockroachDB since it's the default (and only level supported).

Observe that CockroachDB serializable prevent this anomaly:

begin; set transaction isolation level serializable; -- T1
begin; set transaction isolation level serializable; -- T2
update marbles set color = 'black' where color = 'white'; -- T1
update marbles set color = 'white' where color = 'black'; -- T2
commit; -- T1. First commit wins.
commit; -- T2. ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh due to a conflict: committed value on key /Table/137/1/6/0): "sql txn" meta={id=f8ee6d8c key=/Table/137/1/1/0 pri=0.00066739 epo=0 ts=1682691951.358336984,2 min=1682691934.561490359,0 seq=5} lock=true stat=PENDING rts=1682691934.561490359,0 wto=false gul=1682691935.061490359,0
SQLSTATE: 40001
HINT: See: https://www.cockroachlabs.com/docs/v22.2/transaction-retry-error-reference.html#retry_serializable

All the colours must match:

SELECT * from marbles order by id;
  id | color
-----+--------
   1 | black
   2 | black
   3 | black
   4 | black
   5 | black
   6 | black
   7 | black
   8 | black
   9 | black
  10 | black
(10 rows)

If you would run in SNAPSHOT (which CockroachDB doesn't provide) then it would not prevent write skew and look like this instead:

+----+-------+
| id | color |
+----+-------+
|  1 | white |
|  2 | white |
|  3 | white |
|  4 | white |
|  5 | white |
|  6 | black |
|  7 | black |
|  8 | black |
|  9 | black |
| 10 | black |
+----+-------+
-- (10 rows)

The colours have been flipped due to write skew, which is expected under concurrent execution with snapshot isolation (or read committed).

Red, Green and Blue Marbles

This example is similar to the previous one, only this time involving three transactions.

Setup schema:

create table if not exist marbles (
  id    bigint      not null primary key,
  color varchar(25) not null
);

delete from marbles where 1=1;
insert into marbles (id,color) values (1,'red');
insert into marbles (id,color) values (2,'red');
insert into marbles (id,color) values (3,'red');
insert into marbles (id,color) values (4,'yellow');
insert into marbles (id,color) values (5,'yellow');
insert into marbles (id,color) values (6,'yellow');
insert into marbles (id,color) values (7,'blue');
insert into marbles (id,color) values (8,'blue');
insert into marbles (id,color) values (9,'blue');

Again, CockrochDB SERIALIZABLE isolation prevents Write Skew (A5B):

begin; set transaction isolation level SERIALIZABLE ; -- T1
update marbles set color = 'yellow' where color = 'red'; -- T1

begin; set transaction isolation level SERIALIZABLE ; -- T2
update marbles set color = 'blue' where color = 'yellow'; -- T2. Blocks on T1 intents.

begin; set transaction isolation level SERIALIZABLE ; -- T3
update marbles set color = 'red' where color = 'blue'; -- T3. Blocks.

commit; --T1
(T2 unblocks - rows affected 6)
(T3 unblocks - rows affected 3)
commit; -- T3
commit; -- T2. ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh due to a conflict

The correct outcome (only yellow and red):

select * from marbles;
+----+--------+
| id | color  |
+----+--------+
|  1 | yellow |
|  2 | yellow |
|  3 | yellow |
|  4 | yellow |
|  5 | yellow |
|  6 | yellow |
|  7 | red    |
|  8 | red    |
|  9 | red    |
+----+--------+
(9 rows)

Intersecting Data

Two concurrent transactions read data, and each uses it to update the range read by the other.

Setup schema:

create table if not exists tab (
  id bigint not null,
  value bigint not null
);

delete from tab where 1=1;

INSERT INTO tab VALUES
(1, 10), (1, 20), (2, 100), (2, 200);

Observe CockroachDB guarantees a serial execution:

begin; set transaction isolation level serializable; -- T1
SELECT SUM(value) FROM tab WHERE id = 1; -- T1
INSERT INTO tab VALUES (2, 30); -- T1

begin; set transaction isolation level serializable; -- T2
SELECT SUM(value) FROM tab WHERE id = 2; -- T2 (blocks)

commit; -- T1 (unblocks T2)

INSERT INTO tab VALUES (1, 330); -- T2

commit; -- T2

SELECT * from tab;

Yields:

SELECT * from tab;
  id | value
-----+--------
   1 |    10
   1 |    20
   2 |   100
   2 |   200
   2 |    30
   1 |   330
(6 rows)

Overdraft Protection

Here we will protect the invariant that the total of all accounts must exceed the amount requested.

Schema setup:

create table if not exists account
  (
    name VARCHAR(25) not null,
    type VARCHAR(25) not null,
    balance NUMERIC(19, 2) not null,

    primary key (name, type)
  );

delete from account where 1=1;

insert into account values
  ('alice','saving', 500),
  ('alice','checking', 500);

Let's try to play the bank under serializable isolation:

begin; set transaction isolation level serializable ; -- T1
select type, balance from account where name = 'alice'; -- T1

begin; set transaction isolation level serializable ; -- T2
select type, balance from account where name = 'alice'; -- T2

update account set balance = balance - 900.00 where name = 'alice' and type = 'saving'; -- T1
commit; -- T1

update account set balance = balance - 900.00 where name = 'alice' and type = 'checking'; -- T2
commit; -- T2 ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh due to a conflict:

Yields the following where the invariant holds:

select * from account;
  name  |   type   | balance
--------+----------+----------
  alice | checking |  500.00
  alice | saving   | -400.00
(2 rows)

Deposit Report

Setup schema (before every test run):

create table if not exists control
  (
    deposit_no int not null
  );
create table if not exists receipt
  (
    receipt_no bigint NOT NULL PRIMARY KEY DEFAULT unique_rowid(),
    deposit_no int not null,
    payee text not null,
    amount numeric(19,2) not null
  );

DELETE from control where 1=1;
DELETE from receipt where 1=1;
insert into control values (1);

insert into receipt
  (deposit_no, payee, amount)
  values ((select deposit_no from control), 'Crosby', 100.00);
insert into receipt
  (deposit_no, payee, amount)
  values ((select deposit_no from control), 'Stills', 200.00);
insert into receipt
  (deposit_no, payee, amount)
  values ((select deposit_no from control), 'Nash', 300.00);

Test sequence:

begin; set transaction isolation level serializable ; -- T1
insert into receipt (deposit_no, payee, amount) values ( (select deposit_no from control), 'Young', 100.00 ); -- T1
select * from receipt; -- T1

begin; set transaction isolation level serializable ; -- T2   
select deposit_no from control; -- T2
update control set deposit_no = 2 where 1=1; -- T2
commit; -- T2

begin; set transaction isolation level serializable ; -- T3   
select * from receipt where deposit_no = 1; -- T3. Blocks on T1
commit; -- T1
(T3 unblocks)
commit; -- T3

Yields:

select * from receipt;
      receipt_no     | deposit_no | payee  | amount
---------------------+------------+--------+---------
  860561294810873858 |          1 | Crosby | 100.00
  860561295115354114 |          1 | Stills | 200.00
  860561295382970370 |          1 | Nash   | 300.00
  860561358736326657 |          1 | Young  | 100.00
(4 rows)

Rollover

This example was created to show that PostgreSQL can roll back read-only transactions to prevent serialization conflicts. It won't happen in CockroachDB.

Schema setup:

create table if not exists rollover (
  id int primary key, 
  n int not null
  );
delete from rollover where 1=1;
insert into rollover values (1,100), (2,10);

Financial transaction under serializable isolation:

begin; set transaction isolation level serializable ; -- T1
update rollover
  set n = n + (select n from rollover where id = 2)
  where id = 1; -- T1

begin; set transaction isolation level serializable ; -- T2
update rollover set n = n + 1 where id = 2; -- T2 - blocks on T1
commit; --T2 

begin; set transaction isolation level snapshot ; -- T3
select count(*) from rollover; -- T3 - blocks on T1         
commit; -- T1

select n from rollover where id in (1,2); -- T3

Conclusion

This article showcased a few examples outlined in the PostgreSQL SSI description page. It highlights some runtime differences between PostgreSQL SSI and CockroachDB serializable.

The "I" part in ACID stands for serializable isolation, which means that a database that formally claims to support ACID needs to provide the highest isolation standard in SQL - serializable. Serializable isolation guarantees that even though transactions may execute in parallel, the result is the same as if they had executed one at a time, without any concurrency.

Executing transactions serially would lead to the same result, but it would also destroy any performance aspirations. Concurrent execution is a must-have. One way to look at it is that it's a magic show hosted by the database, giving the illusion to clients they are the exclusive users of the database, completely free from interference from others.

(image from: https://blog.acolyer.org/2016/02/24/a-critique-of-ansi-sql-isolation-levels/)

Isolation levels are however confusing and ambiguous, in particular for distributed databases where you don't have a single time source. Not only are isolation levels difficult to understand but can also mean different things. Serializable in Oracle, for example, actually means Snapshot (which is weaker) and Repeatable Read in PostgreSQL means snapshot (which is stronger). Snapshot also permits write skew (A5B), which Repeatable Reads does not. Then we have Oracle Read Consistency, which is like Read Committed, only stronger by advancing the transaction timestamp for each SQL statement.

This ambiguity presents a real challenge for application developers and architects. They are tasked to figure out when a given isolation level is sufficient for correct execution. It also makes it more difficult to think in terms of portability between databases when the behaviour is different. One piece of advice is that unless you are 100% sure of what anomalies business rule invariants are exposed to, then go for a higher level of isolation.

Related Resources:

• http://www.bailis.org/blog/understanding-weak-isolation-is-a-serious-problem/

• http://www.bailis.org/blog/when-is-acid-acid-rarely/

• http://martin.kleppmann.com/2014/11/25/hermitage-testing-the-i-in-acid.html

• https://blog.acolyer.org/2016/02/24/a-critique-of-ansi-sql-isolation-levels/

The goal of transaction isolation is to find a good balance between safety and performance for concurrent transactions. A database should allow concurrent access to data while still being safe, meaning that concurrent operations that happen to interleave, should not observe intermediate state, overwrite other transaction writes or violate invariants guarded by constraints. It's the database being liberal and conservative at the same time.

A higher isolation level reduces and even eliminates most known read/write conflict anomalies, at the expense of performance and rollbacks on contended operations. Performance is increased and transient errors are reduced by lowering the isolation level, effectively requiring less coordination and planning effort by the database to guarantee safe, concurrent execution. It depends on the database implementation though, and in some cases, the difference in performance is small for non-contending operations.

The main downside of lowering isolation is that applications become more exposed to read-write phenomena (anomalies) that may cause data loss or corruption in the worst case. These types of errors are quite difficult to track down and test for.

Read and Write Anomalies

The lowest isolation level is Read Uncommitted (RU) meaning basically that all (most) bets are off. It allows dirty reads (P1) where transaction T1 is allowed to read transaction T2:s writes that haven't been committed yet. Read Uncommitted must prohibit dirty writes (P0) though, where T1 would modify T2:s write before it has committed.

The highest ACID isolation level is serializability which means transactions are not exposed to any read/write anomalies. A client can safely read and write without having to worry about other transactions possibly performing the same operations. The database will guarantee that no client will ever observe any inconsistent state and that all invariants will be preserved at commit.

In between you have all the rest. Anomalies can either be permitted or prevented by using ANSI SQL isolation levels, or something even higher like strict serializability or linearizability (external consistency).

Common anomalies include:

• Dirty write (P0)

• Dirty read (P1)

• Fuzzy read (P2)

• Phantom (P3)

• Strict Phantom (A3)

• Lost update (P4)

• Cursor lost update (P4C)

• Read Skew (A5A)

• Write Skew (A5B)

Surprisingly enough, the default isolation level in most modern databases is read committed (RC). It is a fundamentally unsafe isolation level exposed to lost updates (P4) and more. Still, many applications are using it and seem to work fine most of the time.

But how can you be sure you will not be the next Bitcoin exchange or e-commerce site that gets exploited by weak isolation? Trying to navigate through these things is not far from trying to beat classic Minesweeper.

Isolation Levels in Databases

Modern lock-free MVCC databases (and others) like Oracle and PostgreSQL default to Read Committed (RC). As MVCC databases, they also support snapshot isolation (SI) which is a slightly weaker model than serializable.

SI does not use locking, which is sort of the point with MVCC, but instead every transaction operates on an isolated snapshot of committed data whose values are not visible to other transactions unless the transaction commits.

SI sorts in somewhere between read committed and serializable (Berenson and Adya). It prevents P4 (lost update) by applying a first committer wins policy and like Repeatable Read (RR) it prohibits P0, P1 and P2. It prevents a special version of P3 called A3 (Phantom) that RR allows, but allows A5B (write skew) that RR prevents. Write skew is when two concurrent transactions are writing based on reading a data set which overlaps what the other is writing.

PostgreSQL (since 9.1) implements serializable isolation on top of SI, called serializable snapshot isolation or SSI. It prevents A5B (write skew) by forcing conflicts through either promoting reads to writes or by analyzing dependency cycles in transactions.

If you are not already confused at this point, then congratulations. These conditions and more are outlined in far more detail in the A Critique of ANSI SQL Isolation Levels paper.

Cockroachdb only implements serializable isolation, which narrows down the options. It gives peace of mind if you are concerned about read/write anomalies.

Conclusion

ACID transaction isolation levels are ambiguous and tricky to grok. Most modern databases implement transaction isolation differently and often have weak defaults where applications need to opt-in for higher isolation. In CockroachDB, the only choice is serializable which is the highest level in the SQL standard.

In a previous article, we looked at an architectural pattern named entity-control-boundary (ECB) mapped to Spring meta-annotations for transaction management and retries. That post didn't go very deep into this architectural pattern, which is the purpose of this article.

ECB is an architecture pattern originally coined in Ivar Jacobson's use-case-driven object-oriented software engineering (OOSE) method published in 1992. In other words, it dates way back in time yet it's not super well-known.

This pattern fits really well into organising transaction boundaries in application code and you don't need to go all-in on all the fun stuff like UML, waterfall or unified process to use it. It's really straightforward and mainly serves a documentative and declarative purpose.

The ECB pattern is centred around defining clear responsibilities and interactions between different categories of classes. It can be broken down into four elements of a robustness diagram: Actor, Boundary, Control and Entity.

The following robustness constraints apply:

• Actors may only know and communicate with boundaries.

• Boundaries may communicate with actors and controls only.

• Controls may know and communicate with boundaries and entities, and if needed other controls.

• Entities may only know about other entities but could communicate also with controls.

Source: https://en.wikipedia.org/wiki/Entity-control-boundary

In other words, there could be dependencies and interactions like this in a single service:

ECB offers structure and low-effort consistency to transaction boundary demarcating, simply by using transaction attributes or preferably dedicated meta- or stereotype annotations. Without this level of structure, the chances are that the boundaries become unclear and blurry which may result in hard-to-find errors and system inconsistencies.

Definitions

Let's map the ECB concept to concrete architectural elements (namespaces and annotations) that you typically see in a Spring Boot application.

Boundary

A boundary is coarse-grained and exposes functionality towards users or other systems (actors). It is typically implemented as a web controller or business service facade. It should be thin and delegate business processing to more fine-grained control services, if applicable. It acts both as a remoting and transaction boundary.

A boundary should never be invoked from within a transaction context. It means that only a boundary is allowed to create new transactions. To that end, boundaries must have the REQUIRES_NEW transaction attribute on their public, transactional methods. This propagation attribute will always create a new transaction and suspend any existing one.

Transaction suspension via REQUIRES_NEW and nested transactions via NESTED are different things. Nested transactions allow for a rollback to the beginning of the sub-transaction while keeping the transactional state of the outer transaction. Nested transactions are expressed using savepoints. Unfortunately, savepoints are not fully supported in all Java application frameworks but it's available in Spring Boot, although not in JPA. If you are using something else than JPA then savepoints opens up a few more opportunities. For the transaction boundary type discussed here, however, we don't use nested transactions (via savepoints) but just regular unnested local transactions.

Characteristics

Key characteristics for a transaction and remoting boundary:

• Independent of other service facades or web controllers.

• Granularity is more coarse-grained than a service.

• The layer that exposes functionality outside of the business tier.

• The only layer that is accessible from an external client (typically via a web API).

• Methods are preferably idempotent for client convenience.

• Never invoked within a transaction context.

Solution

Typical implementation elements in Spring Boot:

• Can be a Business Facade, Web Controller or Service Activator (Message Listener) where:

  • A business facade uses @Service

  • A controller uses @RestController

  • A service activator uses @Service

• Implements simple business logic or delegates to services (Control) or even repositories (Entity).

• Always uses transaction demarcation REQUIRES_NEW since it's a boundary.

  • @Transactional(propagation = Propagation.REQUIRES_NEW)

Conventions

• Represents the remoting entry point (when it's a web @Controller).

• An interface or class with thin, coarse-grained methods.

• Should be located in a dedicated boundary or service namespace.

• Should use the documentative meta-annotation to emphasise its architectural role.

• The business interface should be named after business concepts.

Example of a boundary meta-annotation (annotation describing or grouping other annotations). Notice that it incorporates the Spring @Transactional annotation with propagation REQUIRES_NEW.

@Inherited
@Documented
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.TYPE, ElementType.METHOD})
@Transactional(propagation = Propagation.REQUIRES_NEW)
public @interface Boundary {
}

Boundary service facade example:

@Service
public class TransactionServiceImpl implements TransactionService {
    @Autowired
    private AccountRepository accountRepository;

    @Autowired
    private TransactionRepository transactionRepository;

    @Override
    @Boundary
    public Transaction submitTransferRequest(TransferRequest request) {
        if (!TransactionSynchronizationManager.isActualTransactionActive()) {
            throw new IllegalStateException("No transaction context");
        }
    }
}

Boundary web controller example:

@RestController
@RequestMapping(value = "/api/transaction")
public class TransactionController {
    @GetMapping
    @Boundary
    public PagedModel<TransactionModel> listTransactions(@PageableDefault(size = 5) Pageable page) {
        return pagedTransactionResourceAssembler
                .toModel(bankService.find(page), transactionResourceAssembler);
    }
}

Boundary service activator example:

@Service
public class KafkaChangeFeedConsumer {
    @KafkaListener(topics = TOPIC_ACCOUNTS, containerFactory = "accountListenerContainerFactory")
    @Boundary
    public void accountChanged(@Payload AccountPayload event,
                               @Header(KafkaHeaders.RECEIVED_PARTITION) int partition,
                               @Header(KafkaHeaders.OFFSET) int offset) {
    }
}

Control

A control service is a fine-grained realization of activities or sub-processes. It's where business functionality is implemented. It must always be invoked within the context of a transaction and is not allowed to create new transactions. To that end, it must have the MANDATORY transaction attribute. The same policy applies to repository interfaces or classes that perform persistence logic. A repository is not allowed to create a new transaction.

A control service that is just a thin delegation layer between a boundary and repository contract (like Spring Data repository) adds no real value. In that case, to reduce boilerplate code, consider accessing repository resources directly from boundaries, effectively collapsing the boundary and service into one artefact.

Characteristics

• Services should be independent of other services.

• The granularity is finer than a boundary.

• Services are not available or visible outside of the business tier.

• Methods should be idempotent and always be invoked from a transactional context.

Solution

• Can be a business service that implements business logic.

• Not allowed to start new transactions.

• Use MANDATORY transaction propagation attribute.

  • @Transactional(propagation = Propagation.MANDATORY)

Conventions

• Interface or class with fine-grained methods and PDOs.

• Should be located in a dedicated ..service package.

• Should use a documentative meta-annotation to emphasise its architectural role.

• The business interface should be named after business concepts.

Example of a control service meta-annotation. Notice that it incorporates the Spring @Transactional annotation with propagation MANDATORY.

@Inherited
@Documented
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.TYPE, ElementType.METHOD})
@Transactional(propagation = Propagation.MANDATORY)
public @interface Control {
}

A control service example:

@Service
public class DefaultTransactionService implements TransactionService {
    @Autowired
    private AccountRepository accountRepository;

    @Override
    @Control
    public Transaction createTransaction(UUID id, TransactionForm transactionForm) {   Assert.isTrue(TransactionSynchronizationManager.isActualTransactionActive(), "Expected transaction");
    }
}

Entity

Entities are a static model representation of the application state mapped against a database. Usually through some ORM technology such as JPA and Hibernate. Entities must never be visible outside the system boundaries or JVM, but can optionally have DTO or value object/model representations. In most cases, DTOs add little value to hide implementation detail and protect internal entities. One exception could be representation models in Spring HATEOAS that add hypermedia controls on top of domain entities (you can use EntityModel also).

In terms of ECB, the entity element is simply represented by JPA entities and Spring Data repositories with the @Repository annotation. There's not much more to it than emphasising the architectural role.

Transaction Retries

Now that we are familiar with ECB, let's wrap things up by also adding the capability to retry transient SQL errors. When a SQL error with the state code 40001 encountered, it's typically safe to retry the local transaction from a database point of view. If the retried business facade method and its descendants are nonidempotent, then some precautions may be needed to avoid multiple side effects (again strive for idempotency).

The simplest approach is to use Spring Retry with a custom exception classifier and exponential backoff. How this is done is outlined in more detail in this post.

A brief example of a retriable boundary for completeness (notice the @Retryable):

@Service
public class OrderService {
    @Boundary
    @Retryable
    public Order updateOrderStatus(Long orderId, ShipmentStatus status, BigDecimal amount) {
        // Call DB and maybe do other idempotent stuff
        return order;
    }
}

You can also push the @Retryable annotation to @Boundary which then automatically adds the retry capability to all annotated methods.

@Inherited
@Documented
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.TYPE, ElementType.METHOD})
@Transactional(propagation = Propagation.REQUIRES_NEW)
@Retryable(exceptionExpression = "@cockroachExceptionClassifier.shouldRetry(#root)", maxAttempts = 5, backoff = @Backoff(maxDelay = 15_000, multiplier = 1.5))
public @interface Boundary {
}

The exception classifier for completeness:

@Component
public class CockroachExceptionClassifier {
    private final Logger logger = LoggerFactory.getLogger(getClass());

    private static final String SERIALIZATION_FAILURE = "40001";

    public boolean shouldRetry(Throwable ex) {
        if (ex == null) {
            return false;
        }
        Throwable throwable = NestedExceptionUtils.getMostSpecificCause(ex);
        if (throwable instanceof SQLException) {
            return shouldRetry((SQLException) throwable);
        }
        logger.warn("Non-transient exception {}", ex.getClass());
        return false;
    }

    public boolean shouldRetry(SQLException ex) {
        if (SERIALIZATION_FAILURE.equals(ex.getSQLState())) {
            logger.warn("Transient SQL exception detected : sql state [{}], message [{}]",
                    ex.getSQLState(), ex.toString());
            return true;
        }
        return false;
    }
}

General Guidelines

A few general guidelines for transaction management.

Avoid Remote Calls

Avoid remote calls to external resources from within a database transaction context. You may end up locking up resources for a long time in case of network communication problems or issues with the target endpoint. You are also exposed to the challenge of dual writes, where one part succeeds and the other part fails leaving the system in an inconsistent state (typically addressed with the outbox pattern).

Read-Only Implicit Transactions

If you are not performing any writes, then consider using read-only, implicit transactions. The readOnly attribute in @Transactional gives a clue to the transaction management that it's a read-only operation. The JPA provider may then perform certain optimizations.

Non-transactional read-only (implicit transactions) methods can use SUPPORTS propagation. This works as long as the default autoCommit flag is not enabled in the data source.

HikariDataSource ds = properties
        .initializeDataSourceBuilder()
        .type(HikariDataSource.class)
        .build();
ds.setAutoCommit(false); // false is the default, setting it to true makes all transactions explicit

Conclusion

This article describes the ECB architecture pattern to enhance transaction robustness in Spring Boot apps. Database transactions must always be started by boundaries and nowhere else. A boundary is typically a web controller or business service facade.

• Boundaries use REQUIRES_NEW propagation.

• Control services and repositories use MANDATORY propagation.

• Non-transactional read-only methods can use SUPPORTS propagation.

Databases aren't great for storing binary large objects, aka BLOBs. By large meaning several MBs of size. If that is needed, then it's likely much more performant to just use the filesystem or cloud storage and only store references in the database for structure.

Smaller objects are typically fine to store in the database. To that end, this article will demonstrate how to manage BLOBs using JPA and Hibernate along with CockroachDB. In CockroachDB, the BLOB type is an alias for the BYTES data type. As mentioned in the docs, it's recommended to keep values under 1 MB to ensure adequate performance. Above that threshold, write amplification and other considerations may cause significant performance degradation.

Mapping BLOBs in JPA

When using Hibernate, you typically use the @Lob annotation and java.sql.Blob which maps to the SQL BLOB data type.

@Entity
@Table(name = "attachment")
public class Attachment {   
    @Column(name = "content")
    @Basic(fetch = FetchType.LAZY)
    @Lob
    private Blob content;
    ...
}

You can also use a byte[] array or a String, but it's generally more performant to use a streaming approach using the Blob type.

Using BLOB Mappings

You need to use Hibernate’s BlobProxy class to create a Blob. As you can see in the example below, it's pretty straightforward:

// Stores a BLOB represented by the inputStream
Blob content = BlobProxy.generateProxy(inputStream, contentLength);

Attachment attachment = new Attachment();
attachment.setContent(content);
attachment.setName(name);
attachment.setDescription(description);

attachmentRepository.save(attachment);

That's about it, very simple.

To read the BLOB back again, it recommended to use the streaming approach:

// Lookup attachment by ID and stream the blob to the outputStream
Attachment attachment = attachmentRepository.getReferenceById(id);
try (InputStream in = new BufferedInputStream(
        attachment.getContent().getBinaryStream())) {
    FileCopyUtils.copy(in, outputStream);
} catch (SQLException | IOException e) {
    throw new DataRetrievalFailureException("Error reading attachment data", e);
}

If you would provide a REST endpoint for downloading BLOB attachments, then it could look like this when implemented using Spring Boot:

@GetMapping("/download/{id}")
public ResponseEntity<StreamingResponseBody> downloadAttachment(@PathVariable("id") Long id) {
    Attachment attachment = attachmentService.findById(id);
    StreamingResponseBody responseBody =
            outputStream -> attachmentService.streamAttachment(attachment, outputStream);
    return ResponseEntity.ok()
            .header(HttpHeaders.CONTENT_TYPE, attachment.getContentType())
            .header(HttpHeaders.CONTENT_DISPOSITION, "inline")
            .header("Cache-Control", "no-cache, no-store, must-revalidate")
            .header("Pragma", "no-cache")
            .header("Expires", "0")
            .body(responseBody);
}

StreamingResponseBody is used for asynchronous request processing where the application can write directly to the response OutputStream.

Demo Project

This demo project is a runnable Spring Boot application that provides a REST API for querying and uploading attachments with BLOB content.

To build:

git clone git@github.com:kai-niemi/roach-spring-boot-v3
cd roach-spring-boot-v3
chmod +x mvnw
./mvnw clean install

To run:

cockroach sql --insecure --host=localhost -e "CREATE database spring_boot_demo"

java -jar spring-boot-blob/target/spring-boot-blob.jar

Check that the service is up at http://localhost:8090.

Upload an image file using cURL:

curl http://localhost:8090/attachment/form \
-H "Content-Type: multipart/form-data" \
-v \
-F "content=@spring-boot-blob/src/test/resources/test.jpg" \
-F "fileName=test.jpg" \
-F "description=test.jpg"

Source Code

The code for this article is available on GitHub.

Conclusion

This article provides a guide on how to manage binary large objects (BLOBs) using JPA and Hibernate with CockroachDB. It demonstrates the @Lob annotation, java.sql.Blob type, and a runnable Spring Boot example.

Quality attributes are synonymous with non-functional requirements, as in the properties and characteristics of a software system that is not directly related to some functional aspect. Quality attributes are what ultimately define a system's runtime and evolutionary characteristics. These attributes indicate the software architecture style of the system and how different implementation mechanisms support these qualities, including the database.

In this article, we'll take a look at some of these attributes and more specifically how they can be defined, quantified and addressed when using the unique capabilities of CockroachDB.

Commonly used quality attributes for software systems (there's a lot more):

• Scalability - A system’s ability to scale with increasing load and business complexity.

• Reliability - A measure of a system's ability to detect and recover from failures and deliver correct, consistent and reliable results.

• Performance - The unit of time it takes to execute an operation, usually measured in response time, transaction time and throughput (work per time unit).

• Availability - A measure of the ability of a system to function in a state of serious service or infrastructure degradation.

• Evolvability - A system’s ability to make changes with low cost and small client/user impact.

• Maintainability - A system’s ability to be diagnosed and repaired after an error occurs.

• Interoperability - How the system interacts with other subsystems or foreign services.

• Visibility - A system’s support for debugging and real-time monitoring.

• Security - A system’s ability to support security controls including access controls, encryption, data isolation, secure information processing and auditing for compliance.

Functional attributes in a software system are useless without quality, and non-functional attributes are useless without relevant purpose and meaning for a business. It's a task for architects and developers to map the problem domain against a solution domain and deliver solutions that meet both functional and non-functional requirements. The problem domain describes the what & why and the solution describes the how.

The re-usable artifacts are typically software design guidelines and principles that both guide the development of new software components as well as help delay software to deteriorate over time due to change. Change is a natural given for any software component and that's where the real cost sits in the lifecycle of software systems. Not so much in the initial development efforts. The architectural decisions made early on in a product's life cycle to support things like evolvability, maintainability and low cost of change are very important for the total cost of ownership.

There's always a balance between the amount of time invested in quality attributes against getting a new product or feature out on the market. Things need to be prioritized like anything else and the best way is to ask the business stakeholders what is most important to deliver.

Qualifying Quality Attributes

Definiting clear, measurable and quantifiable requirements is an art form. Both business-oriented and of technical nature. To get good at it you need to ask questions and a lot of them.

How can we define, measure and communicate the importance of abstract things like evolvability? Let's give it a try by listing a few questions for each listed quality attribute. Because the database is a critical infrastructure component for any software system, let's also see how each quality attribute can be addressed from a database point of view. Not just any database but CockroachDB.

Scalability

A system’s ability to scale with increasing load and business complexity.

Scalability describes the ability of a system to cope with increased load, most often measured in latency percentiles and throughput. When the load increases on a system, it's relevant to observe much more resources are needed to maintain the same level of performance. When a business grows in terms of increased customer demands, new markets or acquisitions, scalability also describes the ability to adapt systems to that new reality without having to undergo major refactoring or redesign efforts.

Questions:

• What are the data volumes and what’s the expected growth over time?

• What is the impact of traffic, data or customer base growing by 10x?

• What level of scalability is relevant, local, regional or global?

• How important is it to deliver a consistent customer experience globally?

• What does the traffic load look like for steady state, peak, extended peak and stress?

• Does the system need to auto-adjust to increased/decreased spike demands?

• Is data archiving to offline systems needed?

Solutions:

CockroachDB is a geo-distributed SQL database designed to scale horizontally by adding more nodes to a cluster, increasing compute and IO capacity. Given the transactional properties and consistency guarantees, it also enables crafting multi-active systems, where its relevant to distinguish between response times and service times.

Service time is the time it takes to process a synchronous request entering a service's boundary and preparing a response. Response time is service time plus the time it takes to transport traffic over the network including queuing delays. For a cluster spanning for example SA-EU-AP, we can drastically reduce response times by servicing requests in the proximity of where it's stored.

Multi-active systems add the ability to control both service and response time through physical network topologies, data locality and replication policies. Depending on how nodes and replicas are arranged, both latency and fault tolerance or survival goals can also be controlled.

It provides the equivalence of a content-delivery network (CDN) for transactional data, logically spanning the entire globe. No matter which (edge) node you interact with, it will provide a consistent and accurate result.

Reliability

A measure of the system's ability to detect and recover from failures and deliver correct, consistent and reliable results.

Reliability describes the ability of a system to operate correctly (do the right thing) at the desired level of performance, both under heavy concurrent workloads and in the event of infrastructure failures like partial and full zone or region failures. It’s more difficult to measure than scalability but can be defined in different ways.

For instance:

• Not loose or corrupt data because of infrastructure failures (no partial commits)

• Not loose or corrupt data because of concurrency anomalies (ex: lost updates, phantom reads, read/write skew)

• Not provide stale data where authoritative data is expected

• Prevent diverging histories of state and throwing away committed writes when healing from network partitions

• Not breach correctness rules or invariants (ex: negative balance in accounting)

• Tolerate human mistakes and errors

Questions:

• What is the anatomy of a typical business transaction?

  • How does it get triggered?

  • What is the average duration?

  • How much information needs to be scanned vs returned?

  • What are the expected success and failure outcomes?

• How are key business rule invariants to be protected?

• Is reading information always needed to be authoritative or potentially stale?

• How are business transactions spanning multiple services handled?

• How are transient transaction errors handled?

• How are timeouts/indeterminate outcomes handled?

• How is service idempotency implemented?

Solutions:

One feature of multi-active systems is the ability to operate simultaneously from multiple geographies with sustained throughput and transactional integrity, both during steady state and during disruptions or even disasters in other regions. In a 3-region CockroachDB deployment, one entire region can have an outage without affecting forward progress in the other two.

Performance

The unit of time it takes to execute an operation, usually measured in response time or transaction time and throughput (work per time unit).

Questions:

• How is customer experience affected by high response times and low throughput?

• How many concurrent active sessions/users will connect to the system?

• Are there specific performance and throughput targets (service level indicators) on a per-use case basis?

• What is the ratio between reads and writes?

• Can reads be served as potentially stale or always authoritative?

• Will there be a caching tier to scale out reads, in that case, how is the cache invalidated and kept in sync?

Defining performance requirements should ideally be context or journey specific. Most larger systems are composed of different customer journeys such as registration, login, deposit, withdrawal, pay and so on. Not all journeys have the same NFRs and may touch different services, hence it makes sense to define the performance goals based on that rather than at individual service level.

For example, for journey X:

• Raw data size: 4TB

• Active connections: 400 to 500

• Actual users: 7M

• Active users: 500k

• Reads must be authoritative

• Sustained throughput under 60min, qualified with:

  • 5,000 business transactions per sec, equivalent to 15K QPS, at P99 < 120ms, read ratio 75%

• Peak throughput under 30min, qualified with:

  • 7000 business transactions per sec, equivalent to 21K QPS, at P95 < 150ms, read ratio 65%

• Extended peak throughput under 15min, qualified with:

  • 10,000 business transactions per sec, equivalent to 30K QPS, at P95 < 300ms, read ratio 65%

Solutions:

CockroachDB delivers predictable response times and throughput for different workload scales. When optimizing performance characteristics, it is typically a matter of finding opportunities in:

1. Application workload patterns

2. Schema design

3. Cluster hardware capacity and utilization

4. Replica and leaseholder placement

Most opportunities are outlined in SQL performance best practices. A workload should be evenly distributed across all machines of a cluster (no hotspots) which happens automatically given a few schema design and load balancing considerations. By using different multi-region capabilities, the coordination between nodes over the network can also be minimized, drastically improving both read and write performance.

Let's wrap with the other quality attributes by just highlighting a few relevant questions.

Availability

A measure of the ability of a system to function in a state of serious service or infrastructure degradation.

Questions:

• What level of redundancy will the system have (any accepted SPOFs)?

• What type of infrastructure failures must the system survive (cloud, region, zone, rack or server)?

• What’s the business impact on degraded or denied forward progress?

• Will the system continue to function on partial infrastructure failure?

• Are there any specific RTO and RPO metrics?

• What are the requirements for backup and restore (MTTR)?

Evolvability

A system’s ability to make changes with low cost and small client impact.

Questions:

• What's the structure and process around the development and deployment pipeline?

• Are downtime windows allowed for production deployments?

• Is a pre-production deployment environment needed?

• What are the key factors that impact time-to-value in new business initiatives/improvements?

• How does changing/adding functionality impact existing functionality?

Maintainability

A system’s ability to be repaired after an error occurs.

Questions:

• What design principles are applied to reduce maintenance efforts?

• How much QA/OPS effort is needed to verify and deploy the system?

Interoperability

How the system interacts with other subsystems or foreign services.

Questions:

• How does data flow into the system and what is the output?

• Is the system classified as online, nearline or offline?

• What are the major infrastructure components involved (database, message broker)?

• Is the system classified as a system of record or a system of access?

• Is the interaction model a typical request/response based model or an event-driven, async model?

Visibility

A system’s support for debugging and real-time monitoring.

Questions:

• How is the system monitored and acting on alerts?

• How can problems quickly be identified and corrected?

Security

A system’s ability to support security controls including access controls, encryption, data isolation, secure information processing and auditing for compliance.

Questions:

• Will the system run in a PCI or equivalent regulated environment?

• Will the system handle PII data?

• What security mechanisms does the system require on ingress and egress channels, or data protection?

Conclusion

This article discusses non-functional requirements, also known as quality attributes, which are properties and characteristics of a software system that is not directly related to its functional aspects. It looks at how these requirements can be defined, quantified, and addressed when using the capabilities of CockroachDB.

Introduction

CockroachDB uses parallelism in many parts of its architecture to deliver high-scale distributed SQL execution. For example, to improve write performance, it uses a parallel atomic commit protocol designed to cut the commit latency of a transaction from two roundtrips of consensus to one. When combined with transaction pipelining, where write intents are replicated from leaseholders in parallel rather than sequentially, all waiting happens in the end at commit time, thereby drastically reducing latencies for multi-statement write transactions.

To improve read performance in multi-region high-latency deployments, the cost-based optimizer performs what's referred to as a locality-optimized search. The optimizer may begin to scan for rows in the gateway node's local region and fan out to remote regions in parallel, but only if the local region did not satisfy the query. The remote lookup (performed in parallel) result is returned to the gateway once received without having to wait for completion. This increases read performance in multi-region deployments since results can be returned from wherever they are first found, without waiting for the completion of all lookups.

Last but not least, CockroachDB also uses vectorized SQL query execution, designed to process batches of columnar data instead of just a single row at a time. In the longer term, this will also make use of vectorized CPU (SIMD) instructions.

Parallelism is well exploited in the algorithms and mechanisms that CockroachDB uses. This works well for both larger and smaller statements that don't scan large volumes of data, which is typically something you'd want to avoid doing anyway in an OLTP database.

Now to the purpose of this article, what can the client do to take this even further?

Client-side Parallelism

The CockroachDB database (and SQL for that matter) does a decent job to hide the implementation details from clients through all abstraction layers. One of the primary tasks of a SQL database is to provide the illusion to clients that they are the sole users, free to read and write any piece of information without interference from others. In reality, the environment is highly concurrent and parallelized, which in practical terms means that the database is allowed to reorder concurrent transactions as long as the result is the same as if they had executed one at a time (serially), without any concurrency. This is the definition of SERIALIZABLE transaction isolation.

A SQL database is designed to be highly capable of accepting queries from multiple application instances and threads in parallel. In a typical request-response, thread-bound execution model you get a connection from the pool, send a single or multi-statement transaction, await its completion and close the connection (recycled to the pool). While this gives a level of parallelism in terms of multiple application-level threads, it doesn't help that much for larger scans beyond what the database offers.

What if you want to take things a step further in terms of parallel execution and involve the client? For example, by first running a very large query that scans hundreds of thousands of rows to compute an aggregated sum in the database and then do the equivalent client side by decomposing the query into smaller blocks run in parallel. Let's find out if it makes any difference.

Example Use Case

Assume we have a simple product table holding an inventory column.

create table product
(
    id        uuid           not null default gen_random_uuid(),
    version   int            not null default 0,
    inventory int            not null,
    name      varchar(128)   not null,
    price     numeric(19, 2) not null,
    sku       varchar(128)   not null unique,
    country   varchar(128)   not null,

    primary key (id)
);

Next, we add a covering index on the country and insert a huge bunch of products:

CREATE INDEX ON product (country) STORING (inventory,name,price);

insert into product (inventory,name,price,sku,country)
select 10 + random() * 50,
       md5(random()::text),
       500.00 + random() * 500.00,
       gen_random_uuid()::text,
       'US'
from generate_series(1, 500000) as i;
-- Repeat insert for 9 more countries, in total 5M rows

Composed Query

Let's run a single composed query to get the total inventory sum grouped by country:

select sum(p.inventory), p.country from product p group by p.country;

Gives:

select sum(p.inventory), p.country from product p group by p.country;
    sum    | country
-----------+----------
  17251976 | BE
  17253042 | DE
  17234287 | DK
  17253539 | ES
  17229425 | FI
  17250751 | FR
  17247093 | NO
  17257296 | SE
  17237964 | UK
  17261461 | US
(10 rows)


Time: 4.083s total (execution 4.083s / network 0.000s)

This query still runs fairly fast for a total row count of 5M. Let's look a the explain plan to see that we are scanning the entire table:

explain analyze select sum(p.inventory), p.country from product p group by p.country;
                                                            info
-----------------------------------------------------------------------------------------------------------------------------
  planning time: 421µs
  execution time: 4.1s
  distribution: full
  vectorized: true
  rows read from KV: 5,000,000 (466 MiB, 47 gRPC calls)
  cumulative time spent in KV: 3.8s
  maximum memory usage: 10 MiB
  network usage: 0 B (0 messages)
  regions: europe-west1

  • group (streaming)
  │ nodes: n1
  │ regions: europe-west1
  │ actual row count: 10
  │ estimated row count: 10
  │ group by: country
  │ ordered: +country
  │
  └── • scan
        nodes: n1
        regions: europe-west1
        actual row count: 5,000,000
        KV time: 3.8s
        KV contention time: 0µs
        KV rows read: 5,000,000
        KV bytes read: 466 MiB
        KV gRPC calls: 47
        estimated max memory allocated: 10 MiB
        estimated row count: 7,036,818 (100% of the table; stats collected 5 days ago; using stats forecast for 5 days ago)
        table: product@product_country_idx
        spans: FULL SCAN
(31 rows)


Time: 4.091s total (execution 4.090s / network 0.000s)

Decomposed Parallel Queries

Let's decompose the single query into multiple ones and run them in parallel, then combine the results in the end. We refactor the query by removing the GROUP BY and filtering on the indexed country column instead. Effectively the GROUP BY operator is moved client side.

Example of a single country query:

select sum(p1_0.inventory) from product p1_0 where p1_0.country='US';
    sum
------------
  17261461
(1 row)

Time: 231ms total (execution 231ms / network 0ms)

Let's also check the execution plan:

explain analyze select sum(p1_0.inventory) from product p1_0 where p1_0.country='US';
                                                           info
---------------------------------------------------------------------------------------------------------------------------
  planning time: 535µs
  execution time: 248ms
  distribution: full
  vectorized: true
  rows read from KV: 500,000 (47 MiB, 5 gRPC calls)
  cumulative time spent in KV: 225ms
  maximum memory usage: 10 MiB
  network usage: 0 B (0 messages)
  regions: europe-west1

  • group (scalar)
  │ nodes: n1
  │ regions: europe-west1
  │ actual row count: 1
  │ estimated row count: 1
  │
  └── • scan
        nodes: n1
        regions: europe-west1
        actual row count: 500,000
        KV time: 225ms
        KV contention time: 0µs
        KV rows read: 500,000
        KV bytes read: 47 MiB
        KV gRPC calls: 5
        estimated max memory allocated: 10 MiB
        estimated row count: 696,645 (9.9% of the table; stats collected 5 days ago; using stats forecast for 5 days ago)
        table: product@product_country_idx
        spans: [/'US' - /'US']
(29 rows)


Time: 249ms total (execution 249ms / network 0ms)

The estimated row count is about 10% of the table which sounds about right since we inserted 500K rows per country.

Now we apply a parallel fork and join operation at the client side. This means we fire ten concurrent threads with the individual queries and then await completion before proceeding. After that, the results are joined together.

For this example, we'll use Spring Data and a JPA query:

@Query("select sum(p.inventory) from Product p where p.country = :country")
Integer sumInventory(@Param("country") String country);

First queue up the workers, one for each country:

List<Callable<Pair<String, Integer>>> tasks = new ArrayList<>();
  StringUtils.commaDelimitedListToSet("SE,UK,DK,NO,ES,US,FI,FR,BE,DE").forEach(country ->
  tasks.add(() -> Pair.of(country, productRepository.sumInventory(country))));

Next, we unleash the workers to run in parallel while blocking until completion (or cancellation by timeout):

ConcurrencyUtils.runConcurrentlyAndWait(tasks, 10, TimeUnit.MINUTES, sums::add);

This utility method makes use of Java's CompletableFuture was introduced way back in Java 8. It's like a Swiss army knife for asynchronous computations using parallel decomposition constructs. Tasks are decomposed into steps that can be forked and joined in different stages to a final result. It's a very elegantly designed API.

In this example, we are just using a small subset of it to run our query tasks in parallel and join the results. It also adds cancellation, in case queries would go rogue and run for too long. Cancellation is not a natural part of CompletableFuture so there's a small trick in there to add that.

It's also using a bounded thread pool which means that no matter how many tasks are queued it will only run a limited number of concurrent threads by adding backpressure on the client code queuing up tasks. This is more lenient on thread scheduling since the client will be blocking anyway.

ScheduledExecutorService cancellationService
        = Executors.newSingleThreadScheduledExecutor();

ExecutorService executor = boundedThreadPool();

List<CompletableFuture<Void>> allFutures = new ArrayList<>();

final Instant expiryTime = Instant.now().plus(timeout, timeUnit.toChronoUnit());

tasks.forEach(callable -> {
    allFutures.add(CompletableFuture.supplyAsync(() -> {
                if (Instant.now().isAfter(expiryTime)) {
                    logger.warn("Task scheduled after expiration time: " + expiryTime);
                    return null;
                }
                Future<V> future = executor.submit(callable);
                long cancellationTime = Duration.between(Instant.now(), expiryTime).toMillis();
                cancellationService.schedule(() -> future.cancel(true), cancellationTime, TimeUnit.MILLISECONDS);
                try {
                    return future.get();
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                    throw new IllegalStateException(e);
                } catch (ExecutionException e) {
                    throw new IllegalStateException(e.getCause());
                }
            }, executor)
            .thenAccept(completionFunction)
            .exceptionally(throwableFunction)
    );
});

CompletableFuture.allOf(
        allFutures.toArray(new CompletableFuture[]{})).join();

executor.shutdownNow();
cancellationService.shutdownNow();

Once all the query sums are gathered we simply add them up client side using a stream API aggregator:

sums.stream().mapToInt(Pair::getSecond).sum()

OK, the result then? Here's the log output:

09:21:53.253  INFO [i.r.s.p.ParallelApplication$$SpringCGLIB$$0] Inventory sum for UK is 17237964
09:21:53.253  INFO [i.r.s.p.ParallelApplication$$SpringCGLIB$$0] Inventory sum for US is 17261461
09:21:53.253  INFO [i.r.s.p.ParallelApplication$$SpringCGLIB$$0] Inventory sum for DK is 17234287
09:21:53.253  INFO [i.r.s.p.ParallelApplication$$SpringCGLIB$$0] Inventory sum for SE is 17257296
09:21:53.253  INFO [i.r.s.p.ParallelApplication$$SpringCGLIB$$0] Inventory sum for ES is 17253539
09:21:53.253  INFO [i.r.s.p.ParallelApplication$$SpringCGLIB$$0] Inventory sum for NO is 17247093
09:21:53.253  INFO [i.r.s.p.ParallelApplication$$SpringCGLIB$$0] Inventory sum for FR is 17250751
09:21:53.253  INFO [i.r.s.p.ParallelApplication$$SpringCGLIB$$0] Inventory sum for DE is 17253042
09:21:53.253  INFO [i.r.s.p.ParallelApplication$$SpringCGLIB$$0] Inventory sum for FI is 17229425
09:21:53.253  INFO [i.r.s.p.ParallelApplication$$SpringCGLIB$$0] Inventory sum for BE is 17251976
09:21:53.254  INFO [i.r.s.p.ParallelApplication$$SpringCGLIB$$0] Total inventory sum is 172476834
09:21:53.254  INFO [i.r.s.p.ParallelApplication$$SpringCGLIB$$0] Verified inventory sum is 172476834
09:21:53.254  INFO [i.r.s.p.ParallelApplication$$SpringCGLIB$$0] Parallel execution time: PT1.1578745S
09:21:53.254  INFO [i.r.s.p.ParallelApplication$$SpringCGLIB$$0] Serial execution time: PT2.9943538S
09:21:53.254  INFO [i.r.s.p.ParallelApplication$$SpringCGLIB$$0] Execution time diff: 259%

In this simple example, we can notice a 260% performance improvement by decomposing the query and running these independently.

Conclusion

This article explains how CockroachDB uses parallelism to improve read and write performance, and how client-side parallel query execution can be used to further increase large query performance. An example use case is provided to illustrate how this works, using Spring Data and a JPA query to run a parallel fork and join operation at the client side with a bounded thread pool and a cancellation service.

The source code for the article is available on GitHub.

CockroachDB is a geo-distributed SQL database purpose-built from the ground up for high scalability, fault tolerance, cloud neutrality and usability for developers and operators. It also offers the highest SQL standard for transactional integrity - serializable isolation.

The term geo-distribution is to emphasise its capability to break out of the low-latency, stable networking assumptions of a single data center or single region deployment. CockroachDB clusters can span the globe and still offer one logical database towards applications with intact semantics and guarantees. No more need for manual sharding.

One major influence on performance, when nodes need to perform some level of coordination, is network latency. There are numerous mechanisms at work in CockroachDB to mitigate the effects of high cross-link latencies. For performance and to ensure safety and liveness in volatile and ephemeral hosting environments.

One key ingredient for high performance and linear scalability in CockroachDB is the ability to distribute workload both vertically and horizontally and thereby leverage the aggregate compute/IO capacity of a cluster. This is mainly achieved by the database itself but application designers can help by using some best practices around schema design and query patterns and load balance traffic across the cluster. In general terms, it's about avoiding hotspots from forming, avoiding contention if possible and reducing large table scans.

Techniques

CockroachDB uses a distributed SQL execution engine at its core, which means many things:

• The SQL layer is parallelized and pushes processors close to the proximity of data.

• The SQL optimizer is purpose-built with latency as a cost factor and locality awareness.

• The transaction layer uses a sophisticated pipelining and parallel commit protocol to reduce round trips to the theoretical minimum for consensus.

• Backup and restore are locality-aware.

• The vectorized execution engine provides good performance for a wide range of queries.

• Load-based splitting and rebalancing heuristics help to balance load across a cluster of machines.

Scaling reads is generally considered to be slightly easier than scaling writes. In a global or multi-region deployment topology, there are a couple of useful patterns including global tables, regional-by-row table localities and follower reads.

Global Tables

A global table means that all voting range replicas reside on nodes in the primary region, and non-voting replicas in remote regions to service consistent reads. The database automatically adjusts the replication factor (RF) to ensure there are range replicas for these tables in each configured region. It also uses something called non-blocking transactions in combination with non-voting replicas to provide low-latency global reads, also during workload contention. This concept is useful if you have a table which has a low volume of writes but high volumes of reads from different regions and the reads must be authoritative.

alter database test primary region "eu-north-1";
alter database test add region "eu-west-3";
alter database test add region "us-east-1";

create table postal_codes
(
    id   int primary key,
    code string
);

ALTER TABLE postal_codes SET LOCALITY GLOBAL;

Regional by Row

Regional by row is a table locality in which the home region is defined at the row level in a table. In contrast to regional tables, where all rows in a table have the same home region.

alter database test primary region "eu-north-1";
alter database test add region "eu-west-3";
alter database test add region "us-east-1";

CREATE TABLE users
(
    user_id     INT    NOT NULL,
    name        STRING NULL,
    postal_code int    NULL,

    PRIMARY KEY (user_id ASC)
);

ALTER TABLE users SET LOCALITY regional by row;

insert into users (user_id, crdb_region)
select no, 'eu-north-1' from generate_series(1, 100) no;

insert into users (user_id, crdb_region)
select no, 'eu-west-3' from generate_series(101, 200) no;

insert into users (user_id, crdb_region)
select no, 'us-east-1' from generate_series(201, 300) no;

Follower Reads

Follower reads are akin to Content Delivery Networks (CDN) by not having to chase the leaseholder for a given range that can potentially be located in another part of the world. Instead, the closest replica to a gateway node (receiving the request) can service the read with some staleness. There are two variants of follower reads called exact staleness and bounded staleness reads.

On the surface, follower reads may appear similar to global tables but the latter works quite differently through non-voting replicas and non-blocking transactions.

Follower reads are useful for more ad-hoc SQL queries for both partitioned and unpartitioned tables, where reads are allowed to be non-authoritative (potentially stale). Global tables are always authoritative (no staleness bounds) but pay for that in higher write latency.

The choice between follower reads and global tables should be driven by staleness requirements, read vs write volumes and survival goals. In other words, if the decision to write something is based on a read, and the value read must be authoritative, then a global table is a better choice. On the other hand, if write performance is a priority and a staleness window is acceptable, then follower-reads are better.

-- per statement:
SELECT ID FROM USERS AS OF SYSTEM TIME follower_read_timestamp() WHERE id=1;

-- alt via session var:
BEGIN;
SET TRANSACTION AS OF SYSTEM TIME follower_read_timestamp();
SELECT ..
COMMIT;

Further Reading

• Enabling the Next Generation of Multi-Region Applications with CockroachDB
  https://dl.acm.org/doi/10.1145/3514221.3526053

• CockroachDB: The Resilient Geo-Distributed SQL Database
  https://dl.acm.org/doi/10.1145/3318464.3386134

Summary

CockroachDB is a geo-distributed SQL database designed for scalability, fault tolerance, cloud neutrality, and usability. It offers distributed SQL execution and concepts like global tables, and follower reads to help balance read-heavy load across a cluster of machines. When deciding between follower reads and global tables, factors such as staleness requirements, read vs write volumes, and survival goals should be taken into consideration. Global tables are better for authoritative reads, while follower reads are better for write performance with an acceptable staleness window.

In this tutorial, we'll look at using spring retry for serialization conflict errors denoted by the SQL state code 40001.

Maven Setup

To use Spring Retry, you need to add the Spring Retry and Spring AOP dependencies to your pom.xml.

<dependency>
    <groupId>org.springframework.retry</groupId>
    <artifactId>spring-retry</artifactId>
    <version>2.0.1</version>
</dependency>

<dependency>
    <groupId>org.springframework</groupId>
    <artifactId>spring-aspects</artifactId>
    <version>5.3.10</version>
</dependency>

Configuration

To enable Spring Retry in an application, add the @EnableRetry annotation to any of the @Configuration classes:

@EnableRetry
@Configuration
@SpringBootApplication
public class MyApplication {
    public static void main(String[] args) {
        new SpringApplicationBuilder(MyApplication.class)
                .run(args);
    }
}

Example Service

Using Spring Retry is as simple as adding the @Retryable annotation to the methods to-be-retried:

@Service
public class OrderService {
    @Transactional(propagation = Propagation.REQUIRES_NEW)
    @Retryable
    public Order updateOrderStatus(Long orderId,
ShipmentStatus status, BigDecimal amount) {
        Order order = ...;
        return order;
    }
}

In our case, however, we want to be more specific on what type of exceptions qualify for a retry and also tailor the backoff policy to use an exponentially increasing delay with jitter.

@Service
public class OrderService {
    @Transactional(propagation = Propagation.REQUIRES_NEW)
    @Retryable(exceptionExpression = "@exceptionClassifier.shouldRetry(#root)",
            maxAttempts = 5,
            backoff = @Backoff(maxDelay = 15_000, multiplier = 1.5))
    public Order updateOrderStatus(Long orderId,
ShipmentStatus status, BigDecimal amount) {
        Order order = ...;
        return order;
    }
}

The backoff annotation parameters defines a policy that results in the ExponentialRandomBackOffPolicy is used at runtime.

Next, let's look at the exception classifier:

public class CockroachExceptionClassifier {
    private final Logger logger = LoggerFactory.getLogger(getClass());

    private static final String SERIALIZATION_FAILURE = "40001";

    public boolean shouldRetry(Throwable ex) {
        if (ex == null) {
            return false;
        }
        Throwable throwable = NestedExceptionUtils.getMostSpecificCause(ex);
        if (throwable instanceof SQLException) {
            return shouldRetry((SQLException) throwable);
        }
        logger.warn("Non-transient exception {}", ex.getClass());
        return false;
    }

    public boolean shouldRetry(SQLException ex) {
        if (SERIALIZATION_FAILURE.equals(ex.getSQLState())) {
            logger.warn("Transient SQL exception detected : sql state [{}], message [{}]",
                    ex.getSQLState(), ex.toString());
            return true;
        }
        return false;
    }
}

We also add the classifier bean to the configuration:

    @Bean
    public CockroachExceptionClassifier exceptionClassifier() {
        return new CockroachExceptionClassifier();
    }

The shouldRetry method simply looks for the exception type and if it is a SQLException that it has the proper state code 40001.

We could qualify exceptions with other state codes but then there are no guarantees of multiple side effects when retried. For example, if a transaction involves multiple INSERTs and the COMMIT is successful but lost in transit in the reply back to the client. In that case, it wouldn't use state code 40001 but more likely a broken connection error code.

To be safe, only retry on the state code 40001 and nothing else, unless you are sure about the side effects of your SQL transactions and it's considered safe (or the operations are idempotent).

Demo Project

Roach Retry is a project that provides runnable examples of different transaction retry strategies for Spring Boot and the JavaEE stack. It includes Spring Retry along with a simpler AOP-driven approach and JavaEE interceptors for old-style stateless session beans.

Step 1: Startup

Create the database:

cockroach sql --insecure --host=localhost -e "CREATE database roach_retry"

Build the app:

cd spring-retry
../mvnw clean install

Run the app:

java -jar target/roach-retry.jar

Then open another shell window so you have at least two windows. In any of the shells, check that the service is up and connected to the database:

curl --verbose http://localhost:8090/api

Step 2: Get Order Request Form

Print an order form template that we will use to create orders:

curl http://localhost:8090/api/order/template > form.json

Step 3: Submit Order Form

Create at least one purchase order:

curl http://localhost:8090/api/order -H "Content-Type:application/json" -X POST -d "@form.json"

Step 4: Produce a Read/Write Conflict

Assuming that there is now an existing order with ID 1 with status PLACED. We will read that order and change the status to something else, concurrently. This is known as a read-write or unrepeatable-read conflict which is prevented by serializable isolation. As a result, there will be a SQL exception and a rollback.

When this happens, the retry mechanism will kick in and retry the failed transaction. It will then succeed since the two transactions are no longer conflicting since one of them was committed successfully.

To observe this predictably we'll use two separate sessions with a controllable delay between the read and write operations.

Overview of the SQL operations executed (what the service will execute):

BEGIN; -- T1
SELECT * FROM purchase_order WHERE id=1; -- T1 
-- T1: Assert that status is `PLACED`
-- T1: Suspend for 15s  
BEGIN; -- T2
SELECT * FROM purchase_order WHERE id=1; -- T2
-- Assert that status is still `PLACED`
UPDATE purchase_order SET order_status='PAID' WHERE id=1; -- T2 
COMMIT; -- T2 (OK)
UPDATE purchase_order SET order_status='CONFIRMED' WHERE id=1; -- T1 (ERROR!)
ROLLBACK; -- T1

Now prepare the two separate shell windows so you can run the commands concurrently.

First, check that the order with ID 1 exists and has the status PLACED (or anything else other than CONFIRMED)

curl http://localhost:8090/api/order/1

Now let's run the first transaction (T1) where there is a simulated 15-sec delay before the commit (you can increase/decrease the time):

curl http://localhost:8090/api/order/1?status=CONFIRMED\&delay=15000 -i -X PUT

In less than 15 sec and before T1 commits, run the second transaction (T2) from another session which doesn't wait and succeeds with a commit:

curl http://localhost:8090/api/order/1?status=PAID -i -X PUT

At this point, T1 has no other choice than to rollback and that will trigger a retry:

ERROR: restart transaction: TransactionRetryWithProtoRefreshError: WriteTooOldError: write for key /Table/109/1/12/0 at timestamp 1669990868.355588000,0 too old; wrote at 1669990868.778375000,3: "sql txn" meta={id=92409d02 key=/Table/109/1/12/0 pri=0.03022202 epo=0 ts=1669990868.778375000,3 min=1669990868.355588000,0 seq=0} lock=true stat=PENDING rts=1669990868.355588000,0 wto=false gul=1669990868.855588000,0

The retry mechanism will catch that SQL exception, back off for a few hundred millis and then retry until it eventually succeeds (1 attempt).

The expected outcome is a 200 OK returned to both client sessions. The final order status must be CONFIRMED since client 1 request (T1) was retried and eventually committed, thereby overwriting T2.

curl http://localhost:8090/api/order/1

Conclusion

In this tutorial, we explore using Spring Retry, a library for retrying failed method invocations of a transient nature, to handle serialization conflict errors denoted by the SQL state code 40001. We cover how to set up Maven, configure Spring Retry, create a sample service, and demonstrate a retry scenario using a demo project.

Problem Statement

Let's begin by describing what the service does by using a problem statement. A problem statement specifies the system requirements at a high level. Input from business or product owners is critical in composing this statement.

The main characteristics include:

• Uses business domain language.

• Has clear sentences without jargon.

• Describes the project scope.

• Specifies the context of the business capability.

• Specifies the users/actors of the system.

• Specifies known business and technical constraints that are important to consider.

• Could serve as the foundation for identifying candidate domain objects (picking nouns).

Example:

The business requires an accounting system to keep track of monetary transactions. Users of the system are internal components that need to manage financial transactions between accounts.

The system must keep track of monetary accounts, transaction history on those accounts and account balances. Each account is associated with an account owner and a base currency. A transaction is the outcome of moving funds between different accounts. A transaction contains several account legs that may involve accounts with different currencies.

The safety mechanism used is the double-entry bookkeeping principle where each transaction must have a zero balance sum of all legs with the same currency. The system must also be able to produce reports of account activities and transactions for external auditors.

Problem statements are very useful when creating new components and for understanding the purpose and meaning of existing ones. Like for this hypothetical accounting ledger.

Architectural Mechanisms

Moving on from the problem domain to the solution domain. Software architectures can be visualized using UML diagrams or the https://c4model.com/ that I find quite useful. But what if you don't have diagrams or don't fancy drawing them?

One approach is to analyze what key architectural mechanisms are needed to implement all the features. Mechanisms are abstractions so when refining these into usable components or tools, you effectively do technology selection appropriate for the business domain (and organisation).

An architectural mechanism represents a common solution to a frequently encountered architectural problem that is not specific to a project or business domain. Quite similar to design patterns.

Implementation mechanisms are typically selected from a technology baseline within a tech organisation. For instance, when persistence is needed (analysis mechanism) and ACID properties are required, then an RDBMS (design mechanism) should be used where CockroachDB (implementation mechanism) is a great choice.

Key architectural mechanisms and realizations in this accounting ledger:

Design Overview

The ledger is based on a common Spring Boot microservice stack using Spring Boot, Spring Data JDBC/JPA, Spring Hateoas, HikariCP, Flyway and more. Kafka as CDC sink is optional for driving account balance push events to the web front-end. The default is just to send the push events after each successful transaction (with an AOP after-advice).

There are two distinct data access implementations; JPA via Hibernate and plain JDBC. Both are included in a single self-contained executable JAR artifact with an embedded Jetty servlet container. It's possible to configure the retry strategy, data access strategy and more through Spring profiles.

It connects to either a CockroachDB cluster or PostgreSQL. When using PostgreSQL, some features are disabled such as follower reads and geo-partitioning.

The bank client issues concurrent requests towards the service API endpoint which in turn reads and writes to the database. When a transfer request is processed, the outcome is a permanent record in history (the ledger) and balance updates on the affected accounts. These balance updates are also pushed to the frontends via websocket STOMP events for visualization.

Architecture Diagram

Entity Model

The system uses the following entity model for double-entry bookkeeping of monetary transaction history.

• account - Accounts with a derived balance from the sum of transactions

• transaction - Owning entity for balanced multi-legged monetary transactions

• transaction_item - Association table between transaction and account representing a single leg with a running account balance

• region - Static information about deployment regions

• outbox - Optional table for showcasing outbox pattern

Main SQL files

Flyway is used to set up the DB schema and account plan during startup time. The schema is not geo-partitioned by default.

Transaction Workflow

Each monetary transaction creates a transaction record (1) and one leg (2) for each account update and also updates the cached balance on each account (3). A CHECK constraint ensures that balances don't end up negative unless allowed for that account (using allow_negative column).

The UPDATE .. FROM (below) with array unnesting is a workaround for the lack of batch updates over the wire. The pgJDBC driver doesn't batch UPDATE statements, only INSERTs up to a given limit using SQL rewrites (aka multi-value inserts).

In the default workflow, the initial balance check on the accounts is redundant since the invariant check is done by looking at the rows affected on the final UPDATE. An UPDATE also takes an implicit lock in the reading part in CockroachDB (configurable) which will reduce retries. In summary, there are no reads involved in the default workflow (not counting the internal read that is part of each UPDATE).

-- (1) header
INSERT INTO transaction (id,city,balance,currency,name,..);
-- (2) for each leg (batch)
INSERT INTO transaction_item (city,transaction_id,..);
-- (3) for each account (batch)
UPDATE account SET balance = account.balance + data_table.balance, updated=clock_timestamp()
FROM (select unnest(?) as id, unnest(?) as balance) as data_table
WHERE account.id=data_table.id
  AND account.closed=false
  AND (account.balance + data_table.balance) * abs(account.allow_negative-1) >= 0

The actual CHECK constraints:

alter table account
    add constraint check_account_allow_negative check (allow_negative between 0 and 1);
alter table account
    add constraint check_account_positive_balance check (balance * abs(allow_negative - 1) >= 0);

Transaction Workflow (alternative)

The default workflow yields low contention. There's an alternative workflow designed to provoke more contention to visualize the effects of transient rollback errors and retries. The alternative workflow will perform an initial balance check and optionally use select-for-update (SFU) locks. Without the SFUs, the chance of contention is high. The main benefit of this workflow is that the running balance of the accounts can be stored on the transaction legs.

It's enabled by starting the server with the --roachbank.updateRunningBalance=true option.

-- (1) initial query for all involved accounts (lock is optional)
SELECT .. FROM account WHERE id IN (..) AND city IN (..) /* FOR UPDATE */;
-- (2) header 
INSERT INTO transaction (id,city,balance,currency,name,..);
-- (3) for each leg (notice running_balance)
INSERT INTO transaction_item (city,transaction_id,running_balance,..);
-- (4) for each account (batch)
UPDATE account SET balance = account.balance + data_table.balance, updated=clock_timestamp()
FROM (select unnest(?) as id, unnest(?) as balance) as data_table
WHERE account.id=data_table.id
  AND account.closed=false
  AND (account.balance + data_table.balance) * abs(account.allow_negative-1) >= 0

Transaction Retry Strategy

Any database running in serializable (such as CockroachDB) is exposed to transient SQL errors on contended workloads. These errors are tagged with SQL state 40001 and can be safely retried.

The ledger has three main strategies for performing retries:

• Client-side retries using a Spring/AspectJ AOP "around advice" with exponential backoff.

• JDBC driver level retries using the CockroachDB JDBC Driver

• No retries where transient SQL errors propagate to the client.

The default is client-side retries.

For more details on pros/cons with these different retry strategies, see:

• https://github.com/cockroachlabs-field/cockroachdb-jdbc

• https://github.com/cockroachlabs-field/spring-data-cockroachdb

APIs

The ledger provides two main interfaces:

• A hypermedia/REST API for request/response-based interactions based on Spring HATEOAS. The shell client (based on Spring Shell) interacts with the ledger through this API.

• A WebSocket Streaming API for reactive front-ends, driven via CockroachDB CDC (optional) or synthetic events.

The Hypermedia API is used to view data, create accounts and generate monetary transactions. A typical HTTP client follows the hyperlinks provided by the API to guide through different workflows, such as placing a monetary transaction or browsing through pages of account details. As with any REST API, following hyperlinks is optional. A client can also bind directly to the resource URI:s with tight coupling as a result. The semantics of the endpoints are tied to the link relations rather than the opaque URI:s.

• https://en.wikipedia.org/wiki/Hypertext_Application_Language

• https://en.wikipedia.org/wiki/HATEOAS

Conclusion

Roach Bank is a financial accounting ledger demo running on CockroachDB and PostgreSQL. It uses an entity model for double-entry bookkeeping and provides two distinct data access implementations. This article discusses an alternative transaction workflow with a balance check and retry strategy for databases running in serializable mode. The retry strategy is handled via Spring/CGLIB proxies with exponential backoff.

Cloud Deployment

The ledger provides a few convenience scripts for deploying to AWS, GCE and Azure using an internal tool called roachprod. This tool is free to use but at your own risk.

The provided scripts will do the following:

• Provision a single-region or multi-region CockroachDB cluster

• Deploy HAProxy on all client nodes

• Deploy bank server and client JAR on all client nodes

• Start the bank server on all client nodes

• Enable regional-by-row and global tables for multi-region (if needed)

Prerequisites

• Roachprod - a Cockroach Labs internal tool for ramping AWS/GCE/Azure VM clusters

• You will need the AWS/GCE/AZ client SDK and an account.

Deployment Scripts

AWS Deployment

The $basedir/deploy/aws folder contains a few scripts for provisioning different cluster sizes in different regions. Let's look at the aws-multiregion-eu.sh which is a multi-region configuration spanning eu-west-1, eu-west-2 and eu-central-1.

#!/bin/bash
# Script for setting up a multi-region Roach Bank cluster using roachprod in either AWS or GCE.

# Configuration
########################

title="CockroachDB 3-region EU deployment"
# CRDB release version
releaseversion="v22.2.5"
# Number of node instances in total including clients
nodes="12"
# Nodes hosting CRDB
crdbnodes="1-9"
# Array of client nodes (must match size of regions)
clients=(10 11 12)
# Array of regions localities (must match zone names)
regions=('eu-west-1' 'eu-west-2' 'eu-central-1')
# AWS/GCE cloud (aws|gce)
cloud="aws"
# AWS/GCE region zones (must align with nodes count)
zones="\
eu-west-1a,\
eu-west-1b,\
eu-west-1c,\
eu-west-2a,\
eu-west-2b,\
eu-west-2c,\
eu-central-1a,\
eu-central-1b,\
eu-central-1c,\
eu-west-1a,\
eu-west-2a,\
eu-central-1a"
# AWS/GCE machine types
machinetypes="c5d.4xlarge"

# DO NOT EDIT BELOW THIS LINE
#############################

functionsdir="../common"

source "${functionsdir}/core_functions.sh"

main.sh

By the end of running this script, you would have an AWS provisioned 12-node (instances) cluster, out of which the nodes 10, 11 and 12 are hosting the bank application stack including HAProxy.

Something like in this diagram:

The setup script is interactive and each step will ask for confirmation. It's launched by this simple command:

./aws-multiregion-eu.sh

After the steps are completed, you should have a page automatically opened in your default browser along with the service landing page showing account boxes.

GCE Deployment

For GGE, the process is quite similar just using different regions and instance types.

For example:

cd deploy/gce
chmod +x *.sh
./gce-multiregion-eu.sh

Azure Deployment

The same goes for Azure, however, it only contains a single region provisioning script.

cd deploy/azure
chmod +x *.sh
./azure-singleregion.sh

Operating in Multi-Region

When the ledger is deployed in a multi-regional topology (like US-EU-APAC), the accounts and transactions need to be pinned/domiciled to each region for best performance.

This is done by using the regional-by-row table locality in CockroachDB. There's an explicit step in the setup script than executes the SQL statements below. This will provide for low read and write latencies in each region.

For the AWS multi-region example:

ALTER DATABASE roach_bank PRIMARY REGION "eu-central-1";
ALTER DATABASE roach_bank ADD REGION "eu-west-1";
ALTER DATABASE roach_bank ADD REGION "eu-west-2";

ALTER TABLE region SET locality GLOBAL;

ALTER TABLE account ADD COLUMN region crdb_internal_region AS (
    CASE
        WHEN city IN ('dublin','belfast','liverpool','manchester','glasgow') THEN 'eu-west-1'
        WHEN city IN ('london','birmingham','leeds','amsterdam','rotterdam','antwerp','hague','ghent','brussels') THEN 'eu-west-2'
        WHEN city IN ('berlin','hamburg','munich','frankfurt','dusseldorf','leipzig','dortmund','essen','stuttgart','stockholm','copenhagen','helsinki','oslo','riga','tallinn') THEN 'eu-central-1'
        ELSE 'eu-central-1'
        END
    ) STORED NOT NULL;

ALTER TABLE account SET LOCALITY REGIONAL BY ROW AS region;

ALTER TABLE transaction ADD COLUMN region crdb_internal_region AS (
    CASE
        WHEN city IN ('dublin','belfast','liverpool','manchester','glasgow') THEN 'eu-west-1'
        WHEN city IN ('london','birmingham','leeds','amsterdam','rotterdam','antwerp','hague','ghent','brussels') THEN 'eu-west-2'
        WHEN city IN ('berlin','hamburg','munich','frankfurt','dusseldorf','leipzig','dortmund','essen','stuttgart','stockholm','copenhagen','helsinki','oslo','riga','tallinn') THEN 'eu-central-1'
        ELSE 'eu-central-1'
        END
    ) STORED NOT NULL;

ALTER TABLE transaction SET LOCALITY REGIONAL BY ROW AS region;

ALTER TABLE transaction_item ADD COLUMN region crdb_internal_region AS (
    CASE
        WHEN city IN ('dublin','belfast','liverpool','manchester','glasgow') THEN 'eu-west-1'
        WHEN city IN ('london','birmingham','leeds','amsterdam','rotterdam','antwerp','hague','ghent','brussels') THEN 'eu-west-2'
        WHEN city IN ('berlin','hamburg','munich','frankfurt','dusseldorf','leipzig','dortmund','essen','stuttgart','stockholm','copenhagen','helsinki','oslo','riga','tallinn') THEN 'eu-central-1'
        ELSE 'eu-central-1'
        END
    ) STORED NOT NULL;

ALTER TABLE transaction_item SET LOCALITY REGIONAL BY ROW AS region;

When transactions are issued against accounts in these different cities, the read-and-write operations will be constrained to the home regions. For example, creating monetary transactions involving accounts in "stockholm" and "helsinki" will be serviced only by the 3 nodes in the region eu-central-1. Read operations will have local latency and write operations will have only one single roundtrip to the next closest region.

As an option to limit the amount of data and overhead of replicating cross regions, you could disable the non-voting replicas with placement restrictions. This would result in no replicas placed outside of the home regions with the consequence of higher latency for follower reads in the other regions.

The tradeoff is regional survival. To combine both regional survival and data domiciling with regional-by-row, you can use super-regions which is covered more in this post.

SET enable_multiregion_placement_policy=on;
ALTER DATABASE roach_bank PLACEMENT RESTRICTED;

Running a Global Workload

First, SSH to the first client machine which in the AWS example is node (aka instance) 10.

roachprod run:$CLUSTER:10

Next, start the bank client and type connect. It should print something like this:

                                             C O C K R O A C H D B
──▄──▄────▄▀        ___                __     ___            __
───▀▄─█─▄▀▄▄▄      / _ \___  ___ _____/ /    / _ )___ ____  / /__
▄██▄████▄██▄▀█▄   / , _/ _ \/ _ `/ __/ _ \  / _  / _ `/ _ \/  '_/
─▀▀─█▀█▀▄▀███▀   /_/|_|\___/\_,_/\__/_//_/ /____/\_,_/_//_/_/\_\
──▄▄▀─█──▀▄▄     bank-client (v2.0.1.BUILD-SNAPSHOT) powered by Spring Boot (v3.0.4)
                 Active profiles: ${spring.profiles.active}
15:30:37.219  INFO [main] [io.roach.bank.client.ClientApplication] Starting ClientApplication v2.0.1.BUILD-SNAPSHOT using Java 17.0.6 with PID 10267 (/home/ubuntu/bank-client.jar started by ubuntu in /home/ubuntu)
15:30:37.220  INFO [main] [io.roach.bank.client.ClientApplication] No active profile set, falling back to 1 default profile: "default"
15:30:38.414  INFO [main] [io.roach.bank.client.ClientApplication] Started ClientApplication in 1.564 seconds (process running for 2.095)
disconnected:$ connect
15:30:42.949  INFO [main] [io.roach.bank.client.command.Connect] Connecting to http://localhost:8090/api..
15:30:43.084  INFO [main] [io.roach.bank.client.command.Connect] Welcome to text-only Roach Bank. You are in a dark, cold lobby.
15:30:43.084  INFO [main] [io.roach.bank.client.command.Connect] Type help for commands.
localhost:$

Next, let's run some account transfers across the cities in the local region. First, we need to verify that the local gateway region is eu-west-1:

localhost:$ gateway-region
eu-west-1

Then start the transfers:

localhost:$ transfer --regions eu-west-1

If you look in the browser tab pointing at the regional bank service, you should see some effects on the accounts in that region. If you are not sure about the URL, use roachprod ip:

roachprod ip $CLUSTER:10-12 --external

Pick the first IP and append port 8090 and you should see:

Note: For simplicity, the push events that update the balances are not broadcasted across regions, so you can only see effects at a regional level.

The transfer command runs with a very low volume by default but it can be ramped up with more concurrent threads and a higher selection of accounts to avoid contention. The low amount range reduce the risk of ending up with a negative balance causing aborts.

transfer --regions eu-west-1 --concurrency 10 --limit 1000 --amount 0.01-0.15

To run a 100% read-based workload we can use the balance command. This will start then concurrent workers per city in the given region and run point lookups.

balance --regions eu-west-1 --followerReads --concurrency 10 --limit 1000

Lastly, repeat the steps above for client nodes 11 and 12 so you end up with 3 concurrent clients and servers, one pair per region.

roachprod run:$CLUSTER:11
..

Once the workloads run at full speed, you should see metrics picking up in the DB Console.

As we can see in the hardware dashboard, the vCPU utilization starts reaching the 50% threshold. Using these 16vCPU VMs, we get around 40K QPS at less than 2ms on P99. Keep in mind the cluster stretches across 3 regions in EU.

Summary

This article provides instructions on how to deploy the RoachBank accounting ledger demo on a multi-regional CockroachDB cluster, including instructions for deploying on AWS, GCE and Azure, setting up regional-by-row and global tables, and using the roachprod tool. Prerequisites include the AWS/GCE/AZ client SDK and an account.

Introduction

The concept behind the ledger is to move funds between monetary accounts using balanced, multi-legged transactions, at a high frequency. As a financial system, correctness is defined as conserving money at all times and providing an audit trail of monetary transactions performed towards the accounts. Put simply, when externally observing the system, the total account balance must be constant at all times. Funds are simply moved between different accounts using balanced transactions.

This is visualized by the service using a single page to display accounts as rectangles with their current balance.

Key Invariants

There are a few business rule invariants that must hold at all times regardless of observer and activities. Such as infrastructure failure (nodes crashing) or conflicting operations when concurrently updating the same accounts.

• The total balance of all accounts must be constant.

• User accounts must have a positive balance (account types that disallow negative balance).

• An audit trail of all transactions must be stored from which the account balances can be derived.

The system must refuse forward progress if an operation would result in any of these invariants being compromised. For example, if a variation of the total balances is observed at any given time, then money has either been "invented" or "destroyed".

These invariants are safeguarded by ACID guarantees and real serializable transactions. CockroachDB defaults to only serializable while PostgreSQL defaults to read-committed but can be elevated to serializable-snapshot or SSI.

Double-entry Bookkeeping

To satisfy the audit trail requirement, the ledger follows the double-entry bookkeeping principle. This principle was originally formalized and published by the Italian mathematician Luca Pacioli during the 15th century.

It involves making at least two account entries for every transaction. A debit in one account and a corresponding credit in another account. The sum of all debits must equal the sum of all credits, providing a simple method for error detection. Real accounting doesn't use negative numbers, but for simplicity, this ledger does (it's not about modelling the true complexity of accounting).

A positive value means increasing the value (credit), and a negative value means decreasing the value (debit). A transaction is considered balanced when the sum of the legs with the same currency equals zero.

In the following example, there are four different accounts involved with zero-sum in the end.

Account | Credit(+) | Debit(-) |
A         100               
B                     -50
C          25
D*                    -25 \
                           -75 (coalesced)
D*                    -50 /
------------------------------------------
Σ         125    +   -125 = 0

Building

Prerequisites

• Java 17

  • https://openjdk.org/projects/jdk/17/

  • https://www.oracle.com/java/technologies/downloads/#java17

• Maven 3+

  • https://maven.apache.org/

The service is built with Maven 3.1+. Tanuki's Maven wrapper is included (mvnw) so Maven is optional. All 3rd party dependencies are available in public Maven repositories except for the CockroachDB JDBC driver which is available in GitHub Packages (you only need a GitHub account).

These dependencies are available in GitHub packages:

<dependency>
    <groupId>io.cockroachdb.jdbc</groupId>
    <artifactId>cockroachdb-jdbc-driver</artifactId>
    <version>1.0.0</version>
</dependency>
<dependency>
    <groupId>io.cockroachdb</groupId>
    <artifactId>spring-data-cockroachdb</artifactId>
    <version>1.0.0</version>
</dependency>

To allow Maven to use this repository, add a github profile (or similar) to your Maven settings.xml file. Edit $user.dir/.m2/settings.xml.

An example is provided below:

<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.2.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.2.0 https://maven.apache.org/xsd/settings-1.2.0.xsd">
  <servers>
    <server>
        <id>github</id>
        <username>your-github-id</username>
        <password>your-personal-access-token</password>
    </server>
  </servers>

  <mirrors>
    <!-- default setting -->
    <mirror>
      <id>maven-default-http-blocker</id>
      <mirrorOf>external:http:*</mirrorOf>
      <name>Pseudo repository to mirror external repositories initially using HTTP.</name>
      <url>http://0.0.0.0/</url>
      <blocked>true</blocked>
    </mirror>
  </mirrors>

  <profiles>
      <profile>
        <id>github</id>
        <repositories>
            <repository>
                <id>central</id>
                <url>https://repo1.maven.org/maven2</url>
            </repository>
            <repository>
                <id>github</id>
                <url>https://maven.pkg.github.com/cockroachlabs-field/*</url>
                <snapshots>
                    <enabled>true</enabled>
                </snapshots>
                <releases>
                    <enabled>true</enabled>
                </releases>
            </repository>
        </repositories>
      </profile>
  </profiles>

  <activeProfiles>
    <activeProfile>github</activeProfile>
  </activeProfiles>
</settings>

Clone the project

git clone git@github.com:kai-niemi/roach-bank.git

Build the executable jars

Using installed Maven:

cd roach-bank 
chmod +x mvnw 
mvn clean install

Using the Maven wrapper (where you need to specify settings.xml):

cd roach-bank 
chmod +x mvnw 
./mvnw clean install -s <path-to>/settings.xml

Local Deployment

Assuming you already have a local CockrochDB cluster running.

First, create the database:

cockroach sql --insecure --host=localhost -e "CREATE database roach_bank"

Then start the server:

java -jar bank-server/target/bank-server.jar

Then start the client:

java -jar bank-client/target/bank-client.jar

The client is used to issue business transactions to the server's REST API. The client and server could be on separate hosts with an L7 load balancer in-between, but for convenience, the client connects to localhost by default.

Next Steps

In the second part of this series, we'll cover how to run the bank against a multi-regional cloud deployment. The third part goes into design details and the technology stack.

Conclusion

RoachBank is a full-stack financial accounting ledger demo running on both CockroachDB and PostgreSQL. It follows the double-entry bookkeeping principle and is designed to demonstrate the safety and liveness properties of a globally deployed, system-of-record type of financial workload. This article provides instructions on how to set up and run the demo, as well as details on the technology stack and design.

Article series on the JDBC driver:

• https://blog.cloudneutral.se/series/cockroachdb-jdbc-driver

Introduction

This article will highlight one specific optimization feature, namely, batch DML rewrites for bulk operations using FROM clause with array unnesting. It's a mouthful, but conceptually it's a transparent rewrite at the driver level of batch DML updates like:

UPDATE product SET inventory=?, price=? WHERE id=1
UPDATE product SET inventory=?, price=? WHERE id=2
UPDATE product SET inventory=?, price=? WHERE id=3
...

Which can be collapsed to a single update statement using FROM:

UPDATE product SET inventory=data_table.new_inventory, price=data_table.new_price 
FROM (select unnest(?) as id, unnest(?) as new_inventory, unnest(?) as new_price) as data_table WHERE product.id=data_table.id

This will dramatically improve the performance for bulk operations that send a series of non-aggregated UPDATEs to independent rows. It's also not limited to UPDATEs either and can also rewrite INSERT and UPSERT statements. This may also improve performance since it allows for higher batch sizes than 128 which is the soft limit in the pgJDBC driver for rewriting INSERTs.

Consider the following example schema:

create table if not exists product
(
    id        uuid           not null default gen_random_uuid(),
    inventory int            not null,
    name      varchar(128)   not null,
    price     numeric(19, 2) not null,
    sku       varchar(128)   not null unique,
    primary key (id)
);

Next, let's add a few rows:

insert into product (inventory,name,price,sku)
values (10, 'A', 12.50, gen_random_uuid()),
       (10, 'B', 13.50, gen_random_uuid()),
       (10, 'C', 14.50, gen_random_uuid()),
       (10, 'D', 15.50, gen_random_uuid());

select inventory,name,price,sku from product order by name;

The listed result is equivalent to the next statement, the only difference being its temp data:

select unnest(ARRAY[10, 10, 10, 10]) as inventory,
       unnest(ARRAY['A', 'B', 'C', 'D']) as name,
       unnest(ARRAY[12.50, 13.50, 14.50, 15.50]) as price,
       unnest(ARRAY[gen_random_uuid(),
                    gen_random_uuid(),
                    gen_random_uuid(),
                    gen_random_uuid()]) as sku
order by name;

The unnest function uses arrays to generate a temporary table with each array representing a column. Now, let's flip this over to INSERT into FROM:

insert into product (inventory,name,price,sku)
(select unnest(ARRAY[10, 10, 10, 10]) as inventory,
        unnest(ARRAY['A', 'B', 'C', 'D']) as name,
        unnest(ARRAY[12.50, 13.50, 14.50, 15.50]) as price,
        unnest(ARRAY[gen_random_uuid(),
                     gen_random_uuid(),
                     gen_random_uuid(),
                     gen_random_uuid()]) as sku);

That will create four new products out of the contents of the arrays. When put into a JDBC-prepared statement context:

try (PreparedStatement ps = connection.prepareStatement(
        "INSERT INTO product(id,inventory,price,name,sku)"
                + " select"
                + "  unnest(?) as id,"
                + "  unnest(?) as inventory, unnest(?) as price,"
                + "  unnest(?) as name, unnest(?) as sku")) {
    // chunks is a segmented stream of products
    chunks.forEach(chunk -> {
        List<Integer> qty = new ArrayList<>();
        List<BigDecimal> price = new ArrayList<>();
        List<UUID> ids = new ArrayList<>();
        List<String> name = new ArrayList<>();
        List<String> sku = new ArrayList<>();

        chunk.forEach(product -> {
            ids.add(product.getId());
            qty.add(product.getInventory());
            price.add(product.getPrice());
            name.add(product.getName());
            sku.add(product.getSku());
        });

        try {
            ps.setArray(1, ps.getConnection().createArrayOf("UUID", ids.toArray()));
            ps.setArray(2, ps.getConnection().createArrayOf("BIGINT", qty.toArray()));
            ps.setArray(3, ps.getConnection().createArrayOf("DECIMAL", price.toArray()));
            ps.setArray(4, ps.getConnection().createArrayOf("VARCHAR", name.toArray()));
            ps.setArray(5, ps.getConnection().createArrayOf("VARCHAR", sku.toArray()));

            ps.executeLargeUpdate();
        } catch (SQLException e) {
            throw new RuntimeException(e);
        }
    });
}

This is technically equivalent to using addBatch() or executeLargeBatch() with the important difference that it requires the pgJDBC driver's reWriteBatchedInserts to be set to true.

From a performance standpoint, both approaches are equivalent up to a certain point which is a batch size of 128, which is the hardcoded limit in the pgJDBC driver. Using batch sizes higher than that is not possible so the array unnesting approach is more performant beyond this limit. Depending on the workload you can go up to a batch number of around 16-32K until performance starts to level out again.

The hardcoded limit of 128 may be appropriate for PostgreSQL but CockroachDB is not PostgreSQL (only at the wire protocol level) and can leverage much higher bulk statement sizes. Until this limit is removed or made configurable in pgJDBC, there are only two options:

• Modify the pgJDBC driver and built a custom library. This is quite straightforward but requires a separate forked version to be maintained.

• Rewrite INSERTs, UPSERTs and UPDATEs with unnesting of arrays

Bulk Inserts

To recap, here's an UPSERT example that hits the pgJDBC driver 128 size limit on INSERT rewrites.

List<Product> products = Arrays.asList(
        Product.builder().withName("A").withInventory(1).withPrice(new BigDecimal("10.15")).build(),
        Product.builder().withName("B").withInventory(2).withPrice(new BigDecimal("11.15")).build(),
        Product.builder().withName("C").withInventory(3).withPrice(new BigDecimal("12.15")).build()
        // .. etc to several 1000s
);

Stream<List<Product>> chunks = chunkedStream(products.stream(), 128);

dataSource.addDataSourceProperty("reWriteBatchedInserts",true);

try (Connection connection = dataSource.getConnection()) {
    connection.setAutoCommit(true);

    chunks.forEach(chunk -> {
        try (PreparedStatement ps = connection.prepareStatement(
                "INSERT INTO product (id,inventory,price,name,sku) values (?,?,?,?,?) ON CONFLICT (id) DO NOTHING")) {
            for (Product product : chunk) {
                ps.setObject(1, product.getId());
                ps.setObject(2, product.getInventory());
                ps.setObject(3, product.getPrice());
                ps.setObject(4, product.getName());
                ps.setObject(5, product.getSku());
                ps.addBatch();
            }
            ps.executeBatch();
        } catch (SQLException ex) {
            throw new RuntimeException(ex);
        }
    });
}

For completeness, the chunkedStream method which just slices up a stream into even chunks:

    public static <T> Stream<List<T>> chunkedStream(Stream<T> stream, int chunkSize) {
        AtomicInteger idx = new AtomicInteger();
        return stream.collect(Collectors.groupingBy(x -> idx.getAndIncrement() / chunkSize)).values().stream();
    }

This UPSERT statement is executed using implicit transactions and it's fairly fast with the reWriteBatchedInserts property. The pgJDBC rewrite feature works both for regular INSERTs and INSERT .. ON CONFLICT, aka UPSERTs.

Bulk Updates

Given the previous example, it's fair to assume UPDATEs work in the same way:

try (PreparedStatement ps = connection.prepareStatement(
    "UPDATE product SET inventory=?, price=? WHERE id=?")) {

    chunk.forEach(product -> {
        try {
            ps.setInt(1, product.getInventory());
            ps.setBigDecimal(2, product.getPrice());
            ps.setObject(3, product.getId());
            ps.addBatch();
        } catch (SQLException e) {
            throw new RuntimeException(e);
        }
    });
    ps.executeLargeBatch();  
} catch (SQLException ex) {
    throw new RuntimeException(ex);
}

However, there's no batching done here whatsoever at the JDBC driver level. To apply individual row UPDATEs in bulk format, you can however use arrays again:

try (PreparedStatement ps = connection.prepareStatement(
        "UPDATE product SET inventory=data_table.new_inventory, price=data_table.new_price "
                + "FROM (select "
                + "unnest(?) as id, "
                + "unnest(?) as new_inventory, "
                + "unnest(?) as new_price) as data_table "
                + "WHERE product.id=data_table.id")) {
    List<Integer> qty = new ArrayList<>();
    List<BigDecimal> price = new ArrayList<>();
    List<UUID> ids = new ArrayList<>();

    chunk.forEach(product -> {
        qty.add(product.addInventoryQuantity(1));
        price.add(product.getPrice().add(new BigDecimal("1.00")));
        ids.add(product.getId());
    });

    ps.setArray(1, ps.getConnection()
            .createArrayOf("UUID", ids.toArray()));
    ps.setArray(2, ps.getConnection()
            .createArrayOf("BIGINT", qty.toArray()));
    ps.setArray(3, ps.getConnection()
            .createArrayOf("DECIMAL", price.toArray()));

    ps.executeLargeUpdate(); 
} catch (SQLException e) {
    throw new RuntimeException(e);
}

The performance improvement with this approach is monumental. The only problem is that it's rather clunky and requires code refactoring.

The CockroachDB JDBC driver can however rewrite bulk DML operations on behalf of applications, which makes it transparent. See the github repo https://github.com/cloudneutral/cockroachdb-jdbc for more details.

Conclusion

This article discusses a feature of the CockroachDB JDBC driver to optimize batch updates with array unnesting, allowing for much larger batch sizes than the pgJDBC driver. It also considers the performance improvement of this approach and the code refactoring required. This approach can be used for both INSERTs and UPSERTs, as well as individual row UPDATEs.

The primary goal of the Spring Data project is to make it easier to build Spring-powered applications that use new data access technologies such as relational databases, non-relational databases, map-reduce frameworks, and cloud-based data services.

CockroachDB is a distributed SQL database built on a transactional and strongly-consistent key-value store. It scales horizontally; survives disk, machine, rack, and even datacenter failures with minimal latency disruption and no manual intervention; supports strongly-consistent ACID transactions; and provides a familiar SQL API for structuring, manipulating, and querying data.

Features

• Bundles the CockroachDB JDBC driver

• Meta-annotations for declaring:

  • Retryable transactions

  • Read-only transactions

  • Strong and stale follower-reads

  • Custom session variables including timeouts

• AOP aspects for:

  • Retrying transactions on serialization conflicts

  • Configuring session variables, like follower-reads

• Connection pool factory settings for HikariCP

• Datasource proxy logging via TTDDYY

• Simple JDBC shell client for ad-hoc queries and testing

Getting Started

Here is a quick teaser of an application using Spring Data JPA Repositories in Java:

@Repository
public interface AccountRepository extends JpaRepository<Account, UUID> {
    Optional<Account> findByName(String name);

    @Query(value = "select a.balance "
            + "from Account a "
            + "where a.id = ?1")
    BigDecimal findBalanceById(UUID id);

    @Query(value = "select a.balance "
            + "from account a AS OF SYSTEM TIME follower_read_timestamp() "
            + "where a.id = ?1", nativeQuery = true)
    BigDecimal findBalanceSnapshotById(UUID id);

    @Query(value = "select a "
            + "from Account a "
            + "where a.id in (?1)")
    @Lock(LockModeType.PESSIMISTIC_READ)
    List<Account> findAllForUpdate(Set<UUID> ids);
}

@Service
public class AccountService {
    @Autowired
    private AccountRepository accountRepository;

    @NotTransactional
    public Account create(Account account) {
        return accountRepository.save(account);
    }

    @NotTransactional
    public Account findByName(String name) {
        return accountRepository.findByName(name)
                .orElseThrow(() -> new ObjectRetrievalFailureException(Account.class, name));
    }

    @NotTransactional
    public Account findById(UUID id) {
        return accountRepository.findById(id).orElseThrow(() -> new ObjectRetrievalFailureException(Account.class, id));
    }

    @TransactionBoundary
    @Retryable
    public Account update(Account account) {
        Account accountProxy = accountRepository.getReferenceById(account.getId());
        accountProxy.setName(account.getName());
        accountProxy.setDescription(account.getDescription());
        accountProxy.setBalance(account.getBalance());
        accountProxy.setClosed(account.isClosed());
        return accountRepository.save(accountProxy);
    }

    @NotTransactional
    public BigDecimal getBalance(UUID id) {
        return accountRepository.findBalanceById(id);
    }

    @TransactionBoundary(timeTravel = @TimeTravel(mode = TimeTravelMode.FOLLOWER_READ), readOnly = true)
    public BigDecimal getBalanceSnapshot_Explicit(UUID id) {
        return accountRepository.findBalanceById(id);
    }

    @NotTransactional
    public BigDecimal getBalanceSnapshot_Implicit(UUID id) {
        return accountRepository.findBalanceSnapshotById(id);
    }

    @TransactionBoundary
    public void delete(UUID id) {
        accountRepository.deleteById(id);
    }

    @TransactionBoundary
    public void deleteAll() {
        accountRepository.deleteAll();
    }
}

@Configuration
@EnableTransactionManagement(order = AdvisorOrder.TRANSACTION_ADVISOR)
@EnableJpaRepositories(basePackages = {"org.acme.bank"})
public class BankApplication {
    @Bean
    public TransactionRetryAspect retryAspect() {
        return new TransactionRetryAspect();
    }

    @Bean
    public TransactionBoundaryAspect transactionBoundaryAspect(JdbcTemplate jdbcTemplate) {
        return new TransactionBoundaryAspect(jdbcTemplate);
    }
}

Maven configuration

Add this dependency to your pom.xml file:

<dependency>
    <groupId>io.cockroachdb</groupId>
    <artifactId>spring-data-cockroachdb</artifactId>
    <version>{version}</version>
</dependency>

Then add the Maven repository to your pom.xml file (alternatively in Maven's settings.xml):

<repository>
    <id>cockroachdb-jdbc</id>
    <name>Maven Packages</name>
    <url>https://maven.pkg.github.com/cloudneutral/cockroachdb-jdbc</url>
    <snapshots>
        <enabled>true</enabled>
    </snapshots>
</repository>

Finally, you need to authenticate to GitHub Packages by creating a personal access token (classic) that includes the read:packages scope. For more information, see Authenticating to GitHub Packages.

Add your personal access token to the servers section in your settings.xml:

<server>
    <id>github</id>
    <username>your-github-name</username>
    <password>your-access-token</password>
</server>

Now you should be able to build your project with the JDBC driver as a dependency:

mvn clean install

Alternatively, you can just clone the repository and build it locally using mvn install.

Modules

There are several modules in this project:

spring-data-cockroachdb

Provides a Spring Data module for CockroachDB, bundling the CockroachDB JDBC driver, connection pooling support via Hikari and meta-annotations and AOP aspects for client-side retry logic, as an alternative to JDBC driver level retries.

spring-data-cockroachdb-shell

An interactive spring shell client for ad-hoc SQL queries and CockroachDB settings and metadata introspection using the CockroachDB JDBC driver.

spring-data-cockroachdb-distribution

Distribution packaging of runnable artifacts including the shell client and JDBC driver, in tar.gz format. Activated via Maven profile, see build section further down in this page.

spring-data-cockroachdb-it

Integration and functional test harness. Activated via Maven profile, see build section further down in this page.

Building from Source

Spring Data CockroachDB requires Java 17 (or later) LTS.

Prerequisites

• JDK17+ LTS for building (OpenJDK compatible)

• Maven 3+ (optional, embedded)

If you want to build with the regular mvn command, you will need Maven v3.x or above.

Install the JDK (Linux):

sudo apt-get -qq install -y openjdk-17-jdk

Install the JDK (macOS):

brew install openjdk@17

Dependencies

This project depends on the CockroachDB JDBC driver whose artifacts are available in GitHub Packages.

Clone the project

git clone git@github.com:cloudneutral/spring-data-cockroachdb.git
cd spring-data-cockroachdb

Build the project

chmod +x mvnw
./mvnw clean install

If you want to build with the regular mvn command, you will need Maven v3.5.0 or above.

Build the distribution

./mvnw -P distribution clean install

The distribution tar.gz is now found in spring-data-cockroachdb-distribution/target.

Run Integration Tests

The integration tests will run through a series of contended workloads to exercise the retry mechanism and other JDBC driver features.

First, start a local CockroachDB node or cluster and create the database:

cockroach sql --insecure --host=localhost -e "CREATE database spring_data_test"

Then activate the integration test Maven profile:

./mvnw -P it clean install

See the pom.xml file for changing the database URL and other settings (under ìt profile).

Conclusion

The Spring Data CockroachDB project offers a Spring-based programming model for CockroachDB, a distributed SQL database. It simplifies building Spring-powered applications with new data access technologies and includes features like bundling the CockroachDB JDBC driver, meta-annotations for transactions, connection pooling support, and a shell client for ad-hoc queries. The project requires Java 17 or later and can be built using Maven.

Article series on the JDBC driver:

• https://blog.cloudneutral.se/series/cockroachdb-jdbc-driver

Overview

The CockroachDB JDBC driver wraps the PostgreSQL JDBC driver (pgjdbc) which must also be on the app's classpath. There are no other dependencies besides SLF4J for which any supported logging framework can be used.

It works by the JDBC Driver accepting a unique URL prefix jdbc:cockroachdb to separate itself from jdbc:postgressql. When the driver is asked to open a connection (typically by the connection pool), it passes the call forward to pgJDBC and then wraps the connection in a CockroachDB connection proxy with a custom interceptor (invocation handler). The driver delegates all calls to the underlying pgJDBC driver and does not interact directly with the database itself at any point.

Internal Retries

One of the driver features includes internal retries in contrast to application-side or client-side retries which is the common option. It works by the driver wrapping each JDBC connection, statement and result set in a dynamic proxy and interceptors capable of detecting and retrying aborted transactions, warranted that the SQL exceptions are of a qualified type. The qualifying exception types are of two main categories: Serialization Conflicts and Connection Errors.

Serialization Conflicts

The JDBC driver can optionally perform internal retries of failed transactions due to serialization conflicts denoted by the 40001 state code. Serialization conflict errors are safe to retry by the client, or in this case by the driver. Safe, in terms of not producing duplicate side effects since the transaction was rolled back.

This type of error is more likely to manifest in databases running with serializable transaction isolation (1SR), in particular for contended workloads subject to read/write and write/read conflicts.

There are however more limitations with driver-level retries than application-level. See the implementation section further below for more details.

Connection Errors

The JDBC driver can also perform internal retries on connection errors denoted by any of the 08001, 08003, 08004, 08006, 08007, 08S01 or 57P01 state codes. Connection errors during in-flight transactions are generally safe to retry, but there is a potential for duplicate side effects if the SQL operations performed are non-idempotent like INSERTs or UPDATEs with increment operations (UPDATE x set y=y-1).

This could happen for example if a transaction commit was successful but the response back to the client was lost due to a connection failure. In that case, the result is ambiguous and the driver can't tell if the transaction was successfully committed or rolled back.

Retry Implementation

Transaction conflicts and connection errors can surface at read, write and commit time which means there are retry interceptors wrapped around the following JDBC API artefacts:

• java.sql.Connection

  • implemented by CockroachConnection

  • proxied by ConnectionRetryInterceptor, retries on commit()

• java.sql.Statement

  • implemented by CockroachStatement

  • proxied by StatementRetryInterceptor, retries on write operations

• java.sql.PreparedStatement

  • implemented by CockroachPreparedStatement

  • proxied by PreparedStatementRetryInterceptor, retries on write operations

• java.sql.ResultSet

  • implemented by CockroachResultSet

  • proxied by ResultSetRetryInterceptor, retries on read operations

Retries are possible by recording most JDBC operations during an explicit transaction (autoCommit set to false). If a transaction is aborted due to a transient error it will be rolled back and the connection is closed. The recorded operations are then repeated on a new connection delegate while comparing the results against the initial transaction attempt.

If the results observed by the application client are in any way different (determined by SHA-256 checksums), the driver is forced to give up the retry attempt to preserve a serializable outcome towards the application, still waiting for completion.

To illustrate:

try (Connection connection 
        = DriverManager.getConnection("jdbc:cockroachdb://localhost:26257/jdbc_test?sslmode=disable") {
  try (PreparedStatement ps = connection.prepareStatement("update table set x = ? where id = ?")) {
        ps.setObject(1, x);
        ps.setObject(2, y);
        ps.executeUpdate();
  }
}

In this example, assume the executeUpdate() method throws a SQLException with state code 40001. This exception is caught by the retry interceptor which will roll back and close the current connection, then repeat the recorded operations on a new connection delegate and hope for a different interleaving of other concurrent operations that allow for the transaction to complete.

From the perspective of the application, the executeUpdate() operation will block until this process is either successful or considered futile, in which case a separate SQLException is thrown with the same state code.

Limitations of driver-level retries

By contrast, when using application-level retries you would typically need to apply retry logic. Something like the following:

int numCalls=1;
do {
  try {
      // Must begin and commit/rollback transactions
      return businessService.someTransactionBoundaryOperation(); 
  } catch (SQLException sqlException) { // Catch r/w and commit time exceptions
      // 40001 is the only state code we are looking for in terms of safe retries
      if (PSQLState.SERIALIZATION_FAILURE.getState().equals(sqlException.getSQLState())) {
          // handle by logging and waiting with an exponentially increasing delay
      } else {
          throw sqlException; // Some other error, re-throw instantly
      }
  }
} while (numCalls < MAX_RETRY_ATTEMPTS);

This type of logic fits well into an AOP aspect with an around advice (or interceptor in JavaEE), weaving in between the caller and transaction boundary (typically a service facade, service activator, or web/API controller).

Application-level retries always have a higher chance of success over driver-level because the application logic is applied in each repeat cycle. For example, if you are checking for a negative account balance in the app code, then it may cancel out additional writes based on the value read when the operation is repeated. Neither the JDBC driver nor the database has any visibility to the application logic, which means that a retry attempt can only succeed if all previously observed outcomes are identical to the new ones.

The practical use of driver-level retries is therefore more narrow for common read/write and write/read conflicts, in which case client-side retries are the preferred approach.

Implicit SELECT FOR UPDATE rewrites

The JDBC driver can optionally append a FOR UPDATE clause to qualified SELECT statements.

A SELECT query qualifies for a rewrite when:

• It's not part of a read-only connection

• There are no aggregate functions (max, min, avg, etc.)

• There are no distinct or GROUP BY operators

• There are no internal CockroachDB schema references

A SELECT .. FOR UPDATE will lock the rows returned by a selection query such that other transactions trying to access those rows are forced to wait for the transaction that locked the rows to finish. These other transactions are effectively put into a queue based on when they tried to read the value of the locked rows.

Notice that this does not eliminate the chance of serialization conflicts (which can also be due to time uncertainty) but will greatly reduce it. Combined with driver-level retries, this can eliminate the need for app-level retry logic for some workloads.

The following example shows a write skew (G2-item) scenario which is prevented by CockroachDB serializable isolation:

Running the same sequence with FOR UPDATE:

The initial read in T1 will lock the rows and T2 is forced to wait for T1 to finish. When T1 has finished with a commit, the read in T2 is reflecting the write of T1 and not that of T2 at the initial read timestamp. The T2 read is effectively pushed into the future with the desired effect of these operations resulting in a serializable transaction ordering, allowing for both to commit.

Sequence Diagrams

The driver concepts are illustrated with sequence diagrams using https://www.websequencediagrams.com.

Happy Path

This diagram illustrates executing a single update with a happy outcome, equivalent to:

try (Connection connection 
        = DriverManager.getConnection("jdbc:cockroachdb://localhost:26257/jdbc_test?sslmode=disable") {
  try (PreparedStatement ps = connection.prepareStatement("update table set x = ? where id = ?")) {
        ps.setObject(1, x);
        ps.setObject(2, y);
        ps.executeUpdate();
  }
}

Unhappy Path

This diagram illustrates executing the same single block with an unhappy outcome, equivalent to:

Conclusion

This article discusses the design and implementation of a custom-made CockroachDB JDBC driver, which wraps the PostgreSQL JDBC driver and provides features such as internal retries for serialization conflicts and connection errors. The driver also uses driver-level retries and SELECT FOR UPDATE rewrites to reduce the chance of serialization conflicts in a transaction. Sequence diagrams are provided to illustrate the process.

This article describes the recently released open-source JDBC driver for CockroachDB. It wraps the PostgreSQL JDBC driver (pgjdbc) which in turn communicates in the PostgreSQL native network wire (v3.0) protocol with CockroachDB.

Article series on the JDBC driver:

• https://blog.cloudneutral.se/series/cockroachdb-jdbc-driver

Features

This JDBC driver adds certain features on top of pgJDBC that are relevant to CockroachDB.

• Internal retries on serialization conflicts.

• Internal retries on connection errors.

• Rewriting qualified SQL queries to use SELECT FOR UPDATE to reduce serialization conflicts.

• CockroachDB-specific database metadata and version info.

All these features are disabled by default, which means the driver is operating in a pass-through mode delegating all JDBC API invocations to the pgJDBC driver.

Enabling internal retries may reduce the need for application-level retry logic and thereby enhance compatibility with 3rd-party products that don't implement any transaction retries.

Enabling SELECT FOR UPDATE rewrites may reduce serialization conflicts from appearing in the first place and thereby reduce retries to a bare minimum or none at all, at the expense of imposing locks on every read operation.

SELECT FOR UPDATE rewrites can be scope to connection level where all qualified SELECT queries are rewritten, or to transaction level where all qualified SELECT within a given transaction are rewritten.

For more information about client-side retry logic, see also:

• Transaction Contention

• Connection Retry Loop

Getting Started

Below is an example of creating a JDBC connection and executing a simple SELECT query in an implicit transaction (auto-commit):

try (Connection connection 
        = DriverManager.getConnection("jdbc:cockroachdb://localhost:26257/jdbc_test?sslmode=disable") {
  try (Statement statement = connection.createStatement()) {
    try (ResultSet rs = statement.executeQuery("select version()")) {
      if (rs.next()) {
        System.out.println(rs.getString(1));
      }
    }
  }
}

Next is an example of executing a SELECT and an UPDATE in an explicit transaction with FOR UPDATE rewrites:

try (Connection connection
             = DriverManager.getConnection("jdbc:cockroachdb://localhost:26257/jdbc_test?sslmode=disable")) {
    connection.setAutoCommit(false);

    try (Statement statement = connection.createStatement()) {
        statement.execute("SET implicitSelectForUpdate = true");
    }

    // Will be rewritten by the driver to include suffix "FOR UPDATE"
    try (PreparedStatement ps = connection.prepareStatement("select balance from account where id=?")) {
        ps.setLong(1, 100L);

        try (ResultSet rs = ps.executeQuery()) {
            if (rs.next()) {
                BigDecimal balance = rs.getBigDecimal(1); // check
                try (PreparedStatement ps2 = connection.prepareStatement("update account set balance = balance + ? where id=?")) {
                    ps2.setBigDecimal(1, new BigDecimal("10.50"));
                    ps2.setLong(2, 100L);
                    ps2.executeUpdate(); // check
                }
            }
        }
    }
    connection.commit();
}

Same as above where all qualified SELECTs are suffixed with FOR UPDATE:

try (Connection connection
             = DriverManager.getConnection("jdbc:cockroachdb://localhost:26257/jdbc_test?sslmode=disable&implicitSelectForUpdate=true")) {
    connection.setAutoCommit(false);
    ...
    connection.commit();
}

Maven configuration

Add this dependency to your pom.xml file:

<dependency>
    <groupId>io.cockroachdb.jdbc</groupId>
    <artifactId>cockroachdb-jdbc-driver</artifactId>
    <version>{version}</version>
</dependency>

Then add the Maven repository to your pom.xml file (alternatively in Maven's settings.xml):

<repository>
    <id>github</id>
    <name>Maven Packages</name>
    <url>https://maven.pkg.github.com/cloudneutral/cockroachdb-jdbc</url>
    <snapshots>
        <enabled>true</enabled>
    </snapshots>
</repository>

You need to authenticate to GitHub Packages by creating a personal access token (classic) that includes the read:packages scope. For more information, see Authenticating to GitHub Packages.

Add your personal access token to the servers section in your settings.xml:

<server>
    <id>github</id>
    <username>your-github-name</username>
    <password>your-access-token</password>
</server>

Take note that the server and repository id:s must match (it can be different than github). Now you should be able to build your project with the JDBC driver as a dependency:

mvn clean install

Alternatively, you can just clone the repository and build it locally using mvn install.

Modules

The JDBC driver project is a multi-module Maven project with the following components:

cockroachdb-jdbc-driver

The main library for the CockroachDB JDBC driver. This is all you need and it transitively pulls in pgJDBC and log4j as only dependencies.

cockroachdb-jdbc-it

Integration tests and functional tests that are activated via Maven profiles.

cockroachdb-jdbc-demo

A standalone demo app to showcase the retry mechanism and other features.

Supported CockroachDB and JDK Versions

The driver is CockroachDB version agnostic and supports any version supported by the PostgreSQL JDBC driver v 42.5+ (pgwire protocol v3.0). It's built for Java 8 at the language source and target level but requires Java 17 LTS for building.

URL Properties

The driver uses the jdbc:cockroachdb: JDBC URL prefix and supports all PostgreSQL URL properties on top of that. To configure a data source to use this driver, you typically configure it for PostgreSQL and only change the URL prefix and the driver class name.

The general format for a JDBC URL for connecting to a CockroachDB server:

jdbc:cockroachdb:[//host[:port]/][database][?property1=value1[&property2=value2]...]

See pgjdbc for all supported driver properties and the semantics.

In addition, this driver has the following CockroachDB-specific properties:

retryTransientErrors

(default: false)

The JDBC driver will automatically retry serialization failures (40001 state code) at read, write or commit time. This is done by keeping track of all statements and the results during a transaction, and if the transaction is aborted due to a transient 40001 error, it will roll back and retry the recorded operations on a new connection and compare the results with the initial commit attempt. If the results are different, the driver will be forced to give up the retry attempt to preserve a serializable outcome.

Enable this option if you want to handle aborted transactions internally in the driver, preferably combined with select-for-update locking. Leave this option disabled if you want to handle aborted transactions in your application.

retryConnectionErrors

(default: false)

The CockroachDB JDBC driver will automatically retry transient connection errors with SQL state 08001, 08003, 08004, 08006, 08007, 08S01 or 57P01 at read, write or commit time.

Applicable only when retryTransientErrors is also true.

Disable this option if you want to handle connection errors in your own application or connection pool.

CAUTION! Retrying on non-serializable conflict errors (i.e anything but 40001) may produce duplicate outcomes if the SQL statements are non-idempotent. See the design notes for more details.

retryListenerClassName

(default: io.cockroachdb.jdbc.retry.LoggingRetryListener)

Name of the class that implements io.cockroachdb.jdbc.retry.RetryListener to be used to receive callback events when retries occur. One instance is created for each JDBC connection.

retryStrategyClassName

(default: io.cockroachdb.jdbc.retry.ExponentialBackoffRetryStrategy)

Name of the class that implements io.cockroachdb.jdbc.retry.RetryStrategy to be used when retryTransientErrors property is set to true. If this class also implements io.cockroachdb.jdbc.proxy.RetryListener it will receive callback events when retries happen. One instance of this class is created for each JDBC connection.

The default ExponentialBackoffRetryStrategy will use an exponentially increasing delay with jitter and a multiplier of 2 up to the limit set by retryMaxBackoffTime.

retryMaxAttempts

(default: 15)

A maximum number of retry attempts on transient failures (connection errors/serialization conflicts). If this limit is exceeded, the driver will throw a SQL exception with the same state code signalling yielding further retry attempts.

retryMaxBackoffTime

(default: 30s)

Maximum exponential backoff time in the format of a duration expression (like 12s). The duration applies to the total time for all retry attempts at transaction level.

Applicable only when retryTransientErrors is true.

implicitSelectForUpdate

(default: false)

The driver will automatically append a FOR UPDATE clause to all qualified SELECT statements within connection scope. This parameter can also be set in an explicit transaction as a session variable in which case its scope to the transaction.

The qualifying requirements include:

• Not used in a read-only connection

• No time travel clause (as of system time)

• No aggregate functions

• No group by or distinct operators

• Not referencing internal table schema

A SELECT .. FOR UPDATE will lock the rows returned by a selection query such that other transactions trying to access those rows are forced to wait for the transaction that locked the rows to finish. These other transactions are effectively put into a queue based on when they tried to read the value of the locked rows. It does not eliminate the chance of serialization conflicts but greatly reduces it.

useCockroachMetadata

(default: false)

By default, the driver will use PostgreSQL JDBC driver metadata provided in java.sql.DatabaseMetaData rather than CockroachDB-specific metadata. While the latter is more correct, it causes incompatibilities with libraries that bind to PostgreSQL version details, such as Flyway and other tools.

Logging

This driver uses SLF4J for logging which means it's agnostic to the logging framework used by the application. The JDBC driver module does not include any logging framework dependency transitively.

Additional Examples

Plain Java Example

Class.forName(CockroachDriver.class.getName());

try (Connection connection 
        = DriverManager.getConnection("jdbc:cockroachdb://localhost:26257/jdbc_test?sslmode=disable&implicitSelectForUpdate=true&retryTransientErrors=true") {
  try (Statement statement = connection.createStatement()) {
    try (ResultSet rs = statement.executeQuery("select version()")) {
      if (rs.next()) {
        System.out.println(rs.getString(1));
      }
    }
  }
}

Spring Boot Example

Configure the datasource in src/main/resources/application.yml:

spring:
  datasource:
    driver-class-name: io.cockroachdb.jdbc.CockroachDriver
    url: "jdbc:cockroachdb://localhost:26257/jdbc_test?sslmode=disable&application_name=MyTestAppe&implicitSelectForUpdate=true&retryTransientErrors=true"
    username: root
    password:

Optionally, configure the data source programmatically and use the TTDDYY logging proxy:

@Bean
@Primary
public DataSource dataSource() {
    return ProxyDataSourceBuilder
            .create(hikariDataSource())
            .traceMethods()
            .logQueryBySlf4j(SLF4JLogLevel.DEBUG, "io.cockroachdb.jdbc")
            .asJson()
            .multiline()
            .build();
}

@Bean
@ConfigurationProperties("spring.datasource.hikari")
public HikariDataSource hikariDataSource() {
    HikariDataSource ds = dataSourceProperties()
            .initializeDataSourceBuilder()
            .type(HikariDataSource.class)
            .build();
    ds.setAutoCommit(false);
    ds.addDataSourceProperty(PGProperty.REWRITE_BATCHED_INSERTS.getName(), "true");
    ds.addDataSourceProperty(CockroachProperty.IMPLICIT_SELECT_FOR_UPDATE.getName(), "true");
    ds.addDataSourceProperty(CockroachProperty.RETRY_TRANSIENT_ERRORS.getName(), "true");
    ds.addDataSourceProperty(CockroachProperty.RETRY_MAX_ATTEMPTS.getName(), "5");
    ds.addDataSourceProperty(CockroachProperty.RETRY_MAX_BACKOFF_TIME.getName(), "10000");
    return ds;
}

To configure src/main/resources/logback-spring.xml to capture all SQL statements and JDBC API calls:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include resource="org/springframework/boot/logging/logback/defaults.xml"/>
    <include resource="org/springframework/boot/logging/logback/console-appender.xml" />

    <logger name="org.springframework" level="INFO"/>

    <logger name="io.cockroachdb.jdbc" level="DEBUG"/>

    <root level="INFO">
        <appender-ref ref="CONSOLE"/>
    </root>
</configuration>

Building

The CockroachDB JDBC driver requires Java 17 (or later) LTS but is cross-compiled to run on any platform for which there is a Java 8 runtime.

Prerequisites

• JDK17+ LTS for building (OpenJDK compatible)

• Maven 3+ (optional, embedded)

If you want to build with the regular mvn command, you will need Maven v3.x or above.

Install the JDK (Linux):

sudo apt-get -qq install -y openjdk-17-jdk

Install the JDK (macOS):

brew install openjdk@17

Clone the project

git clone git@github.com:cloudneutral/cockroachdb-jdbc.git
cd cockroachdb-jdbc

Build the project

chmod +x mvnw
./mvnw clean install

The JDBC driver jar is now found in cockroachdb-jdbc-driver/target.

Run Integration Tests

The integration tests will run through a series of contended workloads to exercise the retry mechanism and other driver features.

First, start a local CockroachDB node or cluster.

Create the database:

cockroach sql --insecure --host=localhost -e "CREATE database jdbc_test"

Then activate the integration test Maven profile:

./mvnw -P it -Dgroups=anomaly-test clean install

Test groups include:

• anomaly-test - Runs through a series of RW/WR/WW anomaly tests.

• connection-retry-test - Run a test with connection retries enabled.

• batch-insert-test - Batch inserts load test.

• batch-update-test - Batch updates load test.

See the pom.xml file for changing the database URL and other settings (under ìt profile).

Summary

This article provides instructions on how to configure and build a new open-source JDBC driver for CockroachDB. It covers parameters such as retryTransientErrors, implicitSelectForUpdate to reduce transient SQL exceptions on contended workloads. It also explains the configuration of the driver, including retry strategies, URL properties, and logging settings, as well as examples of how to configure the driver in plain Java and Spring Boot.