[{"content":"Standardized benchmarks such as TPC-C, YCSB and sysbench are useful for comparing database systems under defined workloads. But those workloads rarely match an application\u0026rsquo;s specific transaction patterns, data access paths, contention characteristics or concurrency model. Reproducing those interactions under load provides a different perspective: how the database behaves under traffic shaped like your application.\nBattery is an open-source load-testing tool built for that purpose. You describe the workload using YAML, SQL and a small scripting language, then run it with potentially thousands of concurrent virtual users (VUs), optionally distributed across multiple hosts.\nThis post covers how Battery works and some of the design decisions behind it.\nThe shape of a load test A Battery test is a YAML file with four main parts:\nbefore steps that run once to prepare a schema or load reference data\nphases that control how fast virtual users arrive over time\nscenarios, the sequence of steps each virtual user runs\nafter steps for cleanup\nHere\u0026rsquo;s a complete CRUD workload against a single table:\nbattery: baseDir: config/demo before: steps: - name: Prepare Schema sql: |- create schema if not exists demo; create table if not exists demo.customer ( id uuid not null default gen_random_uuid(), email varchar(64) not null, first_name varchar(64), last_name varchar(64), primary key (id) ); phases: - name: Warm up phase duration: 60s users: 20 - name: Peak load phase duration: 30s startRate: 4 maxRate: 50 maxConcurrency: 64 - name: Sustained peak phase duration: 15s scenarios: - name: Customer CRUD duration: 20s steps: - name: Create Customer capture: true sql: |- insert into demo.customer (id, email, first_name, last_name) values (gen_random_uuid(), \u0026#39;#{ gen.randomEmail() }\u0026#39;, \u0026#39;#{ gen.randomFirstName() }\u0026#39;, \u0026#39;#{ gen.randomLastName() }\u0026#39; ) returning id; - name: Retrieve Customer sql: |- select * from demo.customer where id = :id; params: id: id - name: Update Customer sql: |- update demo.customer set first_name=:first_name, last_name=:last_name where id=:id; params: id: id first_name: gen.randomFirstName() last_name: gen.randomLastName() - name: Delete Customer sql: |- delete from demo.customer where id=:id; params: id: id Two features are doing most of the work here.\nTemplating. #{ ... } expressions are evaluated and spliced into the SQL, and params: binds named :placeholders to expressions that are evaluated on every execution. Each execution can therefore get fresh generated data.\nUnlike templated substitutions, named parameters use JDBC parameter binding, keeping values separate from the SQL statement.\nState capture. Setting capture: true on a step passes its result to the steps that follow. If one row comes back, each column becomes a variable, so the id returned by the insert is available as id in the next step. If several rows come back, they\u0026rsquo;re bound as a list (result by default) that later steps can index into or sample from. This is what turns a sequence of SQL statements into an application-like workflow: create an entity, capture its identity, then read, modify and eventually delete that same entity without writing glue code.\nCaptured state belongs to one virtual user and resets at the start of each iteration. State captured in before steps is handed to every VU as its initial state, which is an easy way to share lookup data such as a list of valid account IDs.\nA realistic example: transferring money The bundled bank configuration shows where this gets interesting. A transfer scenario first picks two accounts in a randomly selected city, then performs a transfer that inserts a transfer record, writes two ledger entries with running balances, and updates both account balances:\nbattery: baseDir: config/bank scenarios: - name: Transfer Funds (explicit + optimistic) alias: a duration: 2m transactional: true steps: - name: Find Accounts capture: true sql: |- SELECT id, balance, city FROM bank.account WHERE city = :city ORDER BY id LIMIT 2; params: city: std.selectRandom(L[\u0026#34;london\u0026#34;,\u0026#34;new york\u0026#34;,\u0026#34;stockholm\u0026#34;]); - name: Transfer (explicit) capture: true path: transfer-explicit.sql params: city: std.selectRandom(result).get(\u0026#34;city\u0026#34;); transfer_id: transfer_id account_id1: result.get(0).get(\u0026#34;id\u0026#34;); account_id2: result.get(1).get(\u0026#34;id\u0026#34;); amount: gen.randomBigDecimal(10.00,500.00) - name: Transfer Funds (implicit - modifying CTE) alias: c duration: 2m transactional: false steps: - name: Find Accounts capture: true sql: |- SELECT id, balance, city FROM bank.account WHERE city = :city ORDER BY id LIMIT 2; params: city: std.selectRandom(L[\u0026#34;london\u0026#34;,\u0026#34;new york\u0026#34;,\u0026#34;stockholm\u0026#34;]); - name: Transfer capture: true path: transfer-implicit.sql params: city: std.selectRandom(result).get(\u0026#34;city\u0026#34;); transfer_id: transfer_id account_id1: result.get(0).get(\u0026#34;id\u0026#34;); account_id2: result.get(1).get(\u0026#34;id\u0026#34;); amount: gen.randomBigDecimal(10.00,500.00) The complete configuration explores several ways of executing the same logical transfer. In the explicit variants, Battery wraps the scenario steps in a transaction, allowing accounts to be locked with FOR UPDATE before the transfer is performed.\nIn the implicit variant, the transfer is expressed as a single SQL statement using data-modifying CTEs. The database executes that statement atomically within an implicit transaction, while the preceding account-selection step remains outside that transaction.\nAccounts are deliberately drawn from a small set of cities, increasing the probability of concurrent transactions contending on the same data. This makes it possible to compare how different transaction and locking strategies behave as contention increases.\nWhen Battery encounters a retryable transaction failure, such as a serialization failure (SQLSTATE 40001), the VU backs off with jitter and restarts the scenario from the beginning rather than retrying only the failed statement. Captured state is rebuilt, and expressions are evaluated again, so a retry may select different accounts or generate different values. This models a fresh execution of the workflow rather than a byte-for-byte replay of the original SQL statements. Connection failures require additional care because the outcome of a transaction may be ambiguous, particularly if the connection is lost during commit.\nIn the implicit variant, the transfer itself is a single atomic SQL statement, so there is no partially completed multi-statement transaction to recover.\nWhen SQL isn\u0026rsquo;t enough: Battery Script Templated SQL handles a lot, but real applications branch. For that, Battery includes Battery Script, a small procedural language.\nIt has dynamic typing, if/while/foreach/for ... from ... to loops, collection literals, high-precision decimals (12.50bd), and a namespaced function library for SQL (jdbc), random data (gen), and general utilities (std).\nHere\u0026rsquo;s a simplified example where the next database operation depends on state read from the database:\naccount = jdbc.queryForMap( \u0026#34;select id, balance from account where id=?\u0026#34;, L[accountId] ); if (account.get(\u0026#34;balance\u0026#34;) \u0026lt; amount) { // Reject transfer } else { jdbc.update( \u0026#34;update account set balance=balance-? where id=?\u0026#34;, L[amount, accountId] ); The important part isn\u0026rsquo;t the SQL itself, but that the workload can make application-side decisions between database operations. That makes it possible to model workflows where subsequent SQL depends on state observed earlier in the request.\nIt also supports fork/join blocks for firing concurrent statements from within a single virtual user, which is how you\u0026rsquo;d model an application issuing parallel queries to render one page.\nBattery Script is meant to be simple and readable by someone who mostly writes SQL. You can try expressions interactively with the shell\u0026rsquo;s execute command and list every available function with functions.\nDesign decisions Virtual threads instead of a thread pool Each VU runs on a Java 21 virtual thread, and so does each fork block.\nA database load generator spends much of its time waiting on blocking I/O, which makes virtual threads a natural fit. A single Battery process can support thousands of concurrent VUs without requiring thousands of dedicated operating-system threads or asynchronous callback code.\nAn open workload model that drops instead of queueing Phases are defined by arrivals rather than a fixed number of continuously looping users. You can specify either a total number of arrivals using users, distributed across the phase duration, or an arrival rate using startRate and maxRate. Each arrival creates a virtual user that executes a scenario for its configured duration. Phases can overlap: a phase doesn\u0026rsquo;t wait for its users to finish before the next one begins.\nThe subtle part is maxConcurrency. When the cap is reached, new arrivals are dropped and counted, not queued. That keeps the arrival rate and phase duration honest. If the system under test falls behind, the test still runs on schedule, and the dropped-user count tells you that demand went unmet.\nQueueing arrivals would quietly stretch the test and make a struggling system look healthier than it is. The load generator would effectively adapt to the slowdown of the system it is supposed to measure. Battery instead keeps offered load independent of how quickly the database can service it.\nThis helps avoid coordinated omission, where a slowing system causes the load generator to produce less work precisely when latency is increasing. By scheduling arrivals independently of completion times, Battery maintains the intended arrival rate rather than adapting to the system\u0026rsquo;s response times. When the concurrency limit is reached, new arrivals are dropped and counted instead of being queued. Latency percentiles should therefore be interpreted alongside the dropped-arrival count, particularly when the system is overloaded.\nThe same thinking applies to the connection pool. When VUs outnumber connections, the time spent waiting for a connection is included in the measured latency. That wait is part of what your users would experience.\nBattery deliberately measures this as client-observed latency rather than database execution time alone. If a VU spends 200 ms waiting for a connection before the database executes a query in 10 ms, the application still experienced roughly 210 ms of latency.\nBattery can size its connection pool according to the maximum configured concurrency across the test phases, subject to a configurable maxSize limit.\nFindings, not just numbers When a run finishes, Battery prints a summary with throughput, latency percentiles, VU and pool statistics, and a list of findings: plain-language diagnoses derived from the data.\nExamples:\nConnection pool saturated: up to N users waited for a connection\u0026hellip; Lower maxConcurrency or raise battery.connectionPool.maxSize.\nThroughput leveling off: offered load increased between phases but achieved throughput didn\u0026rsquo;t keep up \u0026ndash; the \u0026ldquo;knee\u0026rdquo; where adding more load stops producing more useful work\np99 latency degrading sharply from one phase to the next\nA long latency tail (p99.9 far above the median)\nToo few samples for a percentile to be trusted\nThe sample-size check is a small detail that matters. A p99.9 computed from only 300 iterations isn\u0026rsquo;t a meaningful tail estimate, and the tool says so instead of presenting it with false precision. Summaries are kept in memory, exposed through the API, and saved as JSON under .log/runs/ so you can compare runs later.\nDistributed load with a consistency check One machine won\u0026rsquo;t always generate enough load. Any Battery instance can act as an agent, and a controlling instance lists the others under network.agents. You then drive them with agent run, agent show status, and agent cancel.\nThe safeguard is that every agent must run the same test configuration. Before a distributed run starts, the controller sends a hash of the workload model, and an agent whose own workload model hashes differently rejects the run.\nThree ways to run Battery ships as a single executable JAR built on Spring Boot, with three interfaces:\nAn interactive shell with tab completion for commands, scenario names, and script functions. It\u0026rsquo;s the main control plane: validate model, show db, run, cancel, show errors.\nA web UI on port 9090 with live charts streamed over WebSockets, a runs history page, and a script playground.\nA hypermedia REST API for automation. Responses expose state-dependent links, so automation can discover which operations are currently valid.\nPrometheus metrics are exposed at /api/actuator/prometheus if you\u0026rsquo;d rather watch runs in your existing Grafana dashboards.\nGetting started You need JDK 21+ and a local PostgreSQL or CockroachDB:\ncreatedb battery git clone https://github.com/kai-niemi/battery.git \u0026amp;\u0026amp; cd battery ./mvnw clean install ./run.sh # pick the demo-crud profile Then, in the shell:\nbattery:$ validate model battery:$ show db battery:$ run Open http://localhost:9090 to watch it live. PostgreSQL and CockroachDB drivers are bundled. MySQL and Oracle drivers are available through the jdbc-drivers build profile.\nWrapping up Battery isn\u0026rsquo;t intended to produce a universal benchmark score. Its purpose is to answer a more application-specific question: what happens when a database is subjected to traffic that behaves like my application?\nBy combining SQL, YAML and lightweight scripting, Battery makes it possible to reproduce stateful workflows, transaction boundaries, contention and concurrency under controlled load. It handles the mechanics of generating that load, distributing it across hosts and collecting the results.\nIf you\u0026rsquo;re evaluating an OLTP database, testing a schema or transaction change, or investigating where your current setup stops keeping up with demand, give it a try. The README and Battery Script guide are good places to start.\n","permalink":"https://blog.cloudneutral.se/battery-database-load-testing/","summary":"\u003cp\u003eStandardized benchmarks such as TPC-C, YCSB and sysbench are useful for comparing database systems under defined\nworkloads. But those workloads rarely match an application\u0026rsquo;s specific transaction patterns, data access paths,\ncontention characteristics or concurrency model. Reproducing those interactions under load provides a different\nperspective: how the database behaves under traffic shaped like your application.\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://github.com/kai-niemi/battery\"\u003eBattery\u003c/a\u003e is an open-source load-testing tool built for that purpose. You describe\nthe workload using YAML, SQL and a small scripting language, then run it with potentially thousands of concurrent\nvirtual users (VUs), optionally distributed across multiple hosts.\u003c/p\u003e","title":"Battery: A Workload-Driven Database Load Testing Tool"},{"content":"Distributed applications frequently need to coordinate operations across multiple transactional resources. A service might update a database and acknowledge a message, or persist a record and invoke a remote API.\nThe challenge is that these operations are independent. A successful database commit does not guarantee that a message acknowledgement or remote API call will succeed.\nOne commonly adopted strategy is best-effort one-phase commit (1PC). Rather than introducing a distributed transaction coordinator, it relies on careful transaction ordering and idempotent processing to reduce the risk of inconsistent outcomes.\nThis approach is simpler than XA/two-phase commit (2PC), but it comes with an important trade-off: there is no atomicity across the participating resources.\nThe approach is also discussed in David Syer\u0026rsquo;s article Distributed Transactions in Spring, with and without XA.\nThe Dual-Write Problem A dual write occurs when an application performs two operations that must logically succeed together but cannot be committed atomically.\nConsider a service that processes an incoming message and updates a database:\nsequenceDiagram participant B as Message Broker participant A as Application participant D as Database B-\u003e\u003eA: Deliver message A-\u003e\u003eD: Begin transaction A-\u003e\u003eD: Update records A-\u003e\u003eD: Commit transaction D--\u003e\u003eA: Commit successful A-\u003e\u003eB: Acknowledge message The database commit and message acknowledgement are separate operations. Even if both are individually transactional, there is no shared transaction covering them.\nSuppose the database commit succeeds but the application crashes before acknowledging the message. The broker may redeliver it, causing the application to repeat the database operation.\nReversing the order creates a different problem. If the application acknowledges the message before committing the database transaction, a subsequent database failure may cause the message to be lost without its intended effects being persisted.\nThis is the fundamental problem that best-effort 1PC attempts to manage.\nBest-Effort One-Phase Commit Best-effort 1PC avoids coordinating multiple resources through a global transaction manager. Instead, each resource commits independently, with operations ordered to minimize the risk of inconsistency.\nThe term best-effort reflects that the application attempts to coordinate the completion of multiple operations without a global atomic commit protocol. It cannot guarantee that all participating resources reach the same outcome.\nThe general strategy is to:\nPerform business validation before committing. Complete database operations before acknowledging or committing the incoming message. Make processing idempotent so that retries do not repeat business effects. The intention is to expose predictable failures, such as invalid input and business rule violations, before acknowledging successful processing.\nInfrastructure failures can still occur between commits, but these are handled through redelivery and idempotent processing rather than atomic coordination.\nBest-effort 1PC therefore provides a practical compromise between implementation complexity and consistency guarantees. It does not make independent commits atomic.\nDatabase and Message Broker Consider a Spring Boot service that consumes registration events from a JMS broker and persists them in a relational database.\nA simplified consumer might look like this:\n@Component public class RegistrationConsumer { private final RegistrationRepository repository; public RegistrationConsumer(RegistrationRepository repository) { this.repository = repository; } @JmsListener(destination = \u0026#34;registrations\u0026#34;) @Transactional public void receiveMessage(RegistrationEvent event) { repository.save(toEntity(event)); } } The intended processing sequence is:\nThe broker delivers a message. The application begins a database transaction. The application validates and processes the message. The database transaction commits. The broker acknowledges the message. The example assumes that the JMS listener container is configured to acknowledge or commit message consumption only after successful processing. The exact behavior depends on the listener container, transaction manager, and acknowledgement configuration. The @Transactional annotation alone does not establish a distributed transaction between JMS and the database.\nWhy Commit Ordering Matters Suppose the database transaction fails because of a constraint violation.\nIf the message has not yet been acknowledged, the broker can redeliver it according to its retry and error-handling configuration. The database changes are rolled back, and the message remains available for further processing.\nNow consider a different failure:\n1. Receive message 2. Update database 3. Commit database SUCCESS 4. Acknowledge message FAILURE 5. Broker redelivers message The application may process the same message again even though the database update was committed successfully. This is why idempotency is essential for reliable at-least-once message processing.\nFor example, a registration event could contain a stable event identifier:\npublic record RegistrationEvent( UUID eventId, UUID customerId, String email ) {} The consumer can use that identifier to detect previously processed events. A common approach is to maintain a table of processed message identifiers and update it in the same database transaction as the business operation:\nCREATE TABLE processed_messages ( event_id UUID PRIMARY KEY, processed_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ); When processing a message, the application attempts to insert its identifier. If the identifier already exists, the message has been processed before and its business effects should not be repeated.\nThe deduplication record and business update must be committed atomically in the same database transaction. Otherwise, the application introduces another consistency gap.\nDatabase Flush and Commit When using JPA, database operations may be deferred until the persistence context is flushed.\nCalling repository.save() does not necessarily mean that the corresponding SQL statements have been executed. The JPA provider may defer inserts, updates, and deletes until flush or transaction commit, at which point database constraints may also be evaluated.\nExplicitly flushing the persistence context can expose some failures earlier, but it does not guarantee that the transaction will commit successfully. Deadlocks, serialization failures, connection loss, and other commit-time failures may still occur.\nThe important requirement is that the database transaction commits successfully before the broker acknowledges the message.\nWhat About Kafka? The same general problem applies to Kafka consumers, although the mechanics differ.\nUnlike traditional message queues, Kafka does not remove records when consumers acknowledge processing. Instead, consumers track their progress through offsets, while records are retained according to the topic\u0026rsquo;s configured retention and cleanup policies.\nA consumer that updates a database and commits its offset performs two independent operations. If the database commit succeeds but the offset commit fails, the record may be processed again after a restart or rebalance.\nKafka provides additional guarantees through idempotent producers and transactions.\nIdempotent producers prevent duplicate records caused by retries of the same produce operation, using producer identities and sequence numbers maintained by Kafka.\nKafka transactions extend these guarantees by allowing produced records and consumed offsets to be committed atomically within Kafka, supporting exactly-once processing for Kafka-to-Kafka workflows when consumers are configured appropriately.\nHowever, neither mechanism automatically extends atomicity to an external relational database. A database commit and a Kafka offset commit remain independent operations.\nIdempotent database processing therefore remains important when consuming Kafka records and updating external transactional resources.\nDatabase and Remote API A second scenario involves updating a database and invoking a remote API.\nSuppose a service needs to initiate a payment through an external provider and record the result locally.\nA tempting implementation is:\n@Transactional public void createTransfer(TransferRequest request) { paymentClient.createTransfer(request); transferRepository.save(toEntity(request)); } This performs a remote call while holding a database transaction open.\nThere are several problems with this approach. Network calls can be slow or fail unpredictably, increasing transaction duration and potentially holding database locks or connections longer than necessary.\nMore importantly, the remote API call is not part of the database transaction.\nIf the API succeeds but the database transaction rolls back, the external payment may still have been created.\nReordering Does Not Eliminate the Problem One improvement is to invoke the remote API outside the database transaction:\n1. Call remote API 2. Begin database transaction 3. Persist result 4. Commit database transaction This avoids holding a database transaction open during the network call, but the two operations remain independent.\nConsider:\n1. POST /transfers SUCCESS 2. Database transaction FAILURE 3. Application retries 4. POST /transfers SUCCESS Without additional safeguards, the retry may create a second transfer.\nReversing the order does not solve the problem either:\n1. Persist transfer SUCCESS 2. Commit database SUCCESS 3. POST /transfers FAILURE Now the database records a transfer that may not have been created remotely.\nThe correct ordering depends on the business operation and its recovery requirements. Neither ordering provides atomicity.\nIdempotency Keys For APIs that support idempotency, the application can associate a stable identifier with the logical operation.\nFor example:\nPOST /transfers HTTP/1.1 Content-Type: application/json Idempotency-Key: 7d9e2b8a-6f41-4e73-9a2c-814cbdf37e61 { \u0026#34;sourceAccount\u0026#34;: \u0026#34;A123\u0026#34;, \u0026#34;destinationAccount\u0026#34;: \u0026#34;B456\u0026#34;, \u0026#34;amount\u0026#34;: 100.00, \u0026#34;currency\u0026#34;: \u0026#34;EUR\u0026#34; } If the request is retried using the same idempotency key, the receiving service can recognize that the operation has already been processed and avoid creating another transfer.\nThis also addresses an important ambiguity in distributed systems: a request may succeed even when the client never receives its response.\nsequenceDiagram participant C as Client participant A as Remote API C-\u003e\u003eA: POST /transfers activate A Note right of A: Process transfer Note right of A: Commit result A--xC: Response lost (connection failure) deactivate A C-\u003e\u003eA: Retry with same idempotency key activate A Note right of A: Recognize duplicate A--\u003e\u003eC: Return original result deactivate A The client cannot determine whether the original request succeeded merely from a timeout. A stable idempotency key allows the remote service to associate retries with the same logical operation.\nThe receiving service must enforce this guarantee. Simply sending an idempotency key does not make an endpoint idempotent. Idempotency keys must remain stable across retries of the same logical operation. Generating a new key for each retry defeats the purpose.\nFor a deeper discussion, see Designing Idempotent REST APIs.\nAlternatives to Best-Effort 1PC Best-effort 1PC is not the only approach to coordinating operations across transactional boundaries. Several patterns provide different trade-offs between complexity, availability, and consistency.\nTransactional Outbox The transactional outbox pattern addresses the problem of updating a database and publishing an event.\nInstead of writing directly to the database and message broker, the application stores both the business update and an outgoing event in the same database transaction.\nflowchart TD A[\"Application\"] --\u003e T subgraph T[\"Database Transaction\"] direction LR B[\"Business Data\"] O[\"Outbox Event\"] end T --\u003e C[\"COMMIT\"] C --\u003e P[\"Outbox Publisher\"] P --\u003e|At-least-once delivery| M[\"Message Broker\"] A separate process publishes committed outbox events to the broker, commonly through polling or change data capture (CDC). This removes the non-atomic database-and-broker write from the application transaction. However, publishing may still happen more than once, so consumers must tolerate duplicate events.\nIdempotent Consumer and Inbox The inbox pattern complements the outbox by recording incoming messages or their identifiers in the receiving database. The consumer can atomically record that a message has been processed alongside the corresponding business update. If the broker redelivers the same message, the consumer recognizes the duplicate and avoids repeating its effects.\nsequenceDiagram participant B as Message Broker participant C as Consumer participant D as Database B-\u003e\u003eC: Deliver message (eventId) C-\u003e\u003eD: Begin transaction C-\u003e\u003eD: Insert eventId into inbox alt New message D--\u003e\u003eC: Insert successful C-\u003e\u003eD: Apply business updates C-\u003e\u003eD: Commit transaction D--\u003e\u003eC: Commit successful else Already processed D--\u003e\u003eC: Duplicate eventId C-\u003e\u003eD: Rollback transaction end C-\u003e\u003eB: Acknowledge message A duplicate identifier should only be treated as successfully processed once the original transaction has committed. Concurrent attempts must be resolved through the database\u0026rsquo;s uniqueness and transaction semantics.\nThis provides a practical way to achieve effectively-once business processing on top of at-least-once message delivery, provided the deduplication and business updates share the same transaction.\nSaga A saga coordinates a business process across multiple independently committed local transactions. Each participating service maintains its own state and uses local ACID transactions, while the overall workflow progresses through a sequence of state transitions.\nSagas are commonly used in microservice architectures where business transactions span multiple service boundaries. Coordination can be orchestrated through a central coordinator or choreographed through asynchronous events.\nFor example, an order workflow might:\nReserve inventory. Authorize payment. Confirm the order. If payment authorization fails, the system can release the inventory reservation through a compensating action.\nUnlike XA/2PC, sagas do not provide atomicity across participating services. Intermediate states may be visible, and compensation is not equivalent to an atomic rollback. Some operations cannot be fully reversed.\nThe benefit is greater service autonomy and reduced coupling between transactional resources. The trade-off is additional complexity in managing distributed state transitions, retries, compensation, and failure recovery. Asynchronous workflows can also be harder to trace and reason about than a single database transaction.\nSagas are often combined with transactional outbox and idempotent consumer patterns to make message-driven coordination more reliable.\nXA and Two-Phase Commit Two-phase commit (2PC) resembles a wedding ceremony, with the transaction coordinator acting as the priest.\nIn the prepare phase, each participant promises to commit if instructed, much like exchanging vows. Once everyone agrees, the coordinator records the decision and instructs them to commit. If anyone refuses, the transaction is rolled back.\nBut what if the priest disappears before announcing the decision? Participants may be left waiting, unable to proceed independently. This illustrates the blocking problem of 2PC.\nXA defines a standardized interface through which a transaction manager coordinates compatible transactional resources, typically using 2PC to provide atomic commit guarantees.\nNot every database, broker, or external service supports XA. In particular, ordinary HTTP API calls cannot generally participate in such a transaction.\nWhen Is Best-Effort 1PC Appropriate? Best-effort 1PC is useful when operations span independent resources, the remaining failure windows are acceptable, and recovery through retries is practical.\nIt is particularly suitable for message-driven processing where the broker supports redelivery and the database operation can be made idempotent.\nHowever, idempotency is not a substitute for every consistency guarantee. It prevents repeated execution from creating duplicate effects, but it does not ensure that an operation which has never completed will eventually succeed.\nFor critical workflows, the design must also account for retry exhaustion, permanently failing operations, ambiguous outcomes, and reconciliation between systems.\nThe choice between best-effort 1PC, an outbox, a saga, or distributed transactions depends on the required guarantees rather than implementation convenience alone.\nConclusion Best-effort one-phase commit is a pragmatic alternative to distributed transactions when independent resource commits are acceptable. By committing the database before acknowledging incoming messages, applications can favor redelivery over message loss. Idempotency makes that redelivery safe, but does not eliminate the possibility of partial failures.\nFor workflows requiring stronger recovery guarantees, patterns such as transactional outbox, inbox, and saga provide additional options without necessarily introducing XA.\n","permalink":"https://blog.cloudneutral.se/best-effort-one-phase-commit-and-the-dual-write-problem/","summary":"\u003cp\u003eDistributed applications frequently need to coordinate operations across multiple transactional resources. A service\nmight update a database and acknowledge a message, or persist a record and invoke a remote API.\u003c/p\u003e\n\u003cp\u003eThe challenge is that these operations are independent. A successful database commit does not guarantee that a message\nacknowledgement or remote API call will succeed.\u003c/p\u003e\n\u003cp\u003eOne commonly adopted strategy is \u003cstrong\u003ebest-effort one-phase commit (1PC)\u003c/strong\u003e. Rather than introducing a distributed\ntransaction coordinator, it relies on careful transaction ordering and idempotent processing to reduce the risk of\ninconsistent outcomes.\u003c/p\u003e","title":"Best-Effort One-Phase Commit and the Dual-Write Problem"},{"content":"Spring Boot makes it easy to schedule recurring tasks using the @Scheduled annotation. This works well for a single application instance, but what happens when the application is deployed across multiple servers or regions?\nEach instance runs its own scheduler, meaning the same task may execute concurrently on several instances. For some workloads, this is harmless. For others, such as importing data, publishing notifications, or performing periodic cleanup, concurrent execution can cause duplicate processing or conflicting updates.\nWhat we need is a cluster singleton: a scheduled task that runs on only one application instance at a time, regardless of how many instances are deployed. One straightforward approach is to coordinate scheduled execution through a shared database using ShedLock.\nThe Problem with @Scheduled Consider a Spring Boot application with a scheduled cleanup task:\n@Component public class CleanupJob { @Scheduled(cron = \u0026#34;0 */5 * * * *\u0026#34;) public void cleanup() { // Remove expired records } } The task executes every five minutes. With a single application instance, this behaves as expected. However, when the application is deployed across three instances, all three independently execute the same task.\nflowchart TD S[\"Every 5 minutes\"] S --\u003e A[\"Instance A\"] S --\u003e B[\"Instance B\"] S --\u003e C[\"Instance C\"] A --\u003e CA[\"Cleanup\"] B --\u003e CB[\"Cleanup\"] C --\u003e CC[\"Cleanup\"] Spring\u0026rsquo;s scheduling mechanism has no built-in coordination between application instances. One option is to designate a specific instance to run scheduled tasks, but this introduces an operational dependency on that instance and complicates failover.\nA more flexible approach is to allow every instance to schedule the task while coordinating which one is permitted to execute it.\nIntroducing ShedLock ShedLock is a lightweight Java library that provides distributed locking for scheduled tasks. It works alongside Spring\u0026rsquo;s scheduling mechanism rather than replacing it:\nSpring @Scheduled determines when a task should run. ShedLock coordinates which application instance may execute it. A shared database stores the lock state. When a scheduled task is triggered, each instance attempts to acquire a lock identified by a unique name. Only one instance can acquire the lock, while the others skip that execution rather than waiting for the lock to become available.\nWhen the task completes, the lock is released. If the instance crashes before releasing it, the lock eventually expires, allowing another instance to take over. This approach requires no dedicated scheduling service and works with many relational databases through JDBC.\nConfiguring ShedLock The following example uses Spring Boot, Spring\u0026rsquo;s built-in scheduling, and ShedLock\u0026rsquo;s JDBC provider.\nMaven Dependencies Add the ShedLock dependencies to your pom.xml:\n\u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;net.javacrumbs.shedlock\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;shedlock-spring\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;${shedlock.version}\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;net.javacrumbs.shedlock\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;shedlock-provider-jdbc-template\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;${shedlock.version}\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; Use the same ShedLock version for both dependencies. ShedLock 5.x and later require Java 17 or newer, while ShedLock 7.x supports modern Spring Boot versions. Older Spring Boot 2.x applications require an earlier ShedLock generation. Check the compatibility matrix before selecting a version.\nThe example assumes that a JDBC DataSource is already configured.\nDatabase Schema ShedLock\u0026rsquo;s JDBC provider uses a table to store lock information. The following schema works with PostgreSQL. Other databases may require different timestamp types, and ShedLock provides database-specific DDL in its documentation.\nCREATE TABLE shedlock ( name VARCHAR(64) PRIMARY KEY, lock_until TIMESTAMP NOT NULL, locked_at TIMESTAMP NOT NULL, locked_by VARCHAR(255) NOT NULL ); Each row represents a named lock. The lock_until column determines when the lock expires, while locked_at and locked_by record when and where it was acquired. The primary key prevents duplicate lock records with the same name, and ShedLock uses conditional database operations to coordinate lock acquisition between competing application instances.\nSpring Boot Configuration Next, enable Spring scheduling and configure ShedLock to use the application\u0026rsquo;s existing database connection:\n@Configuration @EnableScheduling @EnableSchedulerLock(defaultLockAtMostFor = \u0026#34;10m\u0026#34;) public class SchedulingConfiguration { @Bean public LockProvider lockProvider(DataSource dataSource) { return new JdbcTemplateLockProvider( JdbcTemplateLockProvider.Configuration.builder() .withJdbcTemplate(new JdbcTemplate(dataSource)) .usingDbTime() .build() ); } } The @EnableScheduling annotation activates Spring\u0026rsquo;s scheduling support, while @EnableSchedulerLock enables ShedLock and defines a default maximum lock duration.\nThe usingDbTime() option uses the database server\u0026rsquo;s clock for lock timestamps rather than relying on individual application clocks. It also uses database-specific SQL that helps avoid insert conflicts during lock acquisition. ShedLock recommends enabling this option for supported databases.\nThis does not synchronize Spring\u0026rsquo;s scheduled triggers across application instances. ShedLock still assumes reasonably synchronized application clocks, and all instances must use the same logically consistent database and lock table.\nScheduling a Cluster Singleton With ShedLock configured, we can modify the cleanup task:\n@Component public class CleanupJob { @Scheduled(cron = \u0026#34;0 */5 * * * *\u0026#34;) @SchedulerLock( name = \u0026#34;cleanupJob\u0026#34;, lockAtLeastFor = \u0026#34;1m\u0026#34; ) public void cleanup() { LockAssert.assertLocked(); // Remove expired records } } ShedLock\u0026rsquo;s default PROXY_METHOD mode wraps the method in a Spring AOP proxy. Scheduled methods should be public and non-final. The lock applies to calls intercepted by the proxy, including direct calls through that proxy, not just scheduled invocations. Calls from within the same class bypass the proxy and therefore bypass ShedLock.\nThe @Scheduled annotation triggers the task every five minutes on every application instance, while @SchedulerLock coordinates execution through the shared database.\nThe lock parameters control its behavior:\nname identifies the shared lock. Every instance must use the same name for the same task, while different tasks should use distinct names. lockAtMostFor defines when the lock expires if it is not released. Here, it inherits the ten-minute default from @EnableSchedulerLock. lockAtLeastFor keeps the lock held for a minimum duration, even if the task finishes earlier. This is useful for short-running tasks because scheduled triggers on different instances may not fire at precisely the same time. The LockAssert.assertLocked() call verifies that the method is executing under a ShedLock lock, helping detect configuration problems. It will also throw when the method is called directly in unit tests. ShedLock provides LockAssert.TestHelper.makeAllAssertsPass(true) to disable these assertions on the current thread during testing.\nIn this example, the lock remains held for at least one minute and expires after ten minutes at the latest. If three application instances trigger the task simultaneously, only one acquires the lock and executes the method. The others skip execution.\nOnce the task completes and the minimum lock duration has elapsed, the lock becomes available for the next scheduled execution.\nShedLock does not queue skipped executions. If an instance fails to acquire the lock, that scheduled invocation is simply skipped.\nWhat Happens When an Instance Fails? Distributed locks need a way to recover when the instance holding a lock becomes unavailable. Suppose Instance A acquires the cleanup lock and then crashes. Without an expiration mechanism, the lock could remain held indefinitely, preventing other instances from executing the task.\nShedLock addresses this using time-limited locks, sometimes called leases. The lockAtMostFor setting establishes an upper bound on how long a lock remains valid. If Instance A crashes, the lock eventually expires, allowing another instance to acquire it on a subsequent scheduled trigger.\nThis provides recovery without requiring an external process to detect and release abandoned locks.\nWhen a Task Exceeds Its Lock Duration Consider the following sequence:\nInstance A acquires a lock with a ten-minute expiration. Instance A begins executing the scheduled task. The task takes longer than expected, or the instance experiences a long pause. After ten minutes, the lock expires. Instance B acquires the lock and starts executing the same task. Instance A continues or resumes its original execution. Both instances may now execute the task concurrently. The database has not granted the same unexpired lock to two instances. Rather, Instance A has continued executing after losing ownership of its lock.\nExclusive lock ownership does not necessarily guarantee exclusive task execution.\nIncreasing lockAtMostFor reduces the likelihood of this scenario but does not eliminate it. ShedLock also provides a KeepAliveLockProvider that can periodically extend the lock while a task is running. However, lease renewal cannot guarantee exclusivity if an instance becomes paused or disconnected long enough for the lock to expire.\nFor this reason, lockAtMostFor should be set comfortably above the expected maximum execution time, and scheduled tasks should tolerate occasional duplicate or overlapping execution.\nProduction Considerations Idempotency Where possible, scheduled tasks should be idempotent. An idempotent operation can be executed repeatedly without producing additional effects beyond the first successful execution.\nFor example, deleting expired records is generally safe to repeat, provided the deletion logic does not introduce other side effects. Operations such as charging accounts or sending notifications require additional care because repeating them may produce duplicate business effects.\nLock Duration Choose lockAtMostFor based on the task\u0026rsquo;s expected worst-case execution time, including unusually slow database operations and external dependencies. A duration that is too short increases the risk of overlapping execution, while one that is unnecessarily long delays recovery after a crashed instance.\nDatabase Operations ShedLock relies on atomic conditional updates to coordinate lock acquisition and does not require serializable transaction isolation. PostgreSQL\u0026rsquo;s default READ COMMITTED isolation is sufficient. Higher isolation levels provide no additional benefit for lock acquisition and may introduce serialization errors under contention.\nThe database\u0026rsquo;s replication guarantees also matter. With asynchronous replication, an acknowledged lock acquisition may be lost during failover, allowing another instance to acquire the same lock while the original task is still running. This applies to any primary-replica setup with asynchronous failover, regardless of geographic distribution. Properly configured synchronous or consensus-based replication can prevent this loss of acknowledged writes.\nShedLock also depends on the availability of its shared database. If the database becomes unavailable, instances may be unable to acquire new locks, preventing scheduled tasks from running.\nAvoid manually deleting rows from the lock table while applications are running. ShedLock caches lock-row existence and may not recreate deleted rows until the application restarts.\nMonitoring Monitor task duration, failures, skipped executions, and whether important tasks complete within their expected intervals. ShedLock also offers a Micrometer integration for lock-related metrics.\nWhen Should You Use ShedLock? ShedLock is a good fit when multiple application instances need to coordinate periodic tasks without introducing a dedicated distributed scheduler.\nTypical examples include:\nPeriodic cleanup and data housekeeping. Cache refreshes. Scheduled imports and synchronization. Maintenance operations. Batch jobs that tolerate occasional duplicate execution. ShedLock is not an exactly-once execution mechanism. If a task exceeds its lock duration, another instance may begin executing it before the original instance has finished. Workloads requiring stronger guarantees may need fencing tokens, transactional ownership validation, or idempotency.\nShedLock is intentionally lightweight and does not provide durable job scheduling or execution queues. If every scheduled execution must be tracked and eventually processed, consider a distributed scheduler such as db-scheduler or JobRunr.\nConclusion ShedLock provides a straightforward way to coordinate scheduled tasks across Spring Boot instances using a shared database, without introducing a dedicated scheduling service. It works well for periodic tasks that tolerate occasional overlapping execution, but operations where duplicate effects are unacceptable require additional correctness mechanisms at the application or resource level.\n","permalink":"https://blog.cloudneutral.se/cluster-singleton-scheduling-with-spring-boot-and-shedlock/","summary":"\u003cp\u003eSpring Boot makes it easy to schedule recurring tasks using the \u003ccode\u003e@Scheduled\u003c/code\u003e annotation. This works well for a single\napplication instance, but what happens when the application is deployed across multiple servers or regions?\u003c/p\u003e\n\u003cp\u003eEach instance runs its own scheduler, meaning the same task may execute concurrently on several instances. For some\nworkloads, this is harmless. For others, such as importing data, publishing notifications, or performing periodic\ncleanup, concurrent execution can cause duplicate processing or conflicting updates.\u003c/p\u003e","title":"Cluster Singleton Scheduling with Spring Boot and ShedLock"},{"content":"What happens when a client submits a payment or funds transfer, the server successfully processes it, but the response is lost?\nFrom the client\u0026rsquo;s perspective, the outcome is unknown. Retrying the request might be necessary, but it could also execute the same operation twice.\nThis is where idempotency becomes an important property of API design. An idempotent operation can be repeated without producing additional intended side effects beyond those of the first execution.\nHTTP methods such as GET, PUT, and DELETE are defined as idempotent, but POST is not. Nevertheless, APIs can implement safeguards that make POST operations safer to retry.\nIn this article, we\u0026rsquo;ll explore two approaches:\nStateless preconditions, where a token derived from resource versions determines whether an operation may execute. Idempotency keys, where the server records a unique operation identifier to recognize and deduplicate repeated requests. We\u0026rsquo;ll use a funds-transfer example to demonstrate both approaches, examine their trade-offs, and discuss the guarantees and limitations of each.\nHTTP Safety and Idempotency HTTP defines two important properties for request methods: safety and idempotency.\nA method is safe when its defined semantics are essentially read-only. Safe requests should not cause state changes requested by the client, although incidental effects such as logging and metrics are permitted.\nA method is idempotent when multiple identical requests have the same intended effect on the server as a single request.\nThe common HTTP methods have the following properties:\nMethod Safe Idempotent GET Yes Yes HEAD Yes Yes OPTIONS Yes Yes PUT No Yes DELETE No Yes POST No No PATCH No No These are the methods\u0026rsquo; defined semantics. Individual POST or PATCH operations can still be implemented idempotently.\nAlthough PATCH is not inherently idempotent, some patch operations are. For example, setting a field to a specific value can be idempotent, while incrementing a counter generally is not.\nFor example, consider deleting a resource:\nDELETE /accounts/123 The first request might return:\nHTTP/1.1 204 No Content A subsequent request might return:\nHTTP/1.1 404 Not Found The responses differ, but the intended effect is the same: the resource no longer exists.\nIdempotency concerns the intended effect of an operation, not whether repeated requests return identical responses.\nThis distinction becomes particularly important when implementing idempotent POST operations.\nWhy Idempotency Matters Distributed systems communicate over networks, and networks are unreliable.\nA client can send a request without knowing whether the server received it, processed it, or successfully committed its changes.\nConsider a funds transfer between two accounts:\nsequenceDiagram participant C as Client participant S as Server C-\u003e\u003eS: POST /transfers activate S Note right of S: Validate transfer Note right of S: Debit account A Note right of S: Credit account B Note right of S: Commit transaction S--xC: 201 Created (response lost) deactivate S Note over C: Timeout The transfer completed successfully, but the response was lost.\nThe client now faces an ambiguous outcome.\nDid the server fail before processing the request? Did the transaction commit? Was only the response lost?\nA timeout cannot answer these questions.\nThis is a fundamental problem in distributed systems: the absence of a response does not imply the absence of an effect.\nThe client might retry:\nPOST /transfers Content-Type: application/json { \u0026#34;from\u0026#34;: \u0026#34;A\u0026#34;, \u0026#34;to\u0026#34;: \u0026#34;B\u0026#34;, \u0026#34;amount\u0026#34;: 100 } Without additional safeguards, the server could process the same transfer again, moving another 100 units between the accounts.\nA reliable API should allow clients to recover from uncertain outcomes without accidentally duplicating business operations.\nThere are different ways to address this, each with its own guarantees.\nExample: Transferring Funds Between Accounts Our running example is a transfer of 100 units from account A to account B:\n{ \u0026#34;from\u0026#34;: \u0026#34;A\u0026#34;, \u0026#34;to\u0026#34;: \u0026#34;B\u0026#34;, \u0026#34;amount\u0026#34;: 100 } The operation must satisfy several invariants:\nThe source account must have sufficient funds. The total amount debited must equal the total amount credited. The debit and credit must commit atomically. Retrying the same logical transfer must not create additional transfers. The first three requirements concern the correctness of the transfer itself.\nThe fourth concerns request idempotency.\nWe\u0026rsquo;ll examine two approaches to addressing it.\nApproach 1: Stateless Preconditions A stateless precondition allows the server to determine whether an operation may execute based on the current state of the affected resources, without storing a history of previous requests.\nThe idea is similar to optimistic concurrency control.\nHTTP provides standardized conditional requests through headers such as If-Match, which compares an entity tag (ETag) against the selected representation of the target resource.\nFor a transfer involving multiple accounts, however, we want to validate the combined state of several resources.\nRather than treating a multi-resource token as an ETag for the /transfers collection, we\u0026rsquo;ll use an application-defined precondition header.\nUnlike If-Match, this header defines an application-specific precondition over multiple resources. The API uses 412 Precondition Failed to indicate that the supplied token no longer matches the current resource versions.\nGenerating a Resource-State Token Suppose two accounts have the following state:\nAccount A: Balance: 500 Version: 12 Account B: Balance: 200 Version: 8 Each account has a version number that changes whenever its relevant state is modified.\nThe client first requests a precondition token:\nPOST /transfer-preconditions Content-Type: application/json { \u0026#34;from\u0026#34;: \u0026#34;A\u0026#34;, \u0026#34;to\u0026#34;: \u0026#34;B\u0026#34; } The server reads the current account versions and derives a token:\ntoken = H(accountA.id, accountA.version, accountB.id, accountB.version) Here, H represents a deterministic hash function over an unambiguous, canonical encoding of the values.\nThe server returns:\nHTTP/1.1 200 OK Content-Type: application/json { \u0026#34;token\u0026#34;: \u0026#34;a8f3c9...\u0026#34; } The token represents the observed versions of both accounts.\nIt is not a record of an operation, nor does it prove that a transfer has previously occurred.\nBecause the server can recompute the token from current resource versions, it does not need to store the token itself.\nThe token is a concurrency-control mechanism, not an authorization credential. A deterministic hash does not prevent clients from constructing tokens if the underlying values are predictable. If the token must be tamper-resistant, the server can use a keyed construction such as HMAC, but authorization must still be enforced independently.\nExecuting the Transfer The client submits the transfer with the token:\nPOST /transfers Transfer-Precondition: \u0026#34;a8f3c9...\u0026#34; Content-Type: application/json { \u0026#34;from\u0026#34;: \u0026#34;A\u0026#34;, \u0026#34;to\u0026#34;: \u0026#34;B\u0026#34;, \u0026#34;amount\u0026#34;: 100 } Transfer-Precondition is an illustrative, application-defined header, not a standardized HTTP conditional header.\nThe server validates the token against the current account versions before executing the transfer.\nConceptually:\nBEGIN TRANSACTION Read current account versions expected = H(accountA.id, accountA.version, accountB.id, accountB.version) IF supplied token != expected: ROLLBACK RETURN 412 Precondition Failed Validate transfer invariants Debit account A Credit account B Increment account versions COMMIT RETURN 201 Created The important requirement is that precondition validation and the transfer execute atomically.\nOtherwise, two concurrent requests could both validate the same token before either updates the accounts.\nThe transaction must ensure that only one conflicting operation can commit against a particular set of versions.\nDatabases providing serializable transaction isolation, such as CockroachDB, are well suited to this pattern. Serializable isolation ensures that concurrent transactions produce results equivalent to some serial execution order. Two conflicting transactions cannot both commit based on the same stale resource versions when validation and modification occur within the transaction.\nIn CockroachDB, contention may cause serialization failures that require retrying the transaction. The precondition must be re-evaluated on each attempt using the original client-supplied token, ensuring that a retry cannot silently proceed against resource state that has since changed.\nWhat Happens on a Retry? Suppose the transfer succeeds.\nThe accounts now have the following state:\nAccount A: Balance: 400 Version: 13 Account B: Balance: 300 Version: 9 The original token no longer matches the current account versions.\nIf the client retries using the same token:\nPOST /transfers Transfer-Precondition: \u0026#34;a8f3c9...\u0026#34; Content-Type: application/json { \u0026#34;from\u0026#34;: \u0026#34;A\u0026#34;, \u0026#34;to\u0026#34;: \u0026#34;B\u0026#34;, \u0026#34;amount\u0026#34;: 100 } The server rejects the request:\nHTTP/1.1 412 Precondition Failed No additional transfer is performed.\nHowever, this response does not prove that the original transfer succeeded.\nAnother operation could have modified either account before the first transfer was executed.\nA failed precondition means the relevant resource state has changed, not that a particular request was previously processed.\nThis is the fundamental distinction between state-based preconditions and request deduplication.\nResource Versions and the ABA Problem Why use account versions rather than hashing account balances directly?\nConsider:\nInitial balance: 100 Token: H(100) Transfer A: -20 Balance: 80 Transfer B: +20 Balance: 100 Token: H(100) The account has returned to its original balance, making the original token valid again.\nThis is known as the ABA problem, named after the sequence A → B → A, where a value changes and subsequently returns to its original state, making an intervening modification undetectable through value comparison alone.\nA monotonically increasing resource version avoids this particular issue:\nBalance 100, version 1 Balance 80, version 2 Balance 100, version 3 Even though the balance returns to 100, the original version is no longer current.\nFor concurrency-sensitive operations, versions are therefore generally preferable to hashes derived solely from mutable business values.\nGuarantees and Limitations Stateless preconditions offer several advantages:\nNo need to store idempotency keys or previous responses. Straightforward validation against current resource versions. Protection against executing operations against stale resource state. Compatibility with optimistic concurrency control. However, there are important limitations:\nThe client must obtain a token before executing the operation. The token must cover all relevant resource state. Validation and modification must be atomic. A failed precondition does not reveal whether the original operation succeeded. The server cannot replay the original response without separately recording the operation\u0026rsquo;s outcome. There is also an important distinction between preventing a duplicate effect and making an operation idempotent.\nA state-based precondition can prevent a repeated transfer from executing after the first transfer changes the protected versions.\nHowever, the mechanism is fundamentally conditional execution, not recognition of a previously submitted logical operation.\nFor clients that need to recover the outcome of a request after a timeout, this limitation matters.\nApproach 2: Idempotency Keys An idempotency key identifies a logical operation independently of how many times the client submits it.\nInstead of obtaining a token derived from resource state, the client generates a unique identifier and includes it with the request:\nPOST /transfers Idempotency-Key: \u0026#34;8d27c0f1-7b24-4f9e-8d38-0e7b6d0d18a4\u0026#34; Content-Type: application/json { \u0026#34;from\u0026#34;: \u0026#34;A\u0026#34;, \u0026#34;to\u0026#34;: \u0026#34;B\u0026#34;, \u0026#34;amount\u0026#34;: 100 } The server associates the key with the operation and its outcome.\nIf the same request is submitted again using the same key, the server recognizes it as a retry rather than a new transfer.\nUnlike stateless preconditions, idempotency keys identify the operation itself, not the current state of the affected resources.\nRequest Deduplication A typical idempotency record contains:\nIdempotency record ------------------ Key Request fingerprint Status Response status Response body Created at Expires at The request fingerprint represents the relevant parameters of the original operation.\nThe key should also be scoped appropriately, for example to a particular client or tenant, so unrelated clients cannot accidentally share the same namespace.\nWhen a request arrives, the server distinguishes four situations:\nSituation Server behavior New key Accept the operation for execution Existing key, different request Reject conflicting key reuse Existing key, operation in progress Wait or return an in-progress response Existing key, operation completed Return the previously recorded result This distinction matters because duplicate requests can arrive while the original request is still executing, not only after it has completed.\nAtomic Execution A common mistake is to check whether an idempotency key exists before executing the business operation:\nIF key does not exist: Execute transfer Store key This is unsafe under concurrency.\nTwo requests can both observe that the key is absent and proceed to execute the transfer.\nThe server needs an atomic mechanism for claiming the key, typically backed by a database uniqueness constraint or equivalent concurrency control.\nConceptually:\nFUNCTION submitTransfer(key, request): fingerprint = fingerprint(request) BEGIN TRANSACTION result = atomicallyClaimKey(key, fingerprint) IF result == FINGERPRINT_MISMATCH: ROLLBACK RETURN 422 Unprocessable Content IF result == ALREADY_COMPLETED: ROLLBACK RETURN previouslyRecordedResponse Validate transfer invariants Debit source account Credit destination account response = createTransferResult() Store completed response with key COMMIT RETURN response The atomic claim must serialize concurrent requests using the same key. If another transaction currently holds the claim, the server can wait for its outcome or return a documented in-progress response. If that transaction aborts, a subsequent attempt may acquire the key and execute the operation.\nThis is illustrative pseudocode rather than a complete database algorithm.\nThe critical requirement is that the transfer and its completed idempotency record commit atomically.\nA database transaction can provide this guarantee when both the business operation and idempotency record reside in the same transactional database.\nThe IN_PROGRESS state requires additional care.\nAn implementation might register an in-progress operation before starting the business transaction, allowing concurrent requests to detect that execution is underway.\nIf the server crashes, that registration must not permanently prevent retries. A lease, timeout, or recovery mechanism is needed.\nAlternatively, an implementation can rely on transactional key claiming and database concurrency control without persisting a separate in-progress record. Concurrent requests may block or conflict until the original transaction commits or aborts.\nThe choice depends on the desired API behavior and persistence model.\nIn either case, the server must distinguish between:\nAn operation that is still executing. An operation that completed successfully. An operation that failed before committing any business effects. A failed attempt must not permanently consume an idempotency key unless the API deliberately records that failure as the operation\u0026rsquo;s final outcome.\nReplaying the Original Response Suppose the original request succeeds:\nHTTP/1.1 201 Created Location: /transfers/789 Content-Type: application/json { \u0026#34;id\u0026#34;: \u0026#34;789\u0026#34;, \u0026#34;status\u0026#34;: \u0026#34;completed\u0026#34; } The response is lost.\nThe client retries with the same idempotency key:\nPOST /transfers Idempotency-Key: \u0026#34;8d27c0f1-7b24-4f9e-8d38-0e7b6d0d18a4\u0026#34; Content-Type: application/json { \u0026#34;from\u0026#34;: \u0026#34;A\u0026#34;, \u0026#34;to\u0026#34;: \u0026#34;B\u0026#34;, \u0026#34;amount\u0026#34;: 100 } The server finds the completed operation and returns its recorded result:\nHTTP/1.1 201 Created Location: /transfers/789 Content-Type: application/json { \u0026#34;id\u0026#34;: \u0026#34;789\u0026#34;, \u0026#34;status\u0026#34;: \u0026#34;completed\u0026#34; } No additional transfer is executed.\nThis is the primary advantage of idempotency keys over resource-state preconditions: the server can recognize a previously completed logical operation and return its outcome, even if the underlying resources have changed since then.\nAn API could instead return 200 OK with the existing result. Both behaviors are possible, but the response semantics should be documented and consistent.\nKey Scope and Retention Idempotency keys cannot necessarily be retained forever.\nA server might retain them for a configurable period, such as 24 hours, after which the records are eligible for deletion.\nThis creates an important limitation:\nThe deduplication guarantee applies only while the idempotency record remains valid and available.\nIf a client retries after the retention window, the server may treat the request as a new operation.\nThe API should therefore define:\nHow keys are generated and scoped. How long keys remain valid. What happens when a key is reused with different parameters. How concurrent requests using the same key are handled. Whether completed responses are replayed. Which operation outcomes are retained. The retention period should reflect realistic client retry behavior and the business consequences of duplicate execution.\nFor financial operations, an application may also require durable business-level uniqueness constraints independent of temporary HTTP idempotency records.\nIdempotency Is Not Exactly-Once Execution Idempotency keys are sometimes described as providing exactly-once semantics.\nThat terminology deserves care.\nA request handler may execute multiple times because of client retries, server restarts, or database transaction retries.\nWhat matters is that the intended business effect commits no more than once for a particular logical operation.\nFor example:\nRequest attempt 1 -\u0026gt; Transaction aborted Request attempt 2 -\u0026gt; Transaction committed Request attempt 3 -\u0026gt; Original result returned The operation was attempted three times, but its business effect committed once.\nThis is better described as at-most-once committed effects, within the defined scope and retention period.\nIt does not guarantee that an operation eventually succeeds. That depends on retries, availability, and other failure-handling mechanisms.\nComparing the Approaches Both stateless preconditions and idempotency keys can help prevent unintended duplicate effects, but they provide different guarantees.\nProperty Stateless preconditions Idempotency keys Primary purpose Optimistic concurrency control Request deduplication Identifies Resource state or version Logical operation Requires initial request Usually No, if client generates the key Requires server-side operation history No Yes Protects against concurrent execution With atomic precondition validation With atomic key claiming Recognizes an earlier operation No Yes Can replay the original response Not inherently Yes, if retained Requires retention management Not necessarily Yes Typical retry outcome 412 Precondition Failed Previously recorded result Main limitation Cannot establish why state changed Requires storage and lifecycle management When to Use Stateless Preconditions Stateless preconditions are useful when the client intends to execute an operation against a particular version of one or more resources.\nTypical examples include:\nUpdating a document without overwriting another user\u0026rsquo;s changes. Modifying account configuration only if it has not changed. Executing an operation against a known set of resource versions. The primary concern is whether the relevant resource state has changed since the client last observed it.\nWhen to Use Idempotency Keys Idempotency keys are a better fit when clients need to retry an operation safely after an uncertain outcome.\nTypical examples include:\nPayments and funds transfers. Order creation. Booking requests. Job submissions. Other operations that must not be duplicated. The primary concern is whether a particular logical operation has already been performed.\nCan They Be Combined? Yes. The two mechanisms address different concerns and can complement each other.\nFor example:\nPOST /transfers Idempotency-Key: \u0026#34;abc-123\u0026#34; Transfer-Precondition: \u0026#34;a8f3c9...\u0026#34; Content-Type: application/json { \u0026#34;from\u0026#34;: \u0026#34;A\u0026#34;, \u0026#34;to\u0026#34;: \u0026#34;B\u0026#34;, \u0026#34;amount\u0026#34;: 100 } The idempotency key identifies the logical transfer across retries.\nThe precondition ensures that the initial execution occurs only against the expected resource state.\nA retry of an already completed operation can return its recorded result without re-evaluating the original precondition.\nThe API must define how these mechanisms interact, including which checks take precedence.\nConclusion Reliable APIs must account for requests whose outcomes become uncertain because of network failures.\nStateless preconditions protect operations against changes to relevant resource state. They are useful for optimistic concurrency control but cannot identify whether a particular request has already executed.\nIdempotency keys identify logical operations across retries, enabling request deduplication and recovery of previously recorded outcomes. They provide stronger retry semantics at the cost of maintaining server-side state.\nBoth mechanisms require correct transactional boundaries and concurrency control, and they can be combined when an operation needs both state validation and safe retries.\nThe goal isn\u0026rsquo;t to prevent retries. It\u0026rsquo;s to make them safe and predictable.\n","permalink":"https://blog.cloudneutral.se/designing-idempotent-rest-apis-stateless-preconditions-and-idempotency-keys/","summary":"How do you safely retry a POST request when the outcome is unknown? A comparison of stateless preconditions and idempotency keys, using funds transfers as an example.","title":"Designing Idempotent REST APIs: Stateless Preconditions and Idempotency Keys"},{"content":"Several business drivers can justify adopting a multi-region deployment strategy:\nBusiness continuity: Maintain service availability during data centre or regional disruptions. Customer experience: Serve customers worldwide with predictable, low latency. Scalability: Adapt capacity to different markets and changing traffic volumes. Regulatory compliance: Address data residency, operational resilience, and concentration risk requirements. None of this is free. A multi-region deployment typically costs more than a single-region one, and that cost has to be weighed against the availability and performance it buys.\nA multi-region deployment does not necessarily require a multi-active architecture. Traditional active-passive systems can span multiple regions, with one serving production traffic and another standing by for disaster recovery.\nMulti-active systems take a different approach. Multiple deployment sites serve production traffic simultaneously, allowing geographically distributed infrastructure to contribute to normal operations rather than sitting idle until a failure occurs.\nMulti-Active Systems A multi-active system serves production traffic concurrently from multiple data centres or regions.\nWorkloads are distributed across deployment sites based on factors such as geographic proximity, capacity, and data locality. Each site actively contributes to the production workload rather than acting exclusively as a standby environment.\nThe term covers two different designs.\nIn the first, each site accepts writes independently and replicates them asynchronously, resolving conflicting changes after the fact. Last-write-wins is a common conflict resolution strategy, but it is not information-preserving: concurrent updates may be discarded even when successfully acknowledged. A site failure can also result in the loss of acknowledged writes that have not yet been replicated.\nIn the second, database replicas coordinate through consensus before writes are acknowledged. This article focuses on this model, where consensus-based replication provides the foundation for durability and transactional consistency.\nIf a data centre becomes unavailable, traffic can be redirected to other active sites. Since those sites already serve production traffic, recovery does not require activating an entirely separate environment.\nThis does not mean failover disappears. Load balancers must detect failures, connections may be interrupted, and stateful services may need to relocate ownership of resources. The difference is that these mechanisms are part of normal system operation rather than a separate disaster recovery procedure.\nMulti-active architectures can improve infrastructure utilization, but enough spare capacity must remain to absorb the workload of a failed site.\nAssuming equally sized sites and evenly distributed workloads, two sites must each operate at no more than 50% capacity to survive losing either one. With three sites, each can operate at roughly 67%.\nState and Consistency Distributing stateless application services is relatively straightforward. Managing shared transactional state across multiple locations is considerably harder.\nConsider a financial transaction that transfers funds between two accounts. Regardless of which region processes the request, the system must preserve account balances, apply the transfer atomically, and handle concurrent transactions correctly.\nSome of the challenges include:\nPreserving transactional invariants during concurrency and failures. Coordinating access to shared state across regions. Handling ambiguous transaction outcomes and retries. Preventing duplicate business effects during event processing. Managing replication, data locality, and consistency guarantees. Applications can implement these mechanisms themselves, but delegating much of this responsibility to the database reduces application complexity.\nDistributed SQL databases such as CockroachDB provide transactional consistency and consensus-based replication across multiple failure domains. Applications can work with SQL transactions while the database manages replication, concurrency control, and recovery from supported infrastructure failures.\nApplication-level concerns such as idempotency and transaction retries still need attention, but the database provides a consistent foundation for shared transactional state.\nConsensus and Failure Tolerance Consensus-based replication allows a group of replicas to agree on changes even when some replicas become unavailable.\nFor example, a three-voter consensus group distributed across three independent failure domains can tolerate the loss of one voter while retaining a majority quorum. A five-voter group can tolerate two voter failures.\nTo tolerate the complete loss of any one failure domain while retaining a majority quorum, voting replicas must be distributed across at least three independent failure domains. With only two regions, one must hold a majority of voting replicas. Losing that region makes the affected data unavailable.\nHowever, deploying a database across three regions does not automatically mean it can survive the loss of any one region.\nFault tolerance depends on replica placement, quorum requirements, and whether the remaining replicas can communicate.\nCockroachDB, for example, supports both zone and region survival goals. Zone survival is the default and protects against the loss of an availability zone. Region survival requires at least three regions and uses a replication configuration designed to maintain quorum after losing a region.\nA network partition has similar consequences. Nodes in an isolated region may continue running, but without access to a quorum, they cannot make progress on writes or serve ordinary strongly consistent reads for the affected data. Active does not mean autonomous.\nStrong consistency also has a latency cost. Writes must be replicated to a majority of voting replicas, which can introduce cross-region round trips depending on replica placement. Consistent reads are typically served by a designated replica (the leaseholder, in CockroachDB) without a quorum round trip, but clients in other regions still pay the network latency to reach it.\nCareful data placement can reduce this overhead by keeping frequently accessed data close to the applications using it.\nFailover-Based Systems To contrast multi-active systems, consider traditional active-passive architectures.\nAn active-passive system serves production traffic from a designated primary data centre or region. A secondary environment maintains replicated data and sufficient infrastructure to take over if the primary becomes unavailable.\nThe architecture is typically designed around the assumption that one location owns the active workload.\nDuring a disruption, recovery may involve promoting a secondary database, redirecting traffic, and ensuring the original primary can no longer accept conflicting writes.\nOnce the disruption is resolved, operations may need to be transferred back to the original site.\nThis approach has several limitations:\nUnderutilized infrastructure: Standby environments consume resources without necessarily contributing to production capacity. Potential data loss: Asynchronous replication can leave acknowledged writes missing from the secondary site, and they are lost if it is promoted. Recovery time: Promoting replicas and redirecting traffic takes time. Operational complexity: Failover procedures must account for incomplete replication, network partitions, and the risk of split-brain behavior. Testing difficulties: Disaster recovery procedures may be exercised infrequently, allowing configuration drift and hidden dependencies to accumulate. These limitations are not universal. Active-passive systems can provide effective disaster recovery, particularly when recovery requirements do not justify the complexity of a multi-active deployment.\nHowever, multi-active architectures offer an alternative where resilience is built into normal production operations rather than depending primarily on standby capacity and recovery procedures.\nDisaster Recovery Spectrum The objective of a disaster recovery strategy is to minimize service disruption and data loss following a significant failure.\nTwo common measures are:\nRecovery Time Objective (RTO): The maximum acceptable time required to restore service. Recovery Point Objective (RPO): The maximum acceptable amount of data loss, expressed as a period of time. Both are targets set by the business. What an architecture delivers is the achieved recovery time and recovery point, and the approaches below differ in what drives those.\nDisaster recovery approaches range from backups to continuously operating multi-active deployments.\nBackups Data is periodically backed up to separate infrastructure or cloud storage.\nRecovery time depends on how long restoration takes, while the recovery point depends on backup frequency and available recovery mechanisms, such as transaction log archiving.\nBackups remain essential even for highly available systems because replication does not protect against logical corruption or accidental deletion. It copies both faithfully.\nCold Standby A minimally provisioned secondary environment can take over after the primary fails.\nRecovery may require provisioning infrastructure, restoring state, or starting application services. This reduces infrastructure costs but generally results in longer recovery times.\nWhen state is restored from backups, the recovery point is set by backup frequency, as above.\nWarm Standby A secondary environment is maintained in a more operationally ready state, often scaled down and with continuously replicated data.\nRecovery is generally faster than with cold standby, at the expense of higher infrastructure costs.\nHot Standby A fully provisioned secondary environment runs continuously with replicated data and can take over with little preparation. It often serves read-only workloads in the meantime.\nRecovery is faster again, and the cost approaches that of a second production environment.\nWherever a standby is fed by replication, the recovery point depends on the replication strategy. Asynchronous replication introduces potential data loss. Synchronous replication can eliminate the risk of losing acknowledged writes under supported failure scenarios, at the price of coordination latency and potentially reduced availability.\nMulti-Active Multiple deployment sites serve production traffic simultaneously.\nWhen one site fails, surviving sites can continue processing workloads, provided they have sufficient capacity and access to the required state.\nRecovery time is not zero. Failure detection, the transfer of leadership for affected data, and client reconnects typically take seconds, but no separate environment has to be activated.\nWith synchronous replication, an RPO of zero is achievable for acknowledged writes under supported failure scenarios. Asynchronous replication provides no such guarantee.\nRecovery beyond the system\u0026rsquo;s fault tolerance may require accepting data loss. Backups remain necessary to recover from logical corruption, operator mistakes, and more severe failures.\nMulti-Region Deployments Distributing infrastructure across independent failure domains reduces exposure to individual infrastructure failures. Multiple availability zones can protect against zone-level disruptions, while geographically separated regions can extend resilience to regional failures.\nHowever, an application deployed across several regions may still depend on a database or another critical service hosted entirely within one region.\nA resilient multi-region architecture must therefore account for the availability of its critical dependencies, not just the application tier.\nMulti-active systems address part of this challenge by allowing applications to serve requests from multiple regions while a distributed database manages shared transactional state and replication.\nThe remaining trade-offs concern latency, data locality, regulatory requirements, and operational costs.\nData Locality and Latency Data placement is one way to manage the latency costs of geographic distribution.\nKeeping region-specific data close to the applications and customers accessing it can reduce cross-region communication.\nFor example, customer data associated with European operations might primarily reside in Europe, while North American customer data is placed closer to North American workloads.\nGlobally shared data may require different placement and replication strategies.\nRegulatory requirements can impose additional constraints. Data residency rules may restrict where information can be stored or replicated, potentially limiting the options available for regional resilience.\nCockroachDB provides multi-region capabilities for configuring data placement and survival requirements, allowing these trade-offs to be managed at the database level.\nMulti-Cloud Resilience Multi-region deployments within a single cloud provider may still be vulnerable to provider-wide failures or shared infrastructure dependencies. Multi-cloud architectures can reduce this concentration risk by distributing workloads across independent providers.\nThis is particularly relevant to UK financial institutions, where PRA regulations on operational resilience and third-party risk management require banks to assess concentration risk and maintain critical services during disruptions. Multi-cloud is not a regulatory requirement, but one possible strategy for addressing these concerns.\nOperating across cloud providers introduces additional complexity, including network latency, data transfer costs, and differences in managed services. Infrastructure portability and deployment flexibility become important considerations when designing multi-active systems across clouds.\nOperational Considerations Multi-active deployments reduce dependence on traditional failover procedures, but they introduce other operational requirements.\nTraffic routing must account for regional health and available capacity. Surviving regions need enough resources to absorb displaced workloads.\nExternal dependencies must also be considered. A multi-active application can still become unavailable if it relies on a single-region identity provider, message broker, or integration service.\nFinally, failure handling needs regular testing.\nRegional isolation, interrupted transactions, replication failures, and recovery after network partitions should be treated as expected operating conditions rather than exceptional scenarios.\nConclusion Multi-active architectures allow multiple deployment sites to serve production workloads concurrently, reducing dependence on traditional failover procedures and, with enough sites, making better use of infrastructure capacity.\nCombined with multi-region deployment and appropriate replication, they can improve resilience against regional disruptions while bringing applications closer to geographically distributed customers.\nThe main challenges lie in managing shared state, transactional consistency, and data locality. Distributed databases such as CockroachDB can address much of this complexity through consensus-based replication and transactional guarantees.\nThe objective is not to eliminate infrastructure failures, but to design systems that can continue operating when those failures occur.\n","permalink":"https://blog.cloudneutral.se/multi-active-systems-and-multi-region-architectures/","summary":"\u003cp\u003eSeveral business drivers can justify adopting a multi-region deployment strategy:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eBusiness continuity:\u003c/strong\u003e Maintain service availability during data centre or regional disruptions.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eCustomer experience:\u003c/strong\u003e Serve customers worldwide with predictable, low latency.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eScalability:\u003c/strong\u003e Adapt capacity to different markets and changing traffic volumes.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eRegulatory compliance:\u003c/strong\u003e Address data residency, operational resilience, and concentration risk requirements.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eNone of this is free. A multi-region deployment typically costs more than a single-region one, and that cost has to be\nweighed against the availability and performance it buys.\u003c/p\u003e","title":"Multi-Active Systems and Multi-Region Architectures"},{"content":"Originally published in April 2022. Updated October 2026.\nFinding the best route between two points sounds simple until the map contains hundreds of intersections, roads with different travel costs, and areas that should be avoided altogether. This is the shortest-path problem: given a graph and a cost for traversing each edge, find a path whose total cost is minimal.\nIn this article, we\u0026rsquo;ll use a small graph to explore two classic shortest-path algorithms: Dijkstra\u0026rsquo;s algorithm and A search*. We\u0026rsquo;ll start with the fundamentals of graph theory and briefly explore breadth-first search (BFS), depth-first search (DFS), and Bellman–Ford before examining Dijkstra and A* step by step.\nIf you\u0026rsquo;re already familiar with graph theory, feel free to jump directly to the algorithm walkthroughs. The later sections cover heuristic selection, Java implementation details, and a practical pathfinding example from Arma 3.\nThe accompanying shortest-path demo project includes visualizations and implementations.\nWhy pathfinding matters in games Imagine an AI-controlled squad moving from one position to another without unnecessary exposure to enemy fire. The shortest geometric route may cross a minefield or a defended position. A longer route along a road might be faster, safer, or both.\nA route planner can assign costs to road segments and intersections based on distance, terrain, vehicle accessibility, or danger. The algorithm then minimizes the sum of those costs, not necessarily the physical distance.\nScreenshots from Arma 3.\nGraph theory primer A graph consists of vertices (also called nodes) and edges connecting them. In a road network, intersections can be vertices and road segments can be edges.\nAn edge may be directed (travel is allowed only one way) or undirected (travel is allowed both ways). A graph is cyclic if it contains a cycle; otherwise, it is acyclic. A tree is a connected, acyclic, undirected graph.\nA grid is another useful graph representation: cells are vertices and permitted moves between neighboring cells are edges. Four-way and eight-way movement produce different edge sets.\nEdge weights and path costs For a weighted graph, each edge has a weight representing the cost of traversing it. A route\u0026rsquo;s cost is the sum of its edge weights:\ncost(A → B → D → F) = w(A,B) + w(B,D) + w(D,F) Weights can represent travel time, distance, fuel consumption, or a composite measure. Their meaning matters: a straight-line distance is a useful estimate of remaining cost only when the edge-weight model makes that estimate a lower bound.\nThe algorithms at a glance Algorithm How it explores Shortest-path guarantee Typical use BFS Expands by number of edges from the start Yes, for unweighted or equal-weight edges Grid movement with uniform costs DFS Explores a branch before backtracking No Traversal, connectivity, maze exploration Dijkstra Expands the vertex with the lowest known cost from the start Yes, with nonnegative edge weights Weighted routes, shortest paths from one source A* Expands the vertex with the lowest g + h estimate Yes, with appropriate heuristic and search handling Goal-directed pathfinding Bellman–Ford Repeatedly relaxes edges Yes, absent reachable negative cycles Graphs with negative edge weights All these algorithms traverse graphs, but they answer somewhat different questions. DFS is not a shortest-path algorithm in the usual sense; BFS is optimal only under uniform edge costs; and Bellman–Ford trades speed for the ability to handle negative weights and detect reachable negative cycles.\nBreadth-first search (BFS) BFS visits vertices in layers: first those one edge away, then two edges away, and so on. A FIFO queue ensures that the first time a vertex is discovered, BFS has found a path to it using the fewest edges.\nBFS can stop when it discovers the goal. It does not have to visit every vertex, but it may explore a large portion of the graph. For unequal edge costs, the fewest edges need not be the cheapest route.\nDepth-first search (DFS) DFS follows one branch until it reaches a dead end, then backtracks and explores alternatives. It is useful for graph traversal, reachability, and some maze-solving strategies, but the first path it finds is not necessarily the shortest.\nDFS can reach a goal quickly on some graphs and take much longer on others; it is not generally faster than BFS.\nDijkstra\u0026rsquo;s algorithm Dijkstra\u0026rsquo;s algorithm, developed by Edsger W. Dijkstra in the 1950s, solves the single-source shortest-path problem for graphs with nonnegative edge weights. It works by maintaining the cheapest currently known distance from the start to each vertex.\nConsider the following graph. Our starting vertex is A, our destination is F, and the red numbers indicate edge weights.\nThe relevant edges and weights are:\nA—B: 3 A—C: 2 B—C: 4 B—D: 6 C—E: 8 D—E: 3 D—F: 5 E—F: 4 We\u0026rsquo;ll track two things for each vertex:\nTentative distance dist[v]: the best cost found so far from A to v. Predecessor prev[v]: the vertex immediately before v on that best-known path. Initially, dist[A] = 0, all other distances are infinity, and no predecessors are set. At each step, select the unsettled vertex with the smallest tentative distance, relax its outgoing edges, and then settle it.\nRelaxing an edge u → v means checking whether dist[u] + weight(u,v) improves dist[v]. If it does, update both the distance and predecessor.\nStep 1: Start at A A has distance 0. Its neighbors are B and C:\ndist[B] = min(∞, 0 + 3) = 3 prev[B] = A dist[C] = min(∞, 0 + 2) = 2 prev[C] = A Settle A. Among the remaining vertices, C has the smallest tentative distance (2), so it is next.\nStep 2: Expand C From C, consider B and E:\nvia C to B = 2 + 4 = 6 (keep dist[B] = 3) via C to E = 2 + 8 = 10 (set dist[E] = 10, prev[E] = C) Settle C. The smallest unsettled tentative distance now belongs to B (3).\nStep 3: Expand B B has a distance of 3. Its relevant unsettled neighbor is D:\nvia B to D = 3 + 6 = 9 (set dist[D] = 9, prev[D] = B) Settle B. Next is D (9), ahead of E (10).\nStep 4: Expand D From D, consider E and F:\nvia D to E = 9 + 3 = 12 (keep dist[E] = 10) via D to F = 9 + 5 = 14 (set dist[F] = 14, prev[F] = D) We have discovered a route to F, but we cannot stop yet. A tentative distance is not final until that vertex is selected as the minimum-distance unsettled vertex.\nSettle D. The next vertex is E (10), not F (14).\nStep 5: Expand E, then settle F E offers another route to F:\nvia E to F = 10 + 4 = 14 This ties the existing distance of 14, so we can keep prev[F] = D. Settle E. Now F is the unsettled vertex with the smallest tentative distance. When we remove F from the priority queue, its distance is final and we may stop.\nReconstruct the shortest path Follow predecessor pointers backward from F:\nF ← D ← B ← A Reverse the sequence:\nA → B → D → F Total cost = 3 + 6 + 5 = 14 There is also an equally cheap path:\nA → C → E → F Total cost = 2 + 8 + 4 = 14 Dijkstra\u0026rsquo;s algorithm may return either, depending on how equal-cost alternatives and predecessor updates are handled.\nWhy Dijkstra works With nonnegative weights, once a vertex has the smallest tentative distance among all unsettled vertices, no route through the remaining unsettled vertices can improve it. That is why its distance can be finalized. This reasoning breaks down when negative-weight edges are allowed.\nA* search: adding direction to the search Dijkstra\u0026rsquo;s algorithm expands outward according to the known cost from the start. If we care about one destination, we can often avoid exploring irrelevant areas by estimating which vertices are closer to the goal.\nA* does this with three values:\ng(n) = cheapest known cost from the start to n h(n) = estimated remaining cost from n to the goal f(n) = g(n) + h(n) Dijkstra prioritizes g(n); A* prioritizes f(n). The heuristic changes exploration order, not actual edge weights or accumulated path cost. When relaxing an edge, always calculate the new g from the current vertex\u0026rsquo;s g, never from its f.\nThe original illustration shows the idea of heuristic estimates. The numerical walkthrough below uses a separate, admissible heuristic table rather than the historical values printed in that image.\nChoosing a heuristic An admissible heuristic never overestimates the true remaining cost. A consistent heuristic additionally satisfies h(u) ≤ w(u,v) + h(v) for every edge u → v, with h(goal) = 0.\nConsistency is useful for a graph-search implementation with a closed set: when a vertex is settled, its best path is already known. With an admissible but inconsistent heuristic, an implementation may need to reopen previously closed vertices to preserve optimality.\nEuclidean (straight-line) distance can be a good heuristic when edge costs represent distance and no route can cost less than its straight-line length. If costs measure time, danger, or another quantity, the heuristic must be scaled or designed for that cost model.\nCommon A* heuristics The effectiveness of A* depends heavily on the heuristic function. For spatial graphs, a common approach is to estimate the remaining cost using the geometric distance between the current node and the destination.\nSeveral distance functions are commonly used, depending on the movement model and how edge costs are defined.\nManhattan distance Suitable for movement restricted to four directions (horizontal and vertical), where each step has equal cost.\nh = |dx| + |dy| Manhattan distance measures the total horizontal and vertical displacement. It is an exact distance on an obstacle-free, four-direction grid with unit movement costs.\nEuclidean distance The straight-line distance between two points.\nh = sqrt(dx² + dy²) Euclidean distance is useful for spatial graphs where edge weights represent geometric distance. It provides a lower bound on the actual travel distance because no route between two points can be shorter than the straight-line distance.\nIt is also applicable to grid-based movement, although more specialized heuristics may provide tighter estimates.\nChebyshev distance Suitable for eight-direction movement where diagonal and straight moves have equal cost.\nh = max(|dx|, |dy|) Because diagonal movement reduces both horizontal and vertical displacement simultaneously, the minimum number of steps is determined by the larger coordinate difference.\nOctile distance Suitable for eight-direction movement where straight moves cost 1 and diagonal moves cost √2.\nh = max(|dx|, |dy|) + (√2 - 1) × min(|dx|, |dy|) Octile distance gives the exact shortest-path cost on an obstacle-free, eight-direction grid with unit straight-move costs and diagonal costs of √2. It therefore provides a tighter heuristic than Euclidean distance for this movement model.\nChoosing the right heuristic Here, dx and dy represent the differences in horizontal and vertical coordinates between the current node and the destination.\nThe appropriate heuristic depends on the graph\u0026rsquo;s connectivity and how movement costs are defined:\nManhattan: Four-direction movement with uniform costs. Euclidean: Geometric distance in spatial graphs. Chebyshev: Eight-direction movement with equal straight and diagonal costs. Octile: Eight-direction movement with diagonal costs proportional to geometric distance. These formulas assume the movement costs described above. If actual movement costs differ, the heuristic must be scaled or adapted accordingly.\nIn practice, a more accurate estimate of the remaining cost can significantly reduce the number of vertices A* explores, provided the heuristic remains admissible.\nA* walkthrough on the same graph Let\u0026rsquo;s reuse vertices A through F and the same edge weights as in the Dijkstra example. For this demonstration, choose these consistent, admissible estimates of the remaining cost to F:\nVertex h(n) True cheapest remaining cost A 10 14 B 11 11 C 8 12 D 5 5 E 4 4 F 0 0 These values are illustrative estimates, not distances inferred from the drawing\u0026rsquo;s coordinates. Notice that h(B) = 11 and h(D) = 5 are exact, while h(C) = 8 is a lower bound.\nStep 1: Expand A Start with g(A) = 0 and f(A) = 0 + 10 = 10. Relax its edges:\nB: g = 3, h = 11, f = 14 C: g = 2, h = 8, f = 10 A* chooses C next because its f is smaller. Dijkstra also chose C, but because its g was smaller.\nStep 2: Expand C Calculate tentative actual path costs from g(C) = 2:\nB via C: tentative g = 2 + 4 = 6 (keep B\u0026#39;s g = 3) E via C: tentative g = 2 + 8 = 10 (set g(E) = 10) E: g = 10, h = 4, f = 14 Now B and E both have f = 14. A* may choose either. To illustrate goal-directed search, suppose our tie-breaking rule chooses E.\nStep 3: Expand E From E, we can reach D and F:\nD via E: g = 10 + 3 = 13, h = 5, f = 18 F via E: g = 10 + 4 = 14, h = 0, f = 14 The open set contains B (f = 14), F (f = 14), and D (f = 18). If ties favor the goal, F is selected next and the search ends with:\nA → C → E → F Total cost = 2 + 8 + 4 = 14 The other optimal path, A → B → D → F, also costs 14. A* did not make the route through B more expensive; it simply explored the route through C first.\nStep 4: What if ties are resolved differently? If A* selects B before F, it can discover D with g(D) = 9, improving the earlier value of 13. The eventual result is still optimal. Which equal-cost path is reconstructed depends on tie-breaking and predecessor-update rules.\nThis small graph doesn\u0026rsquo;t demonstrate a dramatic speedup, and that\u0026rsquo;s fine. A* becomes particularly useful on large maps where a good heuristic directs the search toward the goal and avoids much of the area Dijkstra would otherwise explore.\nWhen A* behaves like Dijkstra Set h(n) = 0 for every vertex:\nf(n) = g(n) + 0 = g(n) A* then uses the same priority as Dijkstra. A poorly chosen heuristic may provide little benefit; an overestimating heuristic can sacrifice the shortest-path guarantee.\nGame example: road-network pathfinding in Arma 3 The practical motivation for this project was pathfinding over an Arma 3 road network. Road segments become vertices, and connectivity between nearby segments becomes graph edges. Once the road graph has been constructed, a shortest-path algorithm can search for a route between two positions.\nIn this example, the green markers show road segments in the graph. A preparatory traversal discovers the connected road network; the pathfinding search then operates on that graph. Those are two separate operations: graph construction and shortest-path search.\nA recorded Arma 3 demonstration is available in the project repository.\nDijkstra in Arma script (SQF) The original implementation uses hash tables for tentative distances and predecessors, an unvisited collection, and helper functions to retrieve neighbors and edge weights. Its central relaxation step is:\nprivate _currentDist = [_distanceHash, _current] call CBA_fnc_hashGet; private _unvisitedNeighbors = [_edges, _visitedList, _current] call FUNC(unvisitedNeighbors); { private _neighbor = _x; private _edgeWeight = [_edges, _current, _neighbor] call FUNC(edgeWeight); private _totalDist = _currentDist + _edgeWeight; private _tentativeDist = [_distanceHash, _neighbor] call CBA_fnc_hashGet; if (_totalDist \u0026lt; _tentativeDist) then { [_distanceHash, _neighbor, _totalDist] call CBA_fnc_hashSet; // Update predecessor and add neighbor to the unvisited set. }; } forEach _unvisitedNeighbors; The interesting part is the same as in the mathematical walkthrough: only a cheaper actual path changes a vertex\u0026rsquo;s tentative distance. The full Arma 3 implementation includes the supporting graph and predecessor bookkeeping.\nOne implementation detail matters: if the goal is selected as the lowest-cost unsettled vertex, early termination is safe. Merely encountering the goal as a neighbor is not sufficient.\nJava implementation: Dijkstra and A* The companion Java project demonstrates shortest-path algorithms through a graphical visualization of the search process.\nAlthough Dijkstra and A* use different strategies for selecting the next vertex to explore, they share the same fundamental relaxation rule: update a vertex\u0026rsquo;s best-known distance whenever a cheaper path is discovered.\nThe key difference is how vertices are prioritized:\nDijkstra: Uses the accumulated path cost g(n). A*: Uses f(n) = g(n) + h(n), combining the accumulated cost with an estimate of the remaining distance. The following Java snippets illustrate these principles independently of the repository\u0026rsquo;s implementation.\nThe shared relaxation rule For either algorithm, the core update should look conceptually like this:\ndouble tentativeG = current.g() + edgeWeight(current.node(), neighbor); if (tentativeG \u0026lt; bestG.getOrDefault(neighbor, Double.POSITIVE_INFINITY)) { bestG.put(neighbor, tentativeG); predecessor.put(neighbor, current.node()); double f = tentativeG + heuristic(neighbor, goal); openSet.add(new QueueEntry\u0026lt;\u0026gt;(neighbor, tentativeG, f)); } The distinction between g and f is crucial. The edge relaxation compares g values; the priority queue orders by f for A*, or by g when the heuristic is zero.\nA Java PriorityQueue pitfall Java\u0026rsquo;s PriorityQueue does not automatically reorder an entry when the mutable object used as its sort key changes. If a vertex\u0026rsquo;s g or f is modified after it has entered the queue, the queue\u0026rsquo;s heap ordering may become invalid.\nA robust approach is to enqueue immutable snapshots and discard entries whose g is no longer current:\nrecord QueueEntry\u0026lt;N\u0026gt;(N node, double g, double f) {} PriorityQueue\u0026lt;QueueEntry\u0026lt;N\u0026gt;\u0026gt; openSet = new PriorityQueue\u0026lt;\u0026gt;( Comparator.comparingDouble(QueueEntry\u0026lt;N\u0026gt;::f) ); bestG.put(start, 0.0); openSet.add(new QueueEntry\u0026lt;\u0026gt;(start, 0.0, heuristic(start, goal))); while (!openSet.isEmpty()) { QueueEntry\u0026lt;N\u0026gt; entry = openSet.poll(); // A better route was found after this snapshot was queued. if (entry.g() \u0026gt; bestG.getOrDefault(entry.node(), Double.POSITIVE_INFINITY)) { continue; } // With a consistent heuristic (or Dijkstra), the first valid // removal of the goal is the optimal stopping point. if (entry.node().equals(goal)) { break; } for (N neighbor : neighbors(entry.node())) { double tentativeG = entry.g() + edgeWeight(entry.node(), neighbor); if (tentativeG \u0026lt; bestG.getOrDefault(neighbor, Double.POSITIVE_INFINITY)) { bestG.put(neighbor, tentativeG); predecessor.put(neighbor, entry.node()); openSet.add(new QueueEntry\u0026lt;\u0026gt;( neighbor, tentativeG, tentativeG + heuristic(neighbor, goal) )); } } } This is an illustrative algorithm fragment, not a complete standalone class: it assumes bestG, predecessor, neighbors, edgeWeight, heuristic, start, and goal are defined. It also assumes nonnegative edge weights and, for A*\u0026rsquo;s early-exit guarantee, a consistent heuristic. The immutable-entry approach avoids mutating keys already inside the priority queue.\nThe original Java demo source provides a practical demonstration of pathfinding and its visualization. The examples above illustrate the underlying algorithmic principles and safe priority-queue handling rather than reproducing the project\u0026rsquo;s source code.\nChoosing an algorithm For a small, unweighted graph, BFS is usually the simplest shortest-path solution. For weighted graphs with nonnegative costs, Dijkstra is a reliable default. When you have a specific destination and a meaningful lower-bound estimate, A* can greatly reduce the search space. If negative edge weights are required, Bellman–Ford is one option; Dijkstra and ordinary A* assumptions no longer apply.\nFor a game map, the algorithm is only part of the design. The graph representation, edge-cost model, heuristic, and frequency of route recalculation often matter just as much as the choice between Dijkstra and A*.\nConclusion Dijkstra and A* solve the same fundamental problem: finding a lowest-cost route through a graph. Dijkstra prioritizes the cost already incurred; A* combines that cost with an estimate of what remains. With a suitable heuristic and correct queue handling, A* can reach the same optimal answer while examining fewer vertices.\nThe important details are easy to miss in a high-level description: a discovered goal is not necessarily settled, heuristic estimates are not actual travel costs, and priority queues must reflect updated priorities correctly. Once those distinctions are clear, the algorithms become much easier to reason about—and to apply to real road networks, games, and other routing problems.\nSource and demos: github.com/cloudneutral/shortestpath\n","permalink":"https://blog.cloudneutral.se/shortest-path-algorithms/","summary":"\u003cp\u003e\u003cem\u003eOriginally published in April 2022. Updated October 2026.\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eFinding the best route between two points sounds simple until the map contains hundreds of intersections, roads with\ndifferent travel costs, and areas that should be avoided altogether. This is the \u003cem\u003eshortest-path problem\u003c/em\u003e: given a graph\nand a cost for traversing each edge, find a path whose \u003cstrong\u003etotal cost is minimal\u003c/strong\u003e.\u003c/p\u003e\n\u003cp\u003eIn this article, we\u0026rsquo;ll use a small graph to explore two classic shortest-path algorithms: \u003cstrong\u003eDijkstra\u0026rsquo;s algorithm\u003c/strong\u003e and\n\u003cem\u003e\u003cem\u003eA\u003c/em\u003e search\u003c/em\u003e*. We\u0026rsquo;ll start with the fundamentals of graph theory and briefly explore breadth-first search (BFS),\ndepth-first search (DFS), and Bellman–Ford before examining Dijkstra and A* step by step.\u003c/p\u003e","title":"Shortest Path Algorithms: Dijkstra, A* and Pathfinding in Games"},{"content":"I\u0026rsquo;m Kai Niemi, a software engineer and architect based in Stockholm. I\u0026rsquo;ve spent over 20 years building software, with a particular focus on distributed systems, transactional databases, and the challenges of consistency, concurrency, and failure handling.\nMy background spans hands-on software development, technical leadership, enterprise architecture, and pre-sales engineering.\nAbout CloudNeutral CloudNeutral is where I share technical articles, experiments, and lessons learned from building and working with distributed systems.\nI write about distributed databases, transactional consistency, database performance, Java and Spring Boot, event-driven architecture, and the open-source projects I maintain.\nMostly it\u0026rsquo;s about trade-offs: what a design costs, what it buys, and what breaks when the assumptions change.\nElsewhere GitHub LinkedIn Disclaimer The views and opinions expressed on this blog are my own and do not necessarily reflect those of my current or former employers or any other organizations I\u0026rsquo;m affiliated with.\nPrivacy \u0026amp; Analytics CloudNeutral uses Cloudflare Web Analytics to collect anonymous, aggregated website statistics. The service does not use cookies or track visitors across websites. The site is hosted on GitHub Pages, which may process technical information necessary to deliver its content.\n","permalink":"https://blog.cloudneutral.se/about/","summary":"\u003cp\u003eI\u0026rsquo;m Kai Niemi, a software engineer and architect based in Stockholm. I\u0026rsquo;ve spent over 20 years building software, with a\nparticular focus on distributed systems, transactional databases, and the challenges of consistency, concurrency, and\nfailure handling.\u003c/p\u003e\n\u003cp\u003eMy background spans hands-on software development, technical leadership, enterprise architecture, and pre-sales\nengineering.\u003c/p\u003e\n\u003ch2 id=\"about-cloudneutral\"\u003eAbout CloudNeutral\u003c/h2\u003e\n\u003cp\u003eCloudNeutral is where I share technical articles, experiments, and lessons learned from building and working with\ndistributed systems.\u003c/p\u003e\n\u003cp\u003eI write about distributed databases, transactional consistency, database performance, Java and Spring Boot,\nevent-driven architecture, and the open-source projects I maintain.\u003c/p\u003e","title":"About"}]