Some business drivers that could justify adopting a multi-active and multi-region deployment strategy:
Securing business continuity in the event of regional data centre disruptions
Deliver a good customer experience worldwide
Deliver business adaptability and scalability to different market needs/volumes
Compliance with data locality/placement regulations
Sustainable operational costs as the business grows
Multi-active systems are a prerequisite to effectively adopting a multi-region strategy. Let's find out how.
Multi-Active Systems
A multi-active system is capable to operate and serve online traffic simultaneously from multiple active data centres and regions. With that comes characteristics that will satisfy the goals stated above without adding too much infrastructure/app complexity and cost overhead.
Multi-active systems run live in multiple datacenters in different regions all the time and workloads are dynamically shared across these datacenters. A multi-active system process requests simultaneously for either domestic or global markets without any assumptions on locality, traffic affinity or replication delays.
There are no actual concepts of traffic failover or failback. Instead, failures and disruptions are handled transparently through regional and/or global load balancing and traffic rerouting. If one failure domain (data centre or region) begins to fall behind due to disruptions, parts of its workload can move to other domains transparently. If one domain is completely offline, all its work is rebalanced to the remaining domains.
Using three failure domains allows one to go completely dark, while still allowing systems to make forward progress through consensus decisions. Using five failure domains allows two domains to go dark, and so on. This allows for systems to be both highly available and always consistent and correct.
A multi-active system is not without certain challenges:
Needs a fit-for-purpose solution to manage state and replication (data distribution at a global scale is a difficult problem) such as:
Higher service latency due to mandatory cross-datacenter coordination
Existing design assumptions on single DC/region deployment
Constraints around auxiliary system integrations
Breaking current assumptions on system design
To contain complexity, most of these challenges are preferably pushed down to the resource tier - the database - instead of being managed in the app tier. CockroachDB is one such system with a wide range of multi-region deployment options and first-class support for crafting multi-active systems.
Failover-based Systems
To contrast multi-active systems, let's quickly look at the main predecessor: singly-homed, failover-based systems. A singly-homed system is crafted to operate and serve online traffic from a single data centre or at most a single region. Crafted in terms of design choices, technology selections, network assumptions and supporting infrastructure components.
In the event of a primary domain disruption or disaster, a singly-homed system may failover traffic to an alternative secondary data centre. After the disruption is cleared away, it may "fail back" traffic to the original primary domain.
This type of setup has many limitations:
Unable to scale horizontally beyond a single data centre/region
Unable to load-balance traffic freely across multiple active datacenters
Dependent on standby, underutilized resources, increasing TCO (300% capacity for steady state)
Must use asynchronous replication for availability and performance, with the risk of data loss
Long recovery times after failures
Complex and error-prone failover protocols with manual checkpoints/sign-offs
Unclear when and if a standby system can resume traffic from a safe point
Difficult and risky to test and verify that the protocol works
Disaster Recovery Spectrum
The main objective of a disaster recovery plan is to minimize the time it takes to recover from a severe disruption event and reduce the amount of data loss and other business impacts.
The spectrum of disaster recovery solutions typically ranges from offline backups to full-blown multi-region deployments, also with backups.
Backups - Data is frequently backed up and sent off-side or to cloud storage.
The recovery time objective (RTO) is governed by the time it takes to restore the database to a new setup
The recovery point objective (RPO) is governed by the frequency of incremental and full backups
Cold Standby - A minimally provisioned environment with the ability to take over core services from a failed primary data centre.
Higher TCO due to under-utilized standby capacity
RTO is governed by how fast a switchover can be made to the secondary
RPO is governed by the async replication delay from the primary to the secondary
Warm Standby - A fully provisioned environment with the ability to take over a failed primary data centre.
Higher TCO due to excessive amounts of under-utilized standby capacity
Could serve certain read-only traffic at the same time as the primary
RTO and RPO are quite similar to a cold standby, only a bit lower due to higher readiness
Multi-Active - Each deployment site serves production traffic simultaneously.
All data centres provide traffic at the same time for the entire keyspace
There is no actual notion of fail-over or fail-back, failures and recovery are handled transparently towards the app tier
RTO is governed by how quickly an isolated or crashed node can drop its authority over reads and writes to local data (typically a few seconds)
RPO is zero due to consensus-based replication
Multi-active systems stand out from most fail-over-based models in terms of cost and complexity reduction. It's far more resilient against different categories of disruptions, but not immune to disasters. If a multi-active system loses a majority of its failure domains (like 2 zones in a 3-zone region) or if some operator error corrupts a database, then the music stops. Therefore, backups are still commonly used alongside multi-active systems, which adds a safety harness for recovery.
Combined with multiple regions, the blast radius is extended to cover most conditions and you can also improve customer experience for a global market.
Multi-region deployments
One data centre is a single point of failure, similar to a single region. If that data centre/region goes offline for a longer period without any recovery option, it may have a severe impact on the business and the company's reputation.
Adding two or more data centres to a single region will increase the blast radius and decrease the likelihood of severe, long-lasting service disruptions due to a single DC outage.
Deploying a system (as in many services/components working in concert) across multiple, geo-separated regions extends the blast radius even further. Single-region assumptions cannot however be transferred to this new ecosystem due to how we traditionally manage state and consistency. Leveraging multi-region effectively requires a multi-active system architecture. Not exclusively, but it's very much a state/database undertaking that needs a fit-for-purpose solution like CockroachDB.
Summary
This article discusses the advantages of adopting a region-level deployment strategy for businesses, focusing on multi-active systems. These systems operate simultaneously across multiple data centres, providing increased resiliency and adaptability to market needs. The article also contrasts multi-active systems with traditional failover-based systems and examines the disaster recovery spectrum, including backups, cold standby, warm standby, and multi-active solutions. Ultimately, multi-active systems offer significant benefits in terms of cost and complexity reduction, while still requiring backups to ensure data safety.
