Configure Replication Zones

On this page Carat arrow pointing down

Replication zones give you the power to control what data goes where in your CockroachDB cluster. Specifically, they are used to control the number and location of replicas for data belonging to the following objects:

For each of the above objects you can control:

  • How many copies of each range to spread through the cluster.
  • Which constraints are applied to which data, e.g., "table X's data can only be stored in the German datacenters".
  • The maximum size of ranges (how big ranges get before they are split).
  • How long old data is kept before being garbage collected.
  • New in v2.1: Where you would like the leaseholders for certain ranges to be located, e.g., "for ranges that are already constrained to have at least one replica in region=us-west, also try to put their leaseholders in region=us-west".

This page explains how replication zones work and how to use the CONFIGURE ZONE statement to manage them.

Note:

Currently, only members of the admin role can configure replication zones. By default, the root user belongs to the admin role.

Overview

Every range in the cluster is part of a replication zone. Each range's zone configuration is taken into account as ranges are rebalanced across the cluster to ensure that any constraints are honored.

When a cluster starts, there are two categories of replication zone:

  1. Pre-configured replication zones that apply to internal system data.
  2. A single default replication zone that applies to the rest of the cluster.

You can adjust these pre-configured zones as well as add zones for individual databases, tables, rows, and secondary indexes as needed. Note that adding zones for rows and secondary indexes is enterprise-only.

For example, you might rely on the default zone to spread most of a cluster's data across all of your datacenters, but create a custom replication zone for a specific database to make sure its data is only stored in certain datacenters and/or geographies.

Replication zone levels

There are five replication zone levels for table data in a cluster, listed from least to most granular:

Level Description
Cluster CockroachDB comes with a pre-configured .default replication zone that applies to all table data in the cluster not constrained by a database, table, or row-specific replication zone. This zone can be adjusted but not removed. See View the Default Replication Zone and Edit the Default Replication Zone for more details.
Database You can add replication zones for specific databases. See Create a Replication Zone for a Database for more details.
Table You can add replication zones for specific tables. See Create a Replication Zone for a Table.
Index (Enterprise-only) The secondary indexes on a table will automatically use the replication zone for the table. However, with an enterprise license, you can add distinct replication zones for secondary indexes. See Create a Replication Zone for a Secondary Index for more details.
Row (Enterprise-only) You can add replication zones for specific rows in a table or secondary index by defining table partitions. See Create a Replication Zone for a Table Partition for more details.

For system data

In addition, CockroachDB stores internal system data in what are called system ranges. There are two replication zone levels for this internal system data, listed from least to most granular:

Level Description
Cluster The .default replication zone mentioned above also applies to all system ranges not constrained by a more specific replication zone.
System Range CockroachDB comes with pre-configured replication zones for important system ranges, such as the "meta" and "liveness" ranges. If necessary, you can add replication zones for the "timeseries" range and other system ranges as well. See Create a Replication Zone for a System Range for more details.

CockroachDB also comes with pre-configured replication zones for the internal system database and the system.jobs table, which stores metadata about long-running jobs such as schema changes and backups.

Level priorities

When replicating data, whether table or system, CockroachDB always uses the most granular replication zone available. For example, for a piece of user data:

  1. If there's a replication zone for the row, CockroachDB uses it.
  2. If there's no applicable row replication zone and the row is from a secondary index, CockroachDB uses the secondary index replication zone.
  3. If the row isn't from a secondary index or there is no applicable secondary index replication zone, CockroachDB uses the table replication zone.
  4. If there's no applicable table replication zone, CockroachDB uses the database replication zone.
  5. If there's no applicable database replication zone, CockroachDB uses the .default cluster-wide replication zone.
Note:

Changes to the .default cluster-wide replication zone are not automatically applied to existing replication zones, including pre-configured zones for important system ranges that must remain available for the cluster as a whole to remain available. The zones for these system ranges have an initial replication factor of 5 to make them more resilient to node failure. However, if you increase the .default zone's replication factor above 5, consider increasing the replication factor for important system ranges as well.

Manage replication zones

Use the CONFIGURE ZONE statement to add, modify, reset, and remove replication zones.

Replication zone variables

Use the ALTER ... CONFIGURE ZONE statement to set a replication zone:

icon/buttons/copy
> ALTER TABLE t CONFIGURE ZONE USING range_min_bytes = 0, range_max_bytes = 90000, gc.ttlseconds = 89999, num_replicas = 5, constraints = '[-region=west]';
Variable Description
range_min_bytes The minimum size, in bytes, for a range of data in the zone. When a range is less than this size, CockroachDB will merge it with an adjacent range.

Default: 1048576 (1MiB)
range_max_bytes The maximum size, in bytes, for a range of data in the zone. When a range reaches this size, CockroachDB will split it into two ranges.

Default: 67108864 (64MiB)
gc.ttlseconds The number of seconds overwritten values will be retained before garbage collection. Smaller values can save disk space if values are frequently overwritten; larger values increase the range allowed for AS OF SYSTEM TIME queries, also know as Time Travel Queries.

It is not recommended to set this below 600 (10 minutes); doing so will cause problems for long-running queries. Also, since all versions of a row are stored in a single range that never splits, it is not recommended to set this so high that all the changes to a row in that time period could add up to more than 64MiB; such oversized ranges could contribute to the server running out of memory or other problems.

Default: 90000 (25 hours)
num_replicas The number of replicas in the zone.

Default: 3

For the system database and .meta, .liveness, and .system ranges, the default value is 5.
constraints An array of required (+) and/or prohibited (-) constraints influencing the location of replicas. See Types of Constraints and Scope of Constraints for more details.

To prevent hard-to-detect typos, constraints placed on store attributes and node localities must match the values passed to at least one node in the cluster. If not, an error is signalled.

Default: No constraints, with CockroachDB locating each replica on a unique node and attempting to spread replicas evenly across localities.
lease_preferences An ordered list of required and/or prohibited constraints influencing the location of leaseholders. Whether each constraint is required or prohibited is expressed with a leading + or -, respectively. Note that lease preference constraints do not have to be shared with the constraints field. For example, it's valid for your configuration to define a lease_preferences field that does not reference any values from the constraints field. It's also valid to define a lease_preferences field with no constraints field at all.

If the first preference cannot be satisfied, CockroachDB will attempt to satisfy the second preference, and so on. If none of the preferences can be met, the lease will be placed using the default lease placement algorithm, which is to base lease placement decisions on how many leases each node already has, trying to make all the nodes have around the same amount.

Each value in the list can include multiple constraints. For example, the list [[+zone=us-east-1b, +ssd], [+zone=us-east-1a], [+zone=us-east-1c, +ssd]] means "prefer nodes with an SSD in us-east-1b, then any nodes in us-east-1a, then nodes in us-east-1c with an SSD."

For a usage example, see Constrain leaseholders to specific datacenters.

Default: No lease location preferences are applied if this field is not specified.

Replication constraints

The location of replicas, both when they are first added and when they are rebalanced to maintain cluster equilibrium, is based on the interplay between descriptive attributes assigned to nodes and constraints set in zone configurations.

Tip:
For demonstrations of how to set node attributes and replication constraints in different scenarios, see Scenario-based Examples below.

Descriptive attributes assigned to nodes

When starting a node with the cockroach start command, you can assign the following types of descriptive attributes:

Attribute Type Description
Node Locality Using the --locality flag, you can assign arbitrary key-value pairs that describe the locality of the node. Locality might include country, region, datacenter, rack, etc. The key-value pairs should be ordered from most inclusive to least inclusive (e.g., country before datacenter before rack), and the keys and the order of key-value pairs must be the same on all nodes. It's typically better to include more pairs than fewer. For example:

--locality=region=east,datacenter=us-east-1
--locality=region=east,datacenter=us-east-2
--locality=region=west,datacenter=us-west-1

CockroachDB attempts to spread replicas evenly across the cluster based on locality, with the order determining the priority. However, locality can be used to influence the location of data replicas in various ways using replication zones.

When there is high latency between nodes, CockroachDB also uses locality to move range leases closer to the current workload, reducing network round trips and improving read performance. See Follow-the-workload for more details.
Node Capability Using the --attrs flag, you can specify node capability, which might include specialized hardware or number of cores, for example:

--attrs=ram:64gb
Store Type/Capability Using the attrs field of the --store flag, you can specify disk type or capability, for example:

--store=path=/mnt/ssd01,attrs=ssd
--store=path=/mnt/hda1,attrs=hdd:7200rpm

Types of constraints

The node-level and store-level descriptive attributes mentioned above can be used as the following types of constraints in replication zones to influence the location of replicas. However, note the following general guidance:

  • When locality is the only consideration for replication, it's recommended to set locality on nodes without specifying any constraints in zone configurations. In the absence of constraints, CockroachDB attempts to spread replicas evenly across the cluster based on locality.
  • Required and prohibited constraints are useful in special situations where, for example, data must or must not be stored in a specific country or on a specific type of machine.
Constraint Type Description Syntax
Required When placing replicas, the cluster will consider only nodes/stores with matching attributes or localities. When there are no matching nodes/stores, new replicas will not be added. +ssd
Prohibited When placing replicas, the cluster will ignore nodes/stores with matching attributes or localities. When there are no alternate nodes/stores, new replicas will not be added. -ssd

Scope of constraints

Constraints can be specified such that they apply to all replicas in a zone or such that different constraints apply to different replicas, meaning you can effectively pick the exact location of each replica.

Constraint Scope Description Syntax
All Replicas Constraints specified using JSON array syntax apply to all replicas in every range that's part of the replication zone. constraints = '[+ssd, -region=west]'
Per-Replica Multiple lists of constraints can be provided in a JSON object, mapping each list of constraints to an integer number of replicas in each range that the constraints should apply to.

The total number of replicas constrained cannot be greater than the total number of replicas for the zone (num_replicas). However, if the total number of replicas constrained is less than the total number of replicas for the zone, the non-constrained replicas will be allowed on any nodes/stores.

See the Per-replica constraints example for more details.
constraints: '{"+ssd,-region=west": 2, "+region=east": 1}', num_replicas = 3

Node/replica recommendations

See Cluster Topography recommendations for production deployments.

View replication zones

Use the SHOW ZONE CONFIGURATIONS statement to view details about existing replication zones.

Basic examples

These examples focus on the basic approach and syntax for working with zone configuration. For examples demonstrating how to use constraints, see Scenario-based examples.

For more examples, see CONFIGURE ZONE and SHOW ZONE CONFIGURATIONS.

View all replication zones

icon/buttons/copy
> SHOW ALL ZONE CONFIGURATIONS;
   zone_name  |                     config_sql
+-------------+-----------------------------------------------------+
  .default    | ALTER RANGE default CONFIGURE ZONE USING
              |     range_min_bytes = 1048576,
              |     range_max_bytes = 67108864,
              |     gc.ttlseconds = 90000,
              |     num_replicas = 3,
              |     constraints = '[]',
              |     lease_preferences = '[]'
  system      | ALTER DATABASE system CONFIGURE ZONE USING
              |     range_min_bytes = 1048576,
              |     range_max_bytes = 67108864,
              |     gc.ttlseconds = 90000,
              |     num_replicas = 5,
              |     constraints = '[]',
              |     lease_preferences = '[]'
  system.jobs | ALTER TABLE system.public.jobs CONFIGURE ZONE USING
              |     range_min_bytes = 1048576,
              |     range_max_bytes = 67108864,
              |     gc.ttlseconds = 600,
              |     num_replicas = 5,
              |     constraints = '[]',
              |     lease_preferences = '[]'
  .meta       | ALTER RANGE meta CONFIGURE ZONE USING
              |     range_min_bytes = 1048576,
              |     range_max_bytes = 67108864,
              |     gc.ttlseconds = 3600,
              |     num_replicas = 5,
              |     constraints = '[]',
              |     lease_preferences = '[]'
  .system     | ALTER RANGE system CONFIGURE ZONE USING
              |     range_min_bytes = 1048576,
              |     range_max_bytes = 67108864,
              |     gc.ttlseconds = 90000,
              |     num_replicas = 5,
              |     constraints = '[]',
              |     lease_preferences = '[]'
  .liveness   | ALTER RANGE liveness CONFIGURE ZONE USING
              |     range_min_bytes = 1048576,
              |     range_max_bytes = 67108864,
              |     gc.ttlseconds = 600,
              |     num_replicas = 5,
              |     constraints = '[]',
              |     lease_preferences = '[]'
(6 rows)

For more information, see SHOW ZONE CONFIGURATIONS.

View the default replication zone

icon/buttons/copy
> SHOW ZONE CONFIGURATION FOR RANGE default;
  zone_name |                config_sql
+-----------+------------------------------------------+
  .default  | ALTER RANGE default CONFIGURE ZONE USING
            |     range_min_bytes = 1048576,
            |     range_max_bytes = 67108864,
            |     gc.ttlseconds = 90000,
            |     num_replicas = 3,
            |     constraints = '[]',
            |     lease_preferences = '[]'
(1 row)

For more information, see SHOW ZONE CONFIGURATIONS.

Edit the default replication zone

Note:

Changes to the .default cluster-wide replication zone are not automatically applied to existing replication zones, including pre-configured zones for important system ranges that must remain available for the cluster as a whole to remain available. The zones for these system ranges have an initial replication factor of 5 to make them more resilient to node failure. However, if you increase the .default zone's replication factor above 5, consider increasing the replication factor for important system ranges as well.

To edit the default replication zone, use the ALTER RANGE ... CONFIGURE ZONE statement to define the values you want to change (other values will remain the same):

icon/buttons/copy
> ALTER RANGE default CONFIGURE ZONE USING num_replicas = 5, gc.ttlseconds = 100000;
CONFIGURE ZONE 1
icon/buttons/copy
> SHOW ZONE CONFIGURATION FOR RANGE default;
  zone_name |                config_sql
+-----------+------------------------------------------+
  .default  | ALTER RANGE default CONFIGURE ZONE USING
            |     range_min_bytes = 1048576,
            |     range_max_bytes = 67108864,
            |     gc.ttlseconds = 100000,
            |     num_replicas = 5,
            |     constraints = '[]',
            |     lease_preferences = '[]'
(1 row)

For more information, see CONFIGURE ZONE.

Create a replication zone for a system range

In addition to the databases and tables that are visible via the SQL interface, CockroachDB stores internal data in what are called system ranges. CockroachDB comes with pre-configured replication zones for some of these ranges:

Zone Name Description
.meta The "meta" ranges contain the authoritative information about the location of all data in the cluster.

These ranges must retain a majority of replicas for the cluster as a whole to remain available and historical queries are never run on them, so CockroachDB comes with a pre-configured .meta replication zone with num_replicas set to 5 to make these ranges more resilient to node failure and a lower-than-default gc.ttlseconds to keep these ranges smaller for reliable performance.

If your cluster is running in multiple datacenters, it's a best practice to configure the meta ranges to have a copy in each datacenter.
.liveness The "liveness" range contains the authoritative information about which nodes are live at any given time.

These ranges must retain a majority of replicas for the cluster as a whole to remain available and historical queries are never run on them, so CockroachDB comes with a pre-configured .liveness replication zone with num_replicas set to 5 to make these ranges more resilient to node failure and a lower-than-default gc.ttlseconds to keep these ranges smaller for reliable performance.
.system There are system ranges for a variety of other important internal data, including information needed to allocate new table IDs and track the status of a cluster's nodes.

These ranges must retain a majority of replicas for the cluster as a whole to remain available, so CockroachDB comes with a pre-configured .system replication zone with num_replicas set to 5 to make these ranges more resilient to node failure.
.timeseries The "timeseries" ranges contain monitoring data about the cluster that powers the graphs in CockroachDB's Admin UI. If necessary, you can add a .timeseries replication zone to control the replication of this data.
Warning:

Use caution when editing replication zones for system ranges, as they could cause some (or all) parts of your cluster to stop working.

To control replication for one of the above sets of system ranges, use the ALTER RANGE ... CONFIGURE ZONE statement to define the values you want to change (other values will not be affected):

icon/buttons/copy
> ALTER RANGE meta CONFIGURE ZONE USING num_replicas = 7;
CONFIGURE ZONE 1
icon/buttons/copy
> SHOW ZONE CONFIGURATION FOR RANGE meta;
  zone_name |              config_sql
+-----------+---------------------------------------+
  .meta     | ALTER RANGE meta CONFIGURE ZONE USING
            |     range_min_bytes = 1048576,
            |     range_max_bytes = 67108864,
            |     gc.ttlseconds = 3600,
            |     num_replicas = 7,
            |     constraints = '[]',
            |     lease_preferences = '[]'
(1 row)

For more information, see CONFIGURE ZONE.

Create a replication zone for a database

To control replication for a specific database, use the ALTER DATABASE ... CONFIGURE ZONE statement to define the values you want to change (other values will not be affected):

icon/buttons/copy
> ALTER DATABASE test CONFIGURE ZONE USING num_replicas = 5, gc.ttlseconds = 100000;
CONFIGURE ZONE 1
icon/buttons/copy
> SHOW ZONE CONFIGURATION FOR DATABASE test;
  zone_name |                config_sql
+-----------+------------------------------------------+
  test      | ALTER DATABASE test CONFIGURE ZONE USING
            |     range_min_bytes = 1048576,
            |     range_max_bytes = 67108864,
            |     gc.ttlseconds = 100000,
            |     num_replicas = 5,
            |     constraints = '[]',
            |     lease_preferences = '[]'
(1 row)

For more information, see CONFIGURE ZONE.

Create a replication zone for a table

To control replication for a specific table, use the ALTER TABLE ... CONFIGURE ZONE statement to define the values you want to change (other values will not be affected):

icon/buttons/copy
> ALTER TABLE customers CONFIGURE ZONE USING num_replicas = 5, gc.ttlseconds = 100000;
CONFIGURE ZONE 1
icon/buttons/copy
> SHOW ZONE CONFIGURATION FOR TABLE customers;
    zone_name    |                 config_sql
+----------------+--------------------------------------------+
  test.customers | ALTER TABLE customers CONFIGURE ZONE USING
                 |     range_min_bytes = 1048576,
                 |     range_max_bytes = 67108864,
                 |     gc.ttlseconds = 100000,
                 |     num_replicas = 5,
                 |     constraints = '[]',
                 |     lease_preferences = '[]'
(1 row)

For more information, see CONFIGURE ZONE.

Create a replication zone for a secondary index

Note:

This is an enterprise-only feature.

The secondary indexes on a table will automatically use the replication zone for the table. However, with an enterprise license, you can add distinct replication zones for secondary indexes.

To control replication for a specific secondary index, use the ALTER INDEX ... CONFIGURE ZONE statement to define the values you want to change (other values will not be affected).

Tip:

To get the name of a secondary index, which you need for the CONFIGURE ZONE statement, use the SHOW INDEX or SHOW CREATE TABLE statements.

icon/buttons/copy
> ALTER INDEX tpch.frequent_customers CONFIGURE ZONE USING num_replicas = 5, gc.ttlseconds = 100000;
CONFIGURE ZONE 1
icon/buttons/copy
> SHOW ZONE CONFIGURATION FOR INDEX tpch.customer@frequent_customers;
             zone_name             |                                config_sql
+----------------------------------+--------------------------------------------------------------------------+
  tpch.customer@frequent_customers | ALTER INDEX tpch.public.customer@frequent_customers CONFIGURE ZONE USING
                                   |     range_min_bytes = 1048576,
                                   |     range_max_bytes = 67108864,
                                   |     gc.ttlseconds = 100000,
                                   |     num_replicas = 5,
                                   |     constraints = '[]',
                                   |     lease_preferences = '[]'
(1 row)

For more information, see CONFIGURE ZONE.

Create a replication zone for a table or secondary index partition

Note:

This is an enterprise-only feature.

To control replication for table partitions, use the ALTER PARTITION ... CONFIGURE ZONE statement to define the values you want to change (other values will not be affected):

icon/buttons/copy
> ALTER PARTITION north_america OF TABLE customers CONFIGURE ZONE USING num_replicas = 5, constraints = '[-region=EU]';
CONFIGURE ZONE 1
icon/buttons/copy
> SHOW ZONE CONFIGURATION FOR PARTITION north_america OF TABLE customers;
           zone_name           |                                  config_sql
+------------------------------+-------------------------------------------------------------------------------+
  test.customers.north_america | ALTER PARTITION north_america OF INDEX customers@primary CONFIGURE ZONE USING
                               |     range_min_bytes = 1048576,
                               |     range_max_bytes = 67108864,
                               |     gc.ttlseconds = 100000,
                               |     num_replicas = 5,
                               |     constraints = '[-region=EU]',
                               |     lease_preferences = '[]'
(1 row)
Tip:

Since the syntax is the same for defining a replication zone for a table or index partition (e.g., database.table.partition), give partitions names that communicate what they are partitioning, e.g., north_america_table vs north_america_idx1.

For more information, see CONFIGURE ZONE.

Reset a replication zone

icon/buttons/copy
> ALTER TABLE t CONFIGURE ZONE USING DEFAULT;
CONFIGURE ZONE 1

For more information, see CONFIGURE ZONE.

Remove a replication zone

icon/buttons/copy
> ALTER TABLE t CONFIGURE ZONE DISCARD;
CONFIGURE ZONE 1

For more information, see CONFIGURE ZONE.

Constrain leaseholders to specific datacenters

In addition to constraining replicas to specific datacenters, you may also specify preferences for where the range's leaseholders should be placed. This can result in increased performance in some scenarios.

The ALTER TABLE ... CONFIGURE ZONE statement below requires that the cluster try to place the ranges' leaseholders in zone us-east-1b; if that is not possible, it will try to place them in zone us-east-1a.

For more information about how the lease_preferences field works, see its description in the Replication zone variables section.

icon/buttons/copy
> ALTER TABLE kv CONFIGURE ZONE USING num_replicas = 3, constraints = '{"+zone=us-east-1a": 1, "+zone=us-east-1b": 1}', lease_preferences = '[[+zone=us-east-1b], [+zone=us-east-1a]]';
CONFIGURE ZONE 1
icon/buttons/copy
> SHOW ZONE CONFIGURATION FOR TABLE kv;
 zone_name |                               config_sql
-----------+------------------------------------------------------------------------
 test.kv   | ALTER TABLE kv CONFIGURE ZONE USING                                   +
           |         range_min_bytes = 1048576,                                    +
           |         range_max_bytes = 67108864,                                   +
           |         gc.ttlseconds = 90000,                                        +
           |         num_replicas = 3,                                             +
           |         constraints = '{+zone=us-east-1a: 1, +zone=us-east-1b: 1}',   +
           |         lease_preferences = '[[+zone=us-east-1b], [+zone=us-east-1a]]'
(1 row)

For more information, see CONFIGURE ZONE.

Scenario-based examples

Even replication across datacenters

Scenario:

  • You have 6 nodes across 3 datacenters, 2 nodes in each datacenter.
  • You want data replicated 3 times, with replicas balanced evenly across all three datacenters.

Approach:

Start each node with its datacenter location specified in the --locality flag:

# Start the two nodes in datacenter 1:
$ cockroach start --insecure --advertise-addr=<node1 hostname> --locality=datacenter=us-1 \
--join=<node1 hostname>,<node3 hostname>,<node5 hostname>
$ cockroach start --insecure --advertise-addr=<node2 hostname> --locality=datacenter=us-1 \
--join=<node1 hostname>,<node3 hostname>,<node5 hostname>

# Start the two nodes in datacenter 2:
$ cockroach start --insecure --advertise-addr=<node3 hostname> --locality=datacenter=us-2 \
--join=<node1 hostname>,<node3 hostname>,<node5 hostname>
$ cockroach start --insecure --advertise-addr=<node4 hostname> --locality=datacenter=us-2 \
--join=<node1 hostname>,<node3 hostname>,<node5 hostname>

# Start the two nodes in datacenter 3:
$ cockroach start --insecure --advertise-addr=<node5 hostname> --locality=datacenter=us-3 \
--join=<node1 hostname>,<node3 hostname>,<node5 hostname>
$ cockroach start --insecure --advertise-addr=<node6 hostname> --locality=datacenter=us-3 \
--join=<node1 hostname>,<node3 hostname>,<node5 hostname>

# Initialize the cluster:
$ cockroach init --insecure --host=<any node hostname>

There's no need to make zone configuration changes; by default, the cluster is configured to replicate data three times, and even without explicit constraints, the cluster will aim to diversify replicas across node localities.

Per-replica constraints to specific datacenters

Scenario:

  • You have 5 nodes across 5 datacenters in 3 regions, 1 node in each datacenter.
  • You want data replicated 3 times, with a quorum of replicas for a database holding West Coast data centered on the West Coast and a database for nation-wide data replicated across the entire country.

Approach:

  1. Start each node with its region and datacenter location specified in the --locality flag:

    # Start the four nodes:
    $ cockroach start --insecure --advertise-addr=<node1 hostname> --locality=region=us-west1,datacenter=us-west1-a \
    --join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>
    $ cockroach start --insecure --advertise-addr=<node2 hostname> --locality=region=us-west1,datacenter=us-west1-b \
    --join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>
    $ cockroach start --insecure --advertise-addr=<node3 hostname> --locality=region=us-central1,datacenter=us-central1-a \
    --join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>
    $ cockroach start --insecure --advertise-addr=<node4 hostname> --locality=region=us-east1,datacenter=us-east1-a \
    --join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>
    $ cockroach start --insecure --advertise-addr=<node5 hostname> --locality=region=us-east1,datacenter=us-east1-b \
    --join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>
    
    # Initialize the cluster:
    $ cockroach init --insecure --host=<any node hostname>
    
  2. On any node, open the built-in SQL client:

    icon/buttons/copy
    $ cockroach sql --insecure
    
  3. Create the database for the West Coast application:

    icon/buttons/copy
    > CREATE DATABASE west_app_db;
    
  4. Configure a replication zone for the database:

    icon/buttons/copy
    > ALTER DATABASE west_app_db \
    CONFIGURE ZONE USING constraints = '{"+region=us-west1": 2, "+region=us-central1": 1}', num_replicas = 3;
    
    CONFIGURE ZONE 1
    
  5. View the replication zone:

    icon/buttons/copy
    > SHOW ZONE CONFIGURATION FOR DATABASE west_app_db;
    
       zone_name  |                   config_sql
    +-------------+--------------------------------------------------------------------+
      west_app_db | ALTER DATABASE west_app_db CONFIGURE ZONE USING
                  |     range_min_bytes = 1048576,
                  |     range_max_bytes = 67108864,
                  |     gc.ttlseconds = 90000,
                  |     num_replicas = 3,
                  |     constraints = '{+region=us-west1: 2, +region=us-central1: 1}',
                  |     lease_preferences = '[]'
    (1 row)
    

    Two of the database's three replicas will be put in region=us-west1 and its remaining replica will be put in region=us-central1. This gives the application the resilience to survive the total failure of any one datacenter while providing low-latency reads and writes on the West Coast because a quorum of replicas are located there.

  6. No configuration is needed for the nation-wide database. The cluster is configured to replicate data 3 times and spread them as widely as possible by default. Because the first key-value pair specified in each node's locality is considered the most significant part of each node's locality, spreading data as widely as possible means putting one replica in each of the three different regions.

Multiple applications writing to different databases

Scenario:

  • You have 2 independent applications connected to the same CockroachDB cluster, each application using a distinct database.
  • You have 6 nodes across 2 datacenters, 3 nodes in each datacenter.
  • You want the data for application 1 to be replicated 5 times, with replicas evenly balanced across both datacenters.
  • You want the data for application 2 to be replicated 3 times, with all replicas in a single datacenter.

Approach:

  1. Start each node with its datacenter location specified in the --locality flag:

    # Start the three nodes in datacenter 1:
    $ cockroach start --insecure --advertise-addr=<node1 hostname> --locality=datacenter=us-1 \
    --join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>,<node6 hostname>
    $ cockroach start --insecure --advertise-addr=<node2 hostname> --locality=datacenter=us-1 \
    --join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>,<node6 hostname>
    $ cockroach start --insecure --advertise-addr=<node3 hostname> --locality=datacenter=us-1 \
    --join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>,<node6 hostname>
    
    # Start the three nodes in datacenter 2:
    $ cockroach start --insecure --advertise-addr=<node4 hostname> --locality=datacenter=us-2 \
    --join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>,<node6 hostname>
    $ cockroach start --insecure --advertise-addr=<node5 hostname> --locality=datacenter=us-2 \
    --join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>,<node6 hostname>
    $ cockroach start --insecure --advertise-addr=<node6 hostname> --locality=datacenter=us-2 \
    --join=<node1 hostname>,<node2 hostname>,<node3 hostname>,<node4 hostname>,<node5 hostname>,<node6 hostname>
    
    # Initialize the cluster:
    $ cockroach init --insecure --host=<any node hostname>
    
  2. On any node, open the built-in SQL client:

    icon/buttons/copy
    $ cockroach sql --insecure
    
  3. Create the database for application 1:

    icon/buttons/copy
    > CREATE DATABASE app1_db;
    
  4. Configure a replication zone for the database used by application 1:

    icon/buttons/copy
    > ALTER DATABASE app1_db CONFIGURE ZONE USING num_replicas = 5;
    
    CONFIGURE ZONE 1
    
  5. View the replication zone:

    icon/buttons/copy
    > SHOW ZONE CONFIGURATION FOR DATABASE app1_db;
    
        zone_name  |                config_sql
    +--------------+---------------------------------------------+
         app1_db   | ALTER DATABASE app1_db CONFIGURE ZONE USING
                   |     range_min_bytes = 1048576,
                   |     range_max_bytes = 67108864,
                   |     gc.ttlseconds = 90000,
                   |     num_replicas = 5,
                   |     constraints = '[]',
                   |     lease_preferences = '[]'
    (1 row)
    

    Nothing else is necessary for application 1's data. Since all nodes specify their datacenter locality, the cluster will aim to balance the data in the database used by application 1 between datacenters 1 and 2.

  6. Still in the SQL client, create a database for application 2:

    icon/buttons/copy
    > CREATE DATABASE app2_db;
    
  7. Configure a replication zone for the database used by application 2:

    icon/buttons/copy
    > ALTER DATABASE app2_db CONFIGURE ZONE USING constraints = '[+datacenter=us-2]';
    
  8. View the replication zone:

    icon/buttons/copy
    > SHOW ZONE CONFIGURATION FOR DATABASE app2_db;
    
        zone_name  |                config_sql
    +--------------+---------------------------------------------+
         app2_db   | ALTER DATABASE app2_db CONFIGURE ZONE USING
                   |     range_min_bytes = 1048576,
                   |     range_max_bytes = 67108864,
                   |     gc.ttlseconds = 90000,
                   |     num_replicas = 3,
                   |     constraints = '[+datacenter=us-2]',
                   |     lease_preferences = '[]'
    (1 row)
    

    The required constraint will force application 2's data to be replicated only within the us-2 datacenter.

Stricter replication for a table and its secondary indexes

Scenario:

  • You have 7 nodes, 5 with SSD drives and 2 with HDD drives.
  • You want data replicated 3 times by default.
  • Speed and availability are important for a specific table and its indexes, which are queried very frequently, however, so you want the data in the table and secondary indexes to be replicated 5 times, preferably on nodes with SSD drives.

Approach:

  1. Start each node with ssd or hdd specified as store attributes:

    # Start the 5 nodes with SSD storage:
    $ cockroach start --insecure --advertise-addr=<node1 hostname> --store=path=node1,attrs=ssd \
    --join=<node1 hostname>,<node2 hostname>,<node3 hostname>
    $ cockroach start --insecure --advertise-addr=<node2 hostname> --store=path=node2,attrs=ssd \
    --join=<node1 hostname>,<node2 hostname>,<node3 hostname>
    $ cockroach start --insecure --advertise-addr=<node3 hostname> --store=path=node3,attrs=ssd \
    --join=<node1 hostname>,<node2 hostname>,<node3 hostname>
    $ cockroach start --insecure --advertise-addr=<node4 hostname> --store=path=node4,attrs=ssd \
    --join=<node1 hostname>,<node2 hostname>,<node3 hostname>
    $ cockroach start --insecure --advertise-addr=<node5 hostname> --store=path=node5,attrs=ssd \
    --join=<node1 hostname>,<node2 hostname>,<node3 hostname>
    
    # Start the 2 nodes with HDD storage:
    $ cockroach start --insecure --advertise-addr=<node6 hostname> --store=path=node6,attrs=hdd \
    --join=<node1 hostname>,<node2 hostname>,<node3 hostname>
    $ cockroach start --insecure --advertise-addr=<node7 hostname> --store=path=node7,attrs=hdd \
    --join=<node1 hostname>,<node2 hostname>,<node3 hostname>
    
    # Initialize the cluster:
    $ cockroach init --insecure --host=<any node hostname>
    
  2. On any node, open the built-in SQL client:

    icon/buttons/copy
    $ cockroach sql --insecure
    
  3. Create a database and table:

    icon/buttons/copy
    > CREATE DATABASE db;
    
    icon/buttons/copy
    > CREATE TABLE db.important_table;
    
  4. Configure a replication zone for the table that must be replicated more strictly:

    icon/buttons/copy
    > ALTER TABLE db.important_table CONFIGURE ZONE USING num_replicas = 5, constraints = '[+ssd]'
    
  5. View the replication zone:

    icon/buttons/copy
    > SHOW ZONE CONFIGURATION FOR TABLE db.important_table;
    
              zone_name       |                config_sql
    +-------------------------+---------------------------------------------+
         db.important_table   | ALTER DATABASE app2_db CONFIGURE ZONE USING
                              |     range_min_bytes = 1048576,
                              |     range_max_bytes = 67108864,
                              |     gc.ttlseconds = 90000,
                              |     num_replicas = 5,
                              |     constraints = '[+ssd]',
                              |     lease_preferences = '[]'
    (1 row)
    

    The secondary indexes on the table will use the table's replication zone, so all data for the table will be replicated 5 times, and the required constraint will place the data on nodes with ssd drives.

Tweaking the replication of system ranges

Scenario:

  • You have nodes spread across 7 datacenters.
  • You want data replicated 5 times by default.
  • For better performance, you want a copy of the meta ranges in all of the datacenters.
  • To save disk space, you only want the internal timeseries data replicated 3 times by default.

Approach:

  1. Start each node with a different locality attribute:

    # Start the nodes:
    $ cockroach start --insecure --advertise-addr=<node1 hostname> --locality=datacenter=us-1 \
    --join=<node1 hostname>,<node2 hostname>,<node3 hostname>   
    $ cockroach start --insecure --advertise-addr=<node2 hostname> --locality=datacenter=us-2 \
    --join=<node1 hostname>,<node2 hostname>,<node3 hostname>
    $ cockroach start --insecure --advertise-addr=<node3 hostname> --locality=datacenter=us-3 \
    --join=<node1 hostname>,<node2 hostname>,<node3 hostname>
    $ cockroach start --insecure --advertise-addr=<node4 hostname> --locality=datacenter=us-4 \
    --join=<node1 hostname>,<node2 hostname>,<node3 hostname>
    $ cockroach start --insecure --advertise-addr=<node5 hostname> --locality=datacenter=us-5 \
    --join=<node1 hostname>,<node2 hostname>,<node3 hostname>
    $ cockroach start --insecure --advertise-addr=<node6 hostname> --locality=datacenter=us-6 \
    --join=<node1 hostname>,<node2 hostname>,<node3 hostname>
    $ cockroach start --insecure --advertise-addr=<node7 hostname> --locality=datacenter=us-7 \
    --join=<node1 hostname>,<node2 hostname>,<node3 hostname>
    
    # Initialize the cluster:
    $ cockroach init --insecure --host=<any node hostname>
    
  2. On any node, open the built-in SQL client:

    icon/buttons/copy
    $ cockroach sql --insecure
    
  3. Configure the default replication zone:

    icon/buttons/copy
    > ALTER RANGE default CONFIGURE ZONE USING num_replicas = 5;
    
  4. View the replication zone:

    icon/buttons/copy
    > SHOW ZONE CONFIGURATION FOR RANGE default;
    
      zone_name |                config_sql
    +-----------+------------------------------------------+
      .default  | ALTER RANGE default CONFIGURE ZONE USING
                |     range_min_bytes = 1048576,
                |     range_max_bytes = 67108864,
                |     gc.ttlseconds = 90000,
                |     num_replicas = 5,
                |     constraints = '[]',
                |     lease_preferences = '[]'
    (1 row)
    

    All data in the cluster will be replicated 5 times, including both SQL data and the internal system data.

  5. Configure the .meta replication zone:

    icon/buttons/copy
    > ALTER RANGE meta CONFIGURE ZONE USING num_replicas = 7;
    
      zone_name |              config_sql
    +-----------+---------------------------------------+
      .meta     | ALTER RANGE meta CONFIGURE ZONE USING
                |     range_min_bytes = 1048576,
                |     range_max_bytes = 67108864,
                |     gc.ttlseconds = 3600,
                |     num_replicas = 7,
                |     constraints = '[]',
                |     lease_preferences = '[]'
    (1 row)
    

    The .meta addressing ranges will be replicated such that one copy is in all 7 datacenters, while all other data will be replicated 5 times.

  6. Configure the .timeseries replication zone:

    icon/buttons/copy
    > ALTER RANGE timeseries CONFIGURE ZONE USING num_replicas = 3;
    
       zone_name  |                 config_sql
    +-------------+---------------------------------------------+
      .timeseries | ALTER RANGE timeseries CONFIGURE ZONE USING
                  |     range_min_bytes = 1048576,
                  |     range_max_bytes = 67108864,
                  |     gc.ttlseconds = 90000,
                  |     num_replicas = 3,
                  |     constraints = '[]',
                  |     lease_preferences = '[]'
    (1 row)
    

    The timeseries data will only be replicated 3 times without affecting the configuration of all other data.

See also


Yes No
On this page

Yes No