Skip to content

[local deployment/crdb] Ensure that each USS has at least a replica of the tables#1551

Open
mickmis wants to merge 1 commit into
mainfrom
crdb-zone
Open

[local deployment/crdb] Ensure that each USS has at least a replica of the tables#1551
mickmis wants to merge 1 commit into
mainfrom
crdb-zone

Conversation

@mickmis

@mickmis mickmis commented Jul 2, 2026

Copy link
Copy Markdown
Contributor
  • sets the locality of each CRDB nodes to its USS, which defines zones for CRDB
  • configure the default behavior (applying to all tables) to have as many replicas as there are USSs, and because CRDB will distribute replicas over zones, each USS will get a replica

We do this to produce more consistent results when testing performances of the DSS.
This should address interuss/dss#1559

@mickmis mickmis requested a review from the-glu July 2, 2026 13:00
out=$$(/cockroach/cockroach.sh init --insecure --host=db1.uss1.localutm 2>&1);
if [ $$? -eq 0 ]; then
echo "$$out";
/cockroach/cockroach.sh sql --insecure --host=db1.uss1.localutm -e 'ALTER RANGE default CONFIGURE ZONE USING num_replicas = ${NUM_USS:?};';

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do this work reliably, or there is a chance that migrations create database/tables before and use a different default?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • migrations (from the (rid|scd|aux)_bootstrapper) explicitely wait for crdb-init to be completed before being ran
  • even if this is applied after creation of the table, the setting is applied, as we apply it on the zone

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perfect 👍

Comment thread build/dev/docker-compose.yaml Outdated
entrypoint: /db-entrypoint.sh
# 128MB limits allow larger numbers of nodes on a single machine
command: /cockroach/cockroach.sh start --insecure --join=db1.uss1.localutm --cache=128MiB --max-sql-memory=128MiB
command: /cockroach/cockroach.sh start --insecure --join=db1.uss1.localutm --cache=128MiB --max-sql-memory=128MiB --locality=uss=${USS_IDX:?}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably an open question to be validated, but going one level higher and having one locality per USS_IDX-USS_NODE_IDX may be better at the cost of being less realist.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could indeed have that. IIUC locality keys are hierarchical, i.e. if we have the localities uss=1,node=1-2 defined, CRDB will distribute first over the localities uss, and then over the localities node. So we could already set this in here without impact.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check out the new version.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 LGTM

@the-glu

the-glu commented Jul 2, 2026

Copy link
Copy Markdown
Member

I will do some test to validate that it's remove variations

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants