Skip to content

Do not restore cross-target variables into shared config in config-remote-sync#5593

Open
ilyakuz-db wants to merge 1 commit into
mainfrom
configsync-cross-target-vars
Open

Do not restore cross-target variables into shared config in config-remote-sync#5593
ilyakuz-db wants to merge 1 commit into
mainfrom
configsync-cross-target-vars

Conversation

@ilyakuz-db

Copy link
Copy Markdown
Contributor

Changes

bundle config-remote-sync restores hardcoded remote values back to ${var.X} references. Two restoration paths could introduce a reference into shared configuration that names a variable only resolvable for the currently selected target:

  • the Replace fallback (matchAnyVariable), which re-targets a ${var.X} field to any uniquely matching variable, and
  • sibling-based Add restoration (restoreFromSiblings), which copies references from sibling sequence elements.

Both consult the post-target-selection config, where targets.<t>.variables assignments and invocation-scoped values (BUNDLE_VAR_*, variable-overrides.json) are already merged in. Writing such a reference into the shared resources section breaks every other target of the bundle.

This PR adds a cross-target guard, built from the configuration just before SelectTarget runs (the only point where the full targets.*.variables map is still available). A ${var.X} reference may only be introduced at a position that didn't previously hold one when:

  • the bundle has at most one target (no other target can break), or
  • X has a default or lookup in the root variables block, or
  • X is assigned a default or lookup in every target's variables block, or
  • the write destination is the current target's override section, which other targets never read (classified by checking which path candidate exists in the pre-target-selection config, mirroring the order applyChange tries them in).

Restoration steps that re-emit a reference already present at the same position (matchOriginalRef, restoreCompoundInterpolation on Replace) are intentionally not gated, and guard-rejected variables still count toward the existing ambiguity rule rather than promoting another candidate.

Why

Without the guard, syncing a remote edit on one target could rewrite a shared field such as default: ${var.region} to default: ${var.dev_catalog} (or copy ${var.dev_catalog} into a newly added element) when the edited value happened to match that variable's dev value. Since dev_catalog has no root default and is only assigned for dev, every other target then fails to load. The guard makes this structurally impossible while keeping restoration intact for variables that resolve everywhere and for edits that land in the current target's own override section.

Tests

  • New unit tests for the guard (safe-set construction, destination classification including the conservative indexed-element fallback, varRefsAllowed, and both gated restoration paths).
  • New acceptance test acceptance/bundle/config-remote-sync/cross_target_variables covering: Add restored to a root-default variable, Add restored to a variable assigned in every target, Add of a dev-only match staying hardcoded, Replace fallback to a dev-only variable staying hardcoded, and a Replace inside targets.dev still restoring the dev-only variable. Before the fix, the first run of this test showed ${var.dev_catalog} written into the shared resources section in both leak paths.
  • Full config-remote-sync acceptance suite passes locally; the new test plus resolve_variables and target_override also pass against a real workspace on both engines (direct and terraform).

…mote-sync

Variable restoration in config-remote-sync runs with a single target
selected, so the resolved variables include values that may only exist for
that target (targets.<t>.variables assignments, BUNDLE_VAR_* environment
variables, variable-overrides.json). The fallback lookup (matchAnyVariable)
and sibling-based Add restoration (restoreFromSiblings) substituted such
variables with no check that they resolve in other targets, writing e.g.
${var.dev_catalog} into the shared resources section and breaking every
other target of the bundle.

Add a cross-target guard, built from the configuration just before target
selection (the only point where the full targets.*.variables map is still
available): a ${var.X} reference may only be introduced at a position that
didn't previously hold one when X has a root default or lookup, is assigned
in every target, or the write destination is the current target's override
section (which other targets never read). Single-target bundles are exempt
since no other target can break. Restoration steps that re-emit a reference
already present at the same position (matchOriginalRef,
restoreCompoundInterpolation) are intentionally not gated.
@github-actions

Copy link
Copy Markdown
Contributor

Approval status: pending

/acceptance/bundle/ - needs approval

5 files changed
Suggested: @denik
Also eligible: @andrewnester, @anton-107, @janniklasrose, @pietern, @shreyas-goenka, @lennartkats-db

/bundle/ - needs approval

Files: bundle/configsync/variables.go, bundle/configsync/variables_test.go
Suggested: @denik
Also eligible: @andrewnester, @anton-107, @janniklasrose, @pietern, @shreyas-goenka, @lennartkats-db

Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db) can approve all areas.
See OWNERS for ownership rules.

@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 28c0432

Run: 27452054829

Env 🟨​KNOWN 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
🟨​ aws linux 7 15 264 978 8:14
🟨​ aws windows 7 15 266 976 16:23
💚​ aws-ucws linux 7 15 360 892 6:57
💚​ aws-ucws windows 7 15 362 890 11:50
💚​ azure linux 1 17 267 976 6:56
💚​ azure windows 1 17 269 974 11:16
💚​ azure-ucws linux 1 17 365 888 8:19
💚​ azure-ucws windows 1 17 367 886 12:27
💚​ gcp linux 1 17 263 979 8:32
💚​ gcp windows 1 17 265 977 9:49
22 interesting tests: 15 SKIP, 7 KNOWN
Test Name aws linux aws windows aws-ucws linux aws-ucws windows azure linux azure windows azure-ucws linux azure-ucws windows gcp linux gcp windows
🟨​ TestAccept 🟨​K 🟨​K 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🙈​ TestAccept/bundle/invariant/no_drift 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/permissions 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions 🟨​K 🟨​K 💚​R 💚​R 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions 🟨​K 🟨​K 💚​R 💚​R 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 🟨​K 🟨​K 💚​R 💚​R
🙈​ TestAccept/bundle/resources/postgres_branches/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/recreate 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/replace_existing 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/update_protected 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/without_branch_id 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_endpoints/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_endpoints/recreate 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_projects/update_display_name 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/synced_database_tables/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_indexes/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_indexes/grants/select 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/ssh/connection 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
Top 29 slowest tests (at least 2 minutes):
duration env testname
6:19 azure-ucws windows TestAccept
6:09 aws-ucws windows TestAccept
6:07 azure windows TestAccept
4:55 gcp windows TestAccept
4:41 gcp linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
4:30 gcp windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
4:22 gcp linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
4:06 gcp windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:36 aws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:17 aws-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:12 aws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:08 aws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:08 azure-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:05 aws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:57 azure windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:50 gcp linux TestAccept
2:50 aws-ucws linux TestAccept
2:48 azure linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:48 azure linux TestAccept
2:45 azure-ucws linux TestAccept
2:43 azure windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:42 aws-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:37 aws-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:34 azure-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:32 azure linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:31 azure-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:27 azure-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:26 aws-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:11 gcp linux TestSecretsPutSecretStringValue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants