Skip to content

fix: retry ClusterSummary status update in updateStatusForNonReferencedHelmReleases - #1937

Merged
gianlucam76 merged 1 commit into
projectsveltos:mainfrom
gianlucam76:bug-1933
Sep 1, 2026
Merged

fix: retry ClusterSummary status update in updateStatusForNonReferencedHelmReleases#1937
gianlucam76 merged 1 commit into
projectsveltos:mainfrom
gianlucam76:bug-1933

Conversation

@gianlucam76

@gianlucam76 gianlucam76 commented Sep 1, 2026

Copy link
Copy Markdown
Member

In SyncModeContinuousWithDriftDetection, updateStatusForNonReferencedHelmReleases did a single Get plus a single Status().Update with no retry, unlike its two siblings on the same pass (updateStatusForReferencedHelmReleases, updateValueHashOnHelmChartSummary), which both already wrap their Get+Update in retry.RetryOnConflict.

A concurrent status write (e.g. updateValueHashOnHelmChartSummary a few ms earlier) can bump the ClusterSummary's resourceVersion between this function's cached Get and its own Status().Update, so the write loses with:

Operation cannot be fulfilled on clustersummaries.config.projectsveltos.io "...":
the object has been modified; please apply your changes to the latest version and try again

handleCharts returns that error immediately, even though the Helm deploy already succeeded. Consequences while this keeps happening:

  • the feature hash never advances, so every reconcile re-runs a full no-op deploy pass
  • the Helm feature reports Failed with consecutiveFailures climbing, while the release itself is healthy and deployed
  • drift detection registration (postProcessDeployedHelmCharts) is never reached, so ResourceSummary.spec.chartResources stays empty and out-of-band drift goes undetected

This mostly hits profiles under frequent reconcile pressure (e.g. an HPA-autoscaled Deployment keeps requesting reconciliation), since each failed pass's own status write (consecutiveFailures) supplies the next conflicting write, making the loop self-sustaining once triggered.

Fix: wrap the Get and Status().Update in updateStatusForNonReferencedHelmReleases in retry.RetryOnConflict, matching its two siblings exactly.

Fixes #1933

…edHelmReleases

In SyncModeContinuousWithDriftDetection, updateStatusForNonReferencedHelmReleases did a
single Get plus a single Status().Update with no retry, unlike its two siblings on the
same pass (updateStatusForReferencedHelmReleases, updateValueHashOnHelmChartSummary),
which both already wrap their Get+Update in retry.RetryOnConflict.

A concurrent status write (e.g. updateValueHashOnHelmChartSummary a few ms earlier) can
bump the ClusterSummary's resourceVersion between this function's cached Get and its own
Status().Update, so the write loses with:

    Operation cannot be fulfilled on clustersummaries.config.projectsveltos.io "...":
    the object has been modified; please apply your changes to the latest version and try again

handleCharts returns that error immediately, even though the Helm deploy already
succeeded. Consequences while this keeps happening:

- the feature hash never advances, so every reconcile re-runs a full no-op deploy pass
- the Helm feature reports Failed with consecutiveFailures climbing, while the release
  itself is healthy and deployed
- drift detection registration (postProcessDeployedHelmCharts) is never reached, so
  ResourceSummary.spec.chartResources stays empty and out-of-band drift goes undetected

This mostly hits profiles under frequent reconcile pressure (e.g. an HPA-autoscaled
Deployment keeps requesting reconciliation), since each failed pass's own status write
(consecutiveFailures) supplies the next conflicting write, making the loop
self-sustaining once triggered.

Fix: wrap the Get and Status().Update in updateStatusForNonReferencedHelmReleases in
retry.RetryOnConflict, matching its two siblings exactly.
@gianlucam76
gianlucam76 merged commit 537f7e8 into projectsveltos:main Sep 1, 2026
20 checks passed
@gianlucam76
gianlucam76 deleted the bug-1933 branch September 1, 2026 07:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG: missing RetryOnConflict in updateStatusForNonReferencedHelmReleases keeps the Helm feature Failed and disables drift detection

1 participant