Skip to content

app: release the target lock when deleting a target that is not running - #1035

Open
methakon wants to merge 1 commit into
openconfig:mainfrom
methakon:fix/1011-delete-target-releases-lock
Open

methakon wants to merge 1 commit into
openconfig:mainfrom
methakon:fix/1011-delete-target-releases-lock

Conversation

@methakon

Copy link
Copy Markdown

Problem

DELETE /api/v1/config/targets/{name} returns 200 and removes the target from the config, but the
cluster lock is only released from inside the branch that removes the target from the runtime
map:

if t, ok := a.Targets[name]; ok {
    delete(a.Targets, name)
    t.Close()
    if a.locker != nil {
        return a.locker.Unlock(ctx, a.targetLockKey(name))
    }
}
return nil

A target that is in the config but not in the runtime map therefore returns success having released
nothing.

That is exactly the state a collector is in after a restart: it holds locks for targets it has not
recreated yet, and its fresh runtime map is empty. Every lock carried across a restart became a
ghost, and no subsequent DELETE could clear it, because that delete also took the non-running branch.
This is why the report describes the bug as invisible within a single instance lifetime and only
surfacing after a restart.

There is a second, independent leak: the assignment entry in targetsLockFn has its cancel func
invoked but is never removed from the map, so it continues to be reported as a live assignment for a
deleted target.

Change

  • The lock is released after the runtime cleanup rather than inside it, so both paths release it. The
    lock is held against the config, not against the runtime target, which is what makes that correct
  • The unlock error is still returned
  • delete(a.targetsLockFn, name) alongside the existing cfn() call

Closing and removing a running target is unchanged.

Test

target_delete_lock_test.go uses a locker that records the keys passed to Unlock:

Test Covers
TestDeleteTargetUnlocksWhenTargetNotRunning the reported case: config has the target, runtime map does not
TestDeleteTargetRemovesLockAssignment the assignment entry is dropped
TestDeleteTargetClosesRunningTarget a running target is still removed and unlocked
TestDeleteTargetUnknownStillErrors the unlock path is unreachable for a name that does not exist

The first two fail against the unfixed handler, which is the evidence that they pin the reported
behaviour rather than the implementation:

--- FAIL: TestDeleteTargetUnlocksWhenTargetNotRunning
    target_delete_lock_test.go:61: unlock calls = [], want exactly [ghost]
--- FAIL: TestDeleteTargetRemovesLockAssignment
    target_delete_lock_test.go:81: assignment entry still present after DeleteTarget

Verification

Command Result
go test ./pkg/app/... -count=1 ok, 1.2s
go test ./pkg/app/ -run TestDeleteTarget -count=1 4 pass
go vet ./pkg/app/ clean
gofmt -l on both files clean

The tests drive App.DeleteTarget directly rather than going through the REST handler, so they do
not need a running collector or a consul instance. I have not exercised the HTTP path end to end;
the handler is a thin wrapper that calls this method.

Fixes #1011

@google-cla

google-cla Bot commented Sep 29, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

DeleteTarget only released the cluster lock from inside the branch that
removes the target from the runtime map, so a target that exists in the
config but not in the runtime map returned without unlocking and without
error. The API reported success while the lock stayed held.

That is the state a collector is in after a restart: it holds locks for
targets it has not recreated yet, and its fresh runtime map is empty. Every
such lock became a ghost that a later DELETE against the same config could
not clear, because that delete also took the non-running branch.

The lock is held against the config rather than the runtime target, so it is
now released after the runtime cleanup, on both paths. The close and delete
of a running target are unchanged, and an unlock error is still returned.

Also drop the target's entry from targetsLockFn. The cancel func was invoked
but the map entry stayed, so the target continued to be reported as an
assignment after it was deleted.
@methakon
methakon force-pushed the fix/1011-delete-target-releases-lock branch from 8351c82 to 5b75259 Compare September 29, 2026 19:18

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DELETE /api/v1/config/targets/{name} leaks cluster lock and assignment (collector + consul locker)

1 participant