feat(ingester): Add owned series tracking to prevent false throttling… by anant10 · Pull Request #7758 · cortexproject/cortex

anant10 · 2026-08-11T17:08:56Z

What this PR does:

Adds per-ingester series ownership tracking to prevent false throttling during ingester scale-up and ring resharding.

The problem: When ingesters scale up, the per-ingester local series limit drops immediately (recalculated based on new ingester count), but stale series data remains in the TSDB head for up to 2 hours until head compaction. PreCreation() uses Head().NumSeries() for limit checks, so it incorrectly rejects new writes during this window — the ingester appears over its new lower limit, but many of those series have been resharded to other ingesters and will be cleaned up at next compaction.

The solution: Track which series each ingester actually owns according to the ring, and use that count for limit enforcement. The owned count drops immediately when the ring changes (within 1 minute), eliminating the 2-hour dependency on head compaction.

How it works:

On each push, compute the series' ring token (same hash the distributor uses for routing) and store it in ActiveSeries
Every ~1 min (updateActiveSeries cycle), if the ring changed, re-scan all entries and remove series whose token no longer maps to this ingester
PreCreation() uses activeSeries.Owned() instead of Head().NumSeries() for the limit check

Design decisions:

Two feature flags for safe progressive rollout:
- -ingester.owned-series-metrics-enabled: enables cortex_ingester_owned_series metric emission only (no enforcement change)
- -ingester.owned-series-limit-enforcement-enabled: switches limit enforcement to use owned count (requires first flag)
Zone-local ownership check (3 lines): SearchToken(zoneTokens, key) → is responsible token in this instance's set?
Lock-free reads on push path: ring state stored behind atomic.Pointer[ringState] — zero lock contention
Instance-level max_series: instanceOwnedCount recalculated every ~1 min (not incremental, avoids drift). Startup fallback to instanceSeriesCount when count is 0
Code consolidation: FNV hash → pkg/util/fnv.go, sharding functions → pkg/ring/token.go (eliminates duplication between distributor and ring)

Validation:

Tested in a multi-zone deployment under sustained write load. After scale-up:

memory_series on old ingesters remained high (stale data in head)
owned_series on old ingesters dropped proportionally to ring redistribution
Zero throttle errors during the scale-up window that would have previously caused false rejections

New configuration flags (experimental):

yaml

ingester:
 # Emit cortex_ingester_owned_series metric (no enforcement change)
 # CLI flag: -ingester.owned-series-metrics-enabled
 [owned_series_metrics_enabled: <bool> | default = false]

 # Use owned count for limit enforcement (requires above flag)
 # CLI flag: -ingester.owned-series-limit-enforcement-enabled
 [owned_series_limit_enforcement_enabled: <bool> | default = false]

Which issue(s) this PR fixes: Fixes #7509

… during ring changes When ingesters scale up, the per-ingester local series limit drops immediately but stale series data remains in TSDB head for up to 2 hours. This causes PreCreation() to incorrectly reject new writes. This PR introduces owned series tracking in ActiveSeries: - Each series stores its ring token (computed via TokenForLabels) - Ownership is evaluated against current ring state on each update cycle - When ring changes, unowned series are excluded from limit enforcement Two feature flags for safe rollout: - owned_series_metrics_enabled: enables cortex_ingester_owned_series metric - owned_series_limit_enforcement_enabled: switches PreCreation to use owned count for both per-user and instance-level max_series limits Key design decisions: - Zone-local ownership check via SearchToken + instance token map lookup - Ring state stored behind atomic.Pointer[ringState] for lock-free reads on the hot push path - instanceOwnedCount recalculated every ~1 min (not incremental) to avoid drift from edge cases - Startup fallback: when instanceOwnedCount==0, uses instanceSeriesCount Code consolidation: - FNV hash functions consolidated into pkg/util/fnv.go (single source) - Sharding functions moved to pkg/ring/token.go (eliminates duplication between distributor and ring packages) Production validation: tested with 5M active series, scale-up 9->18 ingesters showed owned_series=984K vs memory_series=1.8M (813K stale series correctly excluded). Zero throttle errors. Fixes cortexproject#7509 Signed-off-by: Anant Shanbhag <anantvas@amazon.com>

danielblando · 2026-08-13T23:48:53Z


+	// If ring tokens are loaded, check ownership before creating.
+	// This prevents tracking series we don't own (e.g., stale distributor routes).
+	if len(ringTokens) > 0 && !isOwnedByInstance(key, ringTokens, instanceTokens) {


we should maintain active behaviour the same. we should make the check impact on the owned metric which is the new one

Addressed this issue in the next revision

danielblando · 2026-08-13T23:56:51Z

+	// This test just validates ActiveSeries itself works correctly.
 }

-func BenchmarkActiveSeries_UpdateSeries(b *testing.B) {


Did we replace tests removing benchmark to add new test?
We should still have benchmark and would be nice to run to see how it perform with owned active

Yes fixing it in the next revision. Removed it by mistake

danielblando · 2026-08-13T23:59:12Z

+// if the responsible token belongs to this instance.
+func isOwnedByInstance(key uint32, ringTokens []uint32, instanceTokens map[uint32]struct{}) bool {
+	i := ring.SearchToken(ringTokens, key)
+	_, found := instanceTokens[ringTokens[i]]


lets add a check here to make sure ringTokens is not empty otherwise it will crash.

Added. SearchToken wraps to 0 on an empty slice, so the index panics. it wasn't reachable since all three call sites guarded with len(ringTokens) > 0, but the check belonged in the function.

It now returns true when the token list is empty or the ownership data doesn't match it, so unknown ownership counts as owned rather than under-counting against the limit.

danielblando · 2026-08-14T00:18:43Z

 * [CHANGE] Cache: Setting `-blocks-storage.bucket-store.metadata-cache.bucket-index-content-ttl` to 0 will disable the bucket-index cache. #7446
 * [CHANGE] HA Tracker: Move `-distributor.ha-tracker.failover-timeout` from a global config to a per-tenant runtime config. The flag name and default value (30s) remain the same. #7481
+* [FEATURE] Ingester: Add owned series tracking to prevent false customer throttling during ingester scale-up and ring resharding. When enabled, the ingester tracks which series it currently owns according to the ring and uses that count (instead of total in-memory series) for limit enforcement. Eliminates a up-to-2-hour window of incorrect throttling after any ring change. Controlled by `-ingester.owned-series-metrics-enabled` (metric emission) and `-ingester.owned-series-limit-enforcement-enabled` (limit enforcement). #7509
+* [ENHANCEMENT] Ring: Consolidate sharding functions (`TokenForLabels`, `ShardByMetricName`, etc.) into `pkg/ring/token.go` for reuse by both distributor and ingester. Export `SearchToken`. #7509


We dont need this entries. Just one for feature is enough

removing it

Ring.Get determined the replica set for a key by inlining the ring walk directly in its body. The walk is the single definition of "which instances hold this data", and the owned-series work needs to ask that same question from a second place: for every token position, rather than for one key. Duplicating the walk there would let the two copies disagree, and a disagreement is exactly the bug that matters, because an ingester would then count series against its limit that the distributor does not route to it. Move the walk into ringTopology.replicaSetAt, keyed on a token position rather than a key, and have Ring.Get call it after resolving the key to a position with SearchToken. ringTopology is a read-only view that copies only slice and map headers, so it can be built per call. replicaSetAt also returns the instance IDs alongside the descriptors. InstanceDesc carries no ID of its own and an instance's ID is not interchangeable with its Addr, so a caller asking "am I in this set?" can only answer that correctly by comparing IDs. The per-zone cap is now skipped when the ring reports no zones at all, rather than dividing by the zone count unconditionally. Ring.Get cannot reach that state for a non-empty ring, so this is a robustness change for the new caller, not a behaviour change here. No behaviour change. The existing pkg/ring suite passes unmodified, and BenchmarkRing_Get is unchanged at ~607 ns/op and 0 allocs/op. Signed-off-by: Anant Shanbhag <anantvas@amazon.com>

Add OwnedTokenPositions, which returns the ring's sorted token list plus a parallel bitmap of the positions owned by a given instance. Ownership is computed with replicaSetAt, the same walk Ring.Get uses, so an instance's view of what it owns cannot drift from where the ring routes. This replaces the "am I the first instance in this token range, among the token list for my own zone?" check the owned-series tracking previously used. That check has exactly one owner per token range, which is only the same thing as the replica set in one configuration: zone awareness enabled with a replication factor equal to the zone count. In every other shape, including the default of zone awareness disabled, it undercounts by roughly the replication factor. Ownership deliberately does not apply the replication strategy's health filter. It has to stay stable across heartbeat flapping, because a series does not stop being this instance's responsibility when a peer misses a heartbeat. The lookup is O(tokens x replicationFactor) and is intended to be called once per ring change, so that the per-series question becomes owned[SearchToken(tokens, key)]: a binary search plus an array index. BenchmarkOwnedTokenPositions covers that precompute, at ~29.7 ms for 100 instances holding 512 tokens each. Tests assert the load-bearing property directly: across ten ring shapes covering zone awareness on and off and replication factors above, below, equal to and not divisible by the zone count, an instance owns a token position if and only if Ring.Get for a key in that position includes it. A conservation test asserts that every position is owned by exactly replicationFactor instances, which is what makes the per-ingester counts sum correctly against a limit that is itself scaled by the replication factor. One test builds a ring whose instance IDs and addresses deliberately differ. The shared fixtures set Addr equal to the instance ID, so an implementation comparing addresses would pass every other test here and then own nothing at all in production. Signed-off-by: Anant Shanbhag <anantvas@amazon.com>

Replace the ownership check used by owned-series tracking. It previously asked "is the first token at or after my series' token, within the token list for my own zone, one of my tokens?". That has exactly one owner per token range, which only coincides with the ring's replica set when zone awareness is enabled and the replication factor equals the zone count. In every other configuration, including the default of zone awareness disabled, it is wrong. With zone awareness off, every instance falls into one unnamed zone, the "zone-filtered" token list becomes the whole ring, and the check degenerates to "am I the single global primary?". That counts roughly 1/replicationFactor of the series the instance actually holds, while the limit it is compared against is scaled up by the replication factor. The net effect is a limit that is too permissive by about the replication factor, so an ingester runs out of memory instead of throttling. That is a worse failure than the stale-count problem this feature exists to fix. Ownership now comes from ring.OwnedTokenPositions, which walks the ring with the same function Ring.Get uses, so the set of series an ingester counts is by construction the set the distributor routes to it. Because the walk honours the operation's replica-set extension rules, instances in JOINING, LEAVING and READONLY are handled exactly as the write path handles them, rather than via a separate READONLY special case. The lifecycler computes the bitmap in updateCounters, alongside the healthy-instance count that the limiter already uses as the denominator of the same comparison. Both therefore come from one ring snapshot taken at one instant under one lock, so numerator and denominator cannot be drawn from different views of the ring. The walk is too expensive to repeat on every heartbeat, and heartbeats are by far the most common reason updateCounters runs, so it is gated on a fingerprint covering instances, zones, states and tokens. Desc.RingCompare cannot serve here: it reports a state change and a timestamp change as the same result, and ownership depends on state. The recompute deliberately runs outside countersLock, since HealthyInstancesCount is on the push path and must not block behind it. Per-series cost drops from a map lookup to a binary search plus an array index, and isOwned no longer indexes the token slice without checking its length, which panicked on an empty ring. Tested with the upstream CI invocation, which matters here: go test -tags "netgo slicelabels" -race. Without the slicelabels tag pkg/ingester panics in cortexpb.CopyLabels, and without -race the //go:build !race file is compiled in and panics too; both reproduce on unmodified upstream. Signed-off-by: Anant Shanbhag <anantvas@amazon.com>

Owned-series tracking previously refused to track a series the ring did not assign to this ingester, and deleted entries when a ring change took ownership away. That made cortex_ingester_active_series silently drop those series, changing the meaning of a long-standing metric, and it made owned and active identical by construction, which left the staged rollout across the two feature flags unable to show any difference. Ownership now gates exactly one counter. Every series is tracked and counts towards active exactly as before; only owned is conditional. Losing ownership moves a series out of owned and leaves it active, which is accurate, because the series is still held in this ingester's head. Retention is now two-tier. The periodic cycle recounts but removes nothing, and entries are released by a purge after head compaction, keyed on the head's new minimum time. An entry therefore lives for as long as the series it describes is in memory. This is what makes the feature work for a tenant with high churn. Tying owned to the idle timeout meant a tenant with a 3M-series head but only 500K recently active series reported owned as 500K, so a 3M limit would admit 2.5M more series on top of the 3M already resident: more permissive than the unmodified code it replaces. Anchoring to the head instead makes owned a measure of what the ingester is actually storing, which is what a series limit exists to bound. The consequence is that owned may exceed active for such a tenant, which is intended and is the reason the two counts are maintained separately rather than one being derived from the other. Totals are cached per tenant and maintained as series are created, because PreCreation consults them for every new series and previously took a read lock on all 512 stripes to do so. The periodic cycle recomputes them authoritatively, so any drift is corrected within one cycle. Limit enforcement follows. The per-tenant check no longer needs its "owned > 0" guard: the cached total is accurate from the first sample rather than only after the first periodic cycle, so there is no startup window to paper over, and the guard had a cliff where a genuinely empty tenant silently fell back to the head count. The instance-level check keeps a fallback, because the instance total really is unavailable until the first cycle runs, but it is now keyed on an explicit negative sentinel rather than on zero, so a true zero is no longer confused with "not yet computed". Measured on 100k series: the periodic recount is 1.67ms with no allocations when the ring is unchanged and 2.42ms when every entry's ownership is re-evaluated. The push path is unchanged within noise, ~440ns/op with the ring loaded against ~400-457ns/op without it. Signed-off-by: Anant Shanbhag <anantvas@amazon.com>

The owned-series work replaced this file wholesale, removing 139 lines: three tests covering the active count and purge behaviour, and four benchmarks covering the push and purge paths. Removing the benchmarks meant there was no longer a baseline to compare the new per-series ownership cost against, which is what the review asked about. All of them are restored, with the ring token argument added. They pass a token of zero with no ring loaded, so ownership is unknown and every series counts as owned, which means they exercise precisely the behaviour that predates this feature. Restoring them surfaced that BenchmarkActiveSeries_UpdateSeries does not run at all. It calls b.Loop() in two separate loops, which is not allowed, and sizes its series slice from b.N before b.Loop() has established an iteration count. It panics with "index out of range [1] with length 1", or fails with "B.Loop called with timer stopped" when an explicit -benchtime is given. This reproduces on unmodified upstream at this branch's merge base, and the same code is present verbatim on master, so it has been broken since the b.Loop() migration rather than by this change. It is converted to the b.N form, which is the correct shape when the iteration count is needed before the timed loop begins. New coverage for the owned-series semantics: - active is unaffected by ownership. The same series are pushed into two trackers, one with a ring where half the tokens belong elsewhere and one with no ring, and the active counts must agree exactly. This is the regression test for the metric's meaning being preserved. - owned exceeds active for a tenant whose series have gone idle but are still held in the head, which is the behaviour the limit fix depends on. - the periodic cycle retains expired entries, repeated cycles are stable rather than progressively dropping entries, and only a purge releases them. - losing and regaining ownership moves series between owned and not-owned without touching active and without deleting anything, so ownership can return without the series being re-pushed. - the cached totals match the per-stripe counters after creation, after a periodic cycle, after a purge and after a clear, since those totals are what limits are enforced against. - ownership change detection fires when the bitmap changes even though the token list does not, which is the case a token-list hash cannot see. - isOwned handles an empty ring and a mismatched bitmap without panicking, which the previous ownership check did not. Two benchmarks are added for the new paths: the push path with and without a ring loaded, to show the per-series ownership cost, and the periodic recount separating the common unchanged-ring case from a full ownership re-evaluation. Signed-off-by: Anant Shanbhag <anantvas@amazon.com>

Ownership is computed from the raw replica-set walk and deliberately does not apply the replication strategy's health filter. That makes ownership differ from routing while an instance is transitioning, and until now that difference was only described in a comment rather than pinned by a test. For each of JOINING, LEAVING and READONLY, assert that the transitioning instance still owns its token ranges, that those ranges end up with more owners than the replication factor because the replica set is extended, and that writes still land on exactly replicationFactor healthy instances and never on the transitioning one. Both the transitioning instance and the instance the walk extends to really are holding that data, so both counting it is correct rather than double counting: the departing instance still has the series in its head, and the distributor really is writing to the extension. Filtering for health here would instead make an instance's own series count depend on whether its peers happened to miss a heartbeat, which is why the asymmetry with Ring.Get is intentional. Note that the aggregate invariant asserted by TestOwnedTokenPositions_Conservation, that every position has exactly replicationFactor owners, holds only for a ring where every instance is ACTIVE. This test covers the complementary case. Signed-off-by: Anant Shanbhag <anantvas@amazon.com>

The CHANGELOG carried three entries for this change. The two ENHANCEMENT lines described internal refactoring, moving the sharding helpers into pkg/ring/token.go and the FNV helpers into pkg/util/fnv.go, which is not something an operator reading release notes needs. Collapsed to the single FEATURE entry, reworded to describe the effect rather than the mechanism, marked experimental to match how the flags are listed in v1-guarantees, and with the "a up-to-2-hour" typo removed. It now says "up to one head compaction cycle", which is what the window actually is rather than assuming the default block range. The two new flags were never added to the generated configuration documentation, so `make check-doc` fails: it regenerates the docs and asserts `git diff --exit-code` over the config reference, the blocks-storage pages and the JSON config schema. Regenerated, which adds the twelve expected lines to docs/configuration/config-file-reference.md and the matching entries to schemas/cortex-config-schema.json. No other generated file changes. Also listed the feature under experimental features in docs/configuration/v1-guarantees.md, alongside the other opt-in ingester metrics, so the stability expectation is explicit while it is behind flags. Signed-off-by: Anant Shanbhag <anantvas@amazon.com>

pull-request-size Bot added the size/XXL label Aug 11, 2026

dosubot Bot added component/ingester component/ring type/feature type/performance labels Aug 11, 2026

anant10 force-pushed the feat/owned-series-tracking branch from 97d8157 to 305bc32 Compare August 11, 2026 17:28

danielblando reviewed Aug 14, 2026

View reviewed changes

anant10 requested a review from a team as a code owner October 7, 2026 01:04

anant10 requested a review from yeya24 October 7, 2026 01:04

anant10 added 7 commits October 7, 2026 01:19

anant10 force-pushed the feat/owned-series-tracking branch from 35b9338 to 640b980 Compare October 7, 2026 01:23

Navigation Menu

Repository navigation

feat(ingester): Add owned series tracking to prevent false throttling… - #7758

feat(ingester): Add owned series tracking to prevent false throttling…#7758
anant10 wants to merge 8 commits into
cortexproject:masterfrom
anant10:feat/owned-series-tracking

anant10 commented Aug 11, 2026 •

edited

Loading

Uh oh!

danielblando Aug 13, 2026

Uh oh!

anant10 Oct 7, 2026

Uh oh!

danielblando Aug 13, 2026

Uh oh!

anant10 Oct 7, 2026

Uh oh!

danielblando Aug 13, 2026

Uh oh!

anant10 Oct 7, 2026

Uh oh!

danielblando Aug 14, 2026

Uh oh!

anant10 Oct 7, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

anant10 commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How it works:

Design decisions:

Validation:

New configuration flags (experimental):

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

anant10 commented Aug 11, 2026 •

edited

Loading