Skip to content

fix(autoclaim): EthTxManager defaults and start-block defaulting - #1875

Merged
arnaubennassar merged 7 commits into
developfrom
fix/autoclaim-claimer-ethtxmanager-defaults
Oct 1, 2026
Merged

arnaubennassar merged 7 commits into
developfrom
fix/autoclaim-claimer-ethtxmanager-defaults

Conversation

@arnaubennassar

@arnaubennassar arnaubennassar commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

This PR collects a small set of operational fixes found while running autoclaim in a production
deployment. It started as a single fix for EthTxManager defaults and has grown a second,
unrelated fix (start-block defaulting) found during the same operating window — each small enough
on its own not to warrant a separate PR, but included here explicitly so reviewers know why this
is multi-part.

1. Apply EthTxManager defaults to claimers

  • Add Config.ApplyDefaults() in autoclaim/config/config.go, called once in config/config.go right after config unmarshalling and before AutoClaim.Validate(). It fills each [[AutoClaim.Claimers]] entry's EthTxManager fields that were left at their Go zero value with aggkit's own established AggOracle.EVMSender.EthTxManager opinion (config/default.go), and never touches a value the operator explicitly set.
  • Fix a key mismatch in config/default.go's [AggOracle.EVMSender.EthTxManager] block: it rendered TOML keys GetReceiptMaxTime / GetReceiptWaitInterval, but ethtxmanager.Config's mapstructure tags are WaitReceiptMaxTime / WaitReceiptCheckInterval (zkevm-ethtx-manager@v0.2.18/ethtxmanager/config.go:19,22), so those two AggOracle defaults were silently never binding. Verified against the vendored module source before including it.

Why

TOML array-of-tables elements (like [[AutoClaim.Claimers]]) get no per-element defaults from the config loader, and config/default.go's [AutoClaim] section stops at BridgeServiceFinder — it defines no [[AutoClaim.Claimers]] at all. So any claimer field an operator does not fully spell out starts from its Go zero value.

Two of those zero values are actively dangerous in github.com/0xPolygon/zkevm-ethtx-manager@v0.2.18/ethtxmanager:

  • FrequencyToMonitorTxs=0 → time.After(0) fires immediately, spinning the tx-monitor loop with no delay (ethtxmanager.go:485).
  • GasPriceMarginFactor=0 → the suggested gas price is multiplied by zero, so every transaction is rejected as transaction underpriced (ethtxmanager.go:989-991).

Both were hit back-to-back in a production deployment: a claimer without an explicit EthTxManager block first submitted underpriced transactions, then — once that surfaced and GasPriceMarginFactor alone was patched at the deploy-config layer — the same already-wedged monitored transaction kept retrying in the busy-loop, because re-pricing an existing transaction is only ever applied to a transaction that reached Sent status, and an underpriced-at-creation transaction never gets there. The result was ~1,500-1,600 log lines/second sustained for roughly 4 days, on the order of 100 million log lines, from a pod Kubernetes reported as perfectly healthy the entire time (no crash, no restart, probes green) — nothing in standard pod-health monitoring would have caught it.

What is defaulted, and why those values

Config.ApplyDefaults() mirrors aggkit's own AggOracle.EVMSender.EthTxManager values (config/default.go), so every EthTxManager instance in the binary behaves the same way absent explicit operator config:

Field Default Rationale
FrequencyToMonitorTxs 1s matches AggOracle; prevents the zero-delay busy-loop
WaitTxToBeMined 2s matches AggOracle
GetReceiptMaxTime (WaitReceiptMaxTime) 250ms matches AggOracle (now that the key is fixed)
GetReceiptWaitInterval (WaitReceiptCheckInterval) 1s matches AggOracle
GasPriceMarginFactor 1 (any value <= 0 is coerced to 1) a zero-or-negative factor is never a legitimate request — it means "never send"; 1 is a neutral no-margin multiplier
SafeStatusL1NumberOfBlocks 5 matches AggOracle
FinalizedStatusL1NumberOfBlocks 10 matches AggOracle
EstimateGasMaxRetries 1 matches AggOracle

A note on the last three: the vendored module documents 0 as a meaningful sentinel for SafeStatusL1NumberOfBlocks/FinalizedStatusL1NumberOfBlocks ("use the network's own safe/finalized tag") and for EstimateGasMaxRetries ("retry forever"). These aren't zero-values that are unconditionally broken the way the durations and gas margin are. But aggkit's own AggOracle path already overrides all three of those module sentinels with a fixed opinion, so this PR mirrors that same override for claimers, for consistency across every EthTxManager instance in the binary — rather than leaving claimers as the only place that silently falls through to the vendored zero-value behavior.

I considered a validation error instead of silent defaulting for some of these fields, but rejected it: unlike StoragePath (already required by ClaimerConfig.Validate), none of these fields have an "obviously correct" value an operator must supply — they're tuning knobs with the project's own established defaults, and requiring every deployment to spell out all eight of them just to avoid a footgun would only reproduce the same gap the next time a field is added to the vendored config. Defaulting to the project's own already-established opinion is safer than either silently zero-valuing or hard-failing.

Testing (fix 1)

  • go build ./... — passes.
  • go test ./autoclaim/... ./config/... — passes (existing suite unaffected; added TestApplyDefaultsFillsUnsetEthTxManagerFields, TestApplyDefaultsPreservesExplicitlySetEthTxManagerFields, TestApplyDefaultsCoercesNonPositiveGasPriceMarginFactor in autoclaim/config/config_test.go).

2. Default bridge-detector start blocks to a recent lookback, not genesis

Evidence

Both bridge detectors' start-block config fields — L1ToL2BridgeDetector.StartBlock and
L2ToLxBridgeDetector.StartL1Block (autoclaim/config/config.go, ~:67 and ~:81) — were plain
uint64, so 0 was indistinguishable from "unset", and 0 means scan from L1 genesis. This is the
same zero-value-means-something-dangerous class as the GasPriceMarginFactor fix in section 1.

In the same production deployment: a rollout that omitted these scanned ~6.9M Sepolia blocks,
taking ~19h, twice (once after a datadir wipe), and would have mass-enqueued years of stale
deposits the moment DryRun was lifted. The workaround was hand-pinning a wall-clock-derived magic
number (head minus 7 days) with a "re-pin if the deploy slips past date X" comment, which goes stale
by construction. On the L2-to-Lx side, if the resolved LER was zero or the configured block predated
the first L1 info tree leaf, initialFromLER (autoclaim/bridgedetector/l2_to_lx.go, ~:592) silently
fell back to full history
— invisible, and with no in-product recovery for bridges that fall below
a detector's start block (they are permanently unclaimable; the admin API only acts on
already-detected requests).

Fix

Both fields become *uint64: nil (the field absent from config) means "resolve automatically",
an explicit value — including 0 — is used verbatim and never adjusted, which is what makes this
change backwards compatible for every deployment that already pins a start block. A new
StartLookback cfgtypes.Duration field (default 24h, applied in ApplyDefaults()) controls how
far back an unset value resolves from.

mapstructure/viper in this codebase decode *uint64 correctly out of the box — verified directly
against this repo's own viper/mapstructure versions before committing to the pointer approach
(absent key → nil; = 0 → pointer to 0; = 42 → pointer to 42) — so no sentinel value or
companion isSet field was needed.

Resolution (autoclaim/bridgedetector/startblock.go, ResolveStartBlock): when unset, it
binary-searches L1 block headers (EthClienter.CustomHeaderByNumber) for the highest block at or
before now - StartLookback — about log2(head-minBlock) header lookups, not a linear scan — then
clamps the result to [minBlock, head]. minBlock is the L1 bridge sync's own configured genesis
block for L1ToL2BridgeDetector (BridgeL1Sync.InitialBlockNum) and the RollupManager contract's
creation block for L2ToLxBridgeDetector (L1NetworkConfig.RollupManagerCreationBlock, which
config validation already guarantees is > 0) — a detector can never usefully start earlier than
the data source it reads from or the contract whose events it needs. It deliberately does not
estimate the block number from an average block-time constant: an orchestration tool used elsewhere
in this operating window assumed 12.0s/block on Sepolia when the real figure was 12.512s, which
skewed a multi-hour ETA by 14 hours — reading real timestamps back from the chain avoids that class
of error entirely.

StartLookback defaults to 24h: a bridge only becomes claimable after certificate settlement
and destination GER injection, which has been observed in production to take over an hour, so
a short window risks leaving valid bridges permanently unclaimable, while 24h of Sepolia is only
~7,000 blocks versus the ~6.9M actually backfilled by the genesis default.

The resolution runs once at startup per enabled detector and logs an INFO naming the lookback, the
resolved block and its timestamp, plus a WARN that bridges originating before that block will
never be autoclaimed and need manual claiming.

config/default.go previously baked StartBlock = 0 / StartL1Block = 0 into the rendered
config template for every deployment. Those lines are removed (not set to a placeholder — simply
absent), since leaving them in would have made every existing deployment "explicit genesis" forever
and defeated this fix entirely; the Go-side ApplyDefaults() is now the only place StartLookback
gets a default, matching how the EthTxManager defaults in section 1 are applied.

Silent-fallback decision: WARN + per-source retry, not a hard error

initialFromLER's two silent-fallback branches (leaf-not-found, zero LER) now return a new
ErrCannotResolveInitialLER sentinel instead of nil, nil — but only when the configured start
block is non-zero; StartL1Block = 0 remains an explicit, honored request for full history, and
since automatic resolution is always clamped to a strictly-positive minBlock, a literal 0 can
only ever be deliberate operator intent, never a resolution artifact.

I initially made this a hard error, but initialFromLER's caller (resolveFromLER →
processSource) propagates any error all the way up through PollOnce, which aborts the entire
poll — meaning one source with a bad StartL1Block relative to its history would silently stall
every other source's discovery too, until an operator intervened. That is too large a blast radius
for a per-source config problem, and risks the same "wedged and silently retrying" pattern the
EthTxManager fix in section 1 exists to prevent. Instead, processSource now catches
ErrCannotResolveInitialLER specifically and treats it like the existing finder-miss/not-synced-yet
cases: it logs a prominent WARN naming the source and remediation ("lower StartL1Block, or set it
to 0"), and returns sourceRetryLater — that source alone is retried every poll until fixed;
every other source in the same poll is unaffected.

Testing (fix 2)

  • go build ./... — passes.
  • go test ./autoclaim/... ./config/... ./cmd/... — passes. Added autoclaim/bridgedetector/startblock_test.go (TestResolveStartBlock*: explicit value/explicit-zero used verbatim with zero RPC calls, unset resolves from lookback, clamping at both the head and the configured floor, L1-client error propagation); updated the two existing l2_to_lx_test.go silent-fallback tests to cover the explicit-genesis path, and added their non-explicit-genesis counterparts asserting the new retry-later behavior (no candidates fetched, poll still succeeds, SkippedSourceCount increments); updated config/config_test.go and autoclaim/runtime/runtime_test.go fixtures for the pointer field.
  • gofmt -l — clean.
  • go vet ./... (whole repo, including test files) — clean.
  • golangci-lint run ./autoclaim/... ./config/... ./cmd/... (v2.4.0) — 0 issues (one mnd finding on the binary-search midpoint calculation addressed with //nolint:mnd, matching the existing convention in types/block_finality.go).

Also updates docs/autoclaim.md

The default-value callouts and config-reference table rows for StartBlock/StartL1Block
described the old "default 0" behavior directly; left as-is they would have actively misled
operators about the very fields this fix changes, so they are updated in place alongside two new
StartLookback rows. No other content in that file changed.

Out of scope (follow-ups, not implemented here)

  • A Prometheus gauge for the resolved start block.
  • Refusing to start when the DB is empty but the chain has history beyond the configured/resolved
    window (the datadir-wipe gap hazard from section 1's incident — a real problem, but too invasive
    for this PR).

🐞 Issues

Cross-references agglayer/aggkit#1874 and
0xPolygon/zkevm-ethtx-manager#139 (does
not close either).

This PR removes the cause (an unconfigured EthTxManager reaching zero values), but the deadlock
filed as zkevm-ethtx-manager#139 remains a separate, module-level defect: an underpriced-at-creation
transaction can never be re-priced, because re-pricing is gated on MonitoredTxStatusSent, a status
that transaction never reaches. A wedged claimer created before this fix ships still needs the
operational recovery described there; this PR only prevents new claimers from getting into that state.

📝 Notes

  • Kept the diff to the config and logging layers only — no vendored zkevm-ethtx-manager changes.
  • Changelog-worthy behavior change: a deployment that previously omitted
    L1ToL2BridgeDetector.StartBlock / L2ToLxBridgeDetector.StartL1Block will now resolve a recent
    block via StartLookback (default 24h) instead of scanning from genesis. A deployment that already
    sets either field explicitly, to any value including 0, is unaffected.
  • Section 2 (start-block defaulting) is meaningfully larger than section 1 — it adds a new
    resolution algorithm and a config-template default, not just a defaulting pass. The owner chose to
    include it here as a second small operational fix from the same deployment window, but it is the
    one most defensible as its own PR if reviewers would rather split it out.
  • The IgnoreNetworkIDs skip-source logging change that was previously part of this PR has been
    dropped from here. It is now handled in
    #1882, together with the underlying l2-to-lx
    cursor fix that log line is the visible symptom of — changing the log level alone would have hidden
    the evidence while leaving the bug in place.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DJuYNKE9jRLRrCWTrrEJ2T

TOML array-of-tables elements like [[AutoClaim.Claimers]] get no per-element
defaults from the config loader, and config/default.go's AutoClaim section
stops at BridgeServiceFinder, so every claimer's EthTxManager that an operator
does not fully spell out starts from Go zero values.

Two zero values are actively dangerous in
github.com/0xPolygon/zkevm-ethtx-manager@v0.2.18/ethtxmanager:
  - FrequencyToMonitorTxs=0 makes time.After(0) fire immediately, spinning the
    tx-monitor loop with no delay (ethtxmanager.go:485).
  - GasPriceMarginFactor=0 multiplies the suggested gas price by zero, so every
    tx is rejected as "transaction underpriced" (ethtxmanager.go:989-991).

Add Config.ApplyDefaults (autoclaim/config/config.go), called once in
config/config.go right after unmarshalling and before Validate, which fills
each claimer's EthTxManager zero-valued fields with aggkit's own
AggOracle.EVMSender.EthTxManager opinion (config/default.go): 1s monitor
frequency, 2s wait-to-be-mined, 250ms/1s receipt polling, a
GasPriceMarginFactor of 1 (<=0 coerced to 1, since a zero factor can never be
a legitimate request), and the network 5/10 safe/finalized block overrides
plus 1 max gas-estimate retry. It never touches a value the operator set.

Also fix config/default.go: AggOracle.EVMSender.EthTxManager rendered the TOML
keys GetReceiptMaxTime/GetReceiptWaitInterval, but ethtxmanager.Config's
mapstructure tags are WaitReceiptMaxTime/WaitReceiptCheckInterval
(zkevm-ethtx-manager@v0.2.18/ethtxmanager/config.go:19,22), so those two
AggOracle defaults were silently never binding.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DJuYNKE9jRLRrCWTrrEJ2T
@arnaubennassar

Copy link
Copy Markdown
Collaborator Author

@claude

@claude

claude Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Claude finished @arnaubennassar's task in 4m 26s —— View job


Review: fix(autoclaim): apply EthTxManager defaults to claimers

The diagnosis is solid and the write-up is unusually good — [[AutoClaim.Claimers]] really does get no per-element defaults from LoadFile's default-TOML merge, and FrequencyToMonitorTxs=0 / GasPriceMarginFactor=0 really are the two zero values that produce the failure mode described. One nice side effect worth noting: because ApplyDefaults runs inside loadString (config/config.go:500) before SaveConfigToFile (config/config.go:461), the effective values show up in the saved merged config, so an operator can actually see them.

Two things I'd want addressed before merge, and a few smaller ones.


🔴 Required

1. Explicit 0 is silently rewritten for the three sentinel fields — and the repo's own docs tell operators to write 0

docs/autoclaim.md:559-560, the example claimer config in this repo, contains:

SafeStatusL1NumberOfBlocks = 0
FinalizedStatusL1NumberOfBlocks = 0

ApplyDefaults operates on the already-unmarshalled struct, so it cannot distinguish "operator omitted the key" from "operator wrote 0 on purpose". After this PR, anyone who copied the documented example gets 5 / 10 instead of the network's own safe/finalized tags, with no log line saying a substitution happened. The PR body anticipates this objection and argues consistency with AggOracle, which is a reasonable position — but it can't also be true that the values are left alone when explicitly set (autoclaim/config/config.go:136-137, and the PR's own "never touches a value the operator explicitly set"). For these three fields it does.

Either of these resolves it:

  • Narrow the defaulted set to the fields whose zero value is unconditionally broken — FrequencyToMonitorTxs, GasPriceMarginFactor, and the two receipt durations — and leave the three documented sentinels alone; or
  • Keep the override, but update docs/autoclaim.md:559-560 so the example no longer instructs something the loader will overwrite.

Independently of which you pick: log each substitution at info level. Silently rewriting operator config is the kind of thing that costs an hour the next time someone is debugging why their configured value "isn't taking". Fix this →

2. The GetReceiptMaxTime → WaitReceiptMaxTime key fix is applied in 1 of ~8 live places

If the mapstructure tags really are WaitReceiptMaxTime / WaitReceiptCheckInterval (I could not verify the vendored source here — see caveat below), then the same silent non-binding still exists after this PR in:

Location Impact
tools/force_ger_update/example-config.toml:47-48 live config for a shipped tool — same footgun, unfixed
tools/force_ger_update/README.md:93-94, docs/ethtxmanager.md:12-13 docs teach the non-binding key
config/config_test.go:391-392, tools/force_ger_update/config_test.go:109-110 fixtures assert nothing about these, so they pass either way
test/e2e/forcegerupdate_test.go:251-252 e2e config
test/config/*.template, test/e2e/kurtosis/*.template.toml, test/e2e/envs/**/aggkit-config.toml commented-out, so harmless, but they document the wrong key

tools/force_ger_update is the interesting one: it embeds ethtxmanager.Config the same way (tools/force_ger_update/config.go:68), has no ApplyDefaults equivalent, and its example config uses the non-binding spelling. That's the identical defect this PR exists to fix. Fixing the spelling repo-wide in this PR is cheap; if you'd rather not, please say so explicitly in the description so it doesn't get lost. Fix this →


🟡 Suggested

3. No test covers the config/default.go fix, which is a real behaviour change for AggOracle. Grepping the repo, nothing asserts on AggOracle.EVMSender.EthTxManager.GetReceiptMaxTime after a default load — which is exactly why the mismatch survived. Add an assertion that loading the default config yields 250ms / 1s there. Also: the PR body frames this as "silently never binding", but the effect of fixing it is that AggOracle's receipt timings move from 0 to 250ms/1s in every existing deployment. That belongs in the release notes, not just in a parenthetical.

4. WaitReceiptMaxTime = "250ms" with WaitReceiptCheckInterval = "1s" looks swapped. A 250 ms total budget with a 1 s poll interval means the budget expires before the first check. These values never bound before, so nobody would have noticed. Worth confirming against the module's receipt loop before making them live — and the same pair is copied verbatim into autoclaim/config/config.go:28-29, so a mistake propagates to claimers too.

5. Constant names don't match the fields they're assigned to. autoclaim/config/config.go:161-166 assigns DefaultEthTxManagerWaitReceiptMaxTime to cfg.GetReceiptMaxTime. Two names for one thing on adjacent lines. Since these constants are consumed by Go code, naming them after the Go fields (DefaultEthTxManagerGetReceiptMaxTime) reads better; leave the TOML-tag spelling to default.go and the docs.

6. docs/autoclaim.md's claimer table (lines 615-625) should record the new defaults. Right now EthTxManager is documented as Yes (required) with no mention that eight of its fields are now filled in for you. That table is where an operator will look.


🟢 Optional

7. All eight Default* constants are exported but used only within the package and its tests — unexport unless the public surface is intentional.

8. Defaults are applied to disabled claimers too. Harmless, and arguably right for consistency, just noting it's unconditional.

9. require.InDelta(t, ..., 0) for float equality (config_test.go:270, :326) works but require.Equal(t, float64(DefaultEthTxManagerGasPriceMarginFactor), ...) states the intent more directly.

10. Config.Checksum() / the saved merged config now include the defaulted values, so InternalConfigChecksum will shift for existing deployments on upgrade. Expected, but someone watching that field will see a change with no config edit behind it.

11. Adjacent gap, not this PR: ClaimerConfig.Validate requires EthTxManager.StoragePath but not EthTxManager.Etherman.URL, which is equally fatal downstream at ethtxmanager.New (autoclaim/runtime/runtime.go:210).


⚠️ Verification caveat

I could not verify the central claim of change #2 in this environment: the Go module cache is outside my accessible paths and network fetches (WebFetch, gh api to the upstream repo) were not permitted, so I could not read zkevm-ethtx-manager@v0.2.18/ethtxmanager/config.go to confirm the mapstructure tags. go build and go test were likewise unavailable, so I did not independently run the build or the test suite — I'm taking the PR's reported results at face value. What I can say is that the claim is corroborated from inside the repo: docs/autoclaim.md:549-550 and all six test/e2e/autoclaim_test.go claimer fixtures already use the WaitReceiptMaxTime spelling, while the older force_ger_update and AggOracle configs use the other one. A reviewer with module access should confirm the tags directly before merging — findings #2 and #4 both hinge on it.


🚀 What's New

autoclaim/config.Config.ApplyDefaults() fills each [[AutoClaim.Claimers]] entry's EthTxManager fields that were left at their Go zero value with aggkit's established AggOracle.EVMSender.EthTxManager values. It's called once from loadString (config/config.go:500), after viper unmarshalling and before AutoClaim.Validate(), so the defaults are also reflected in the saved merged config. TOML array-of-tables elements get no per-element defaults from the default-TOML merge, so before this every unspecified claimer field started from its Go zero value.

🐛 Bug Fixes

  • A claimer configured without an explicit EthTxManager block no longer starts with FrequencyToMonitorTxs = 0, which makes the tx-monitor loop's time.After(0) fire immediately and busy-spin (observed in production at ~1,500 log lines/second for ~4 days, on a pod that reported healthy throughout).
  • A claimer no longer starts with GasPriceMarginFactor = 0, which multiplies the suggested gas price by zero and gets every transaction rejected as transaction underpriced. Any value <= 0 is coerced to 1.
  • config/default.go's [AggOracle.EVMSender.EthTxManager] block used the TOML keys GetReceiptMaxTime / GetReceiptWaitInterval, which don't match the vendored ethtxmanager.Config mapstructure tags, so those two AggOracle defaults never bound. Renamed to WaitReceiptMaxTime / WaitReceiptCheckInterval. Not a no-op: AggOracle's receipt timings move from 0 to 250ms / 1s. The same misspelling remains in tools/force_ger_update and in docs/ethtxmanager.md (see 🔴 Split L1 / L2 responsibilities for lastgersync #2).
  • The module-level deadlock in 0xPolygon/zkevm-ethtx-manager#139 is not fixed — an underpriced-at-creation transaction still can never be re-priced, because re-pricing is gated on MonitoredTxStatusSent. A claimer already wedged before this ships still needs manual recovery.

📋 Config Updates

No new config keys. Eight existing [[AutoClaim.Claimers]].EthTxManager fields are now defaulted when omitted:

Field (Go / TOML) Default
FrequencyToMonitorTxs 1s
WaitTxToBeMined 2s
GetReceiptMaxTime / WaitReceiptMaxTime 250ms
GetReceiptWaitInterval / WaitReceiptCheckInterval 1s
GasPriceMarginFactor 1 (any value <= 0 is coerced)
SafeStatusL1NumberOfBlocks 5
FinalizedStatusL1NumberOfBlocks 10
EstimateGasMaxRetries 1

Two TOML keys renamed under [AggOracle.EVMSender.EthTxManager]: GetReceiptMaxTime → WaitReceiptMaxTime, GetReceiptWaitInterval → WaitReceiptCheckInterval. EthTxManager.StoragePath remains required per claimer and is untouched by the defaulting.

⚠️ Breaking Changes

No API or schema breakage, but three silent behavioural changes on upgrade:

  1. A claimer that explicitly sets SafeStatusL1NumberOfBlocks = 0, FinalizedStatusL1NumberOfBlocks = 0, or EstimateGasMaxRetries = 0 — which the repo's own example at docs/autoclaim.md:559-560 does for the first two — gets 5 / 10 / 1 instead, unlogged. See 🔴 Unify L1 syncers #1.
  2. AggOracle's receipt timings change from 0 to 250ms / 1s now that the keys bind.
  3. InternalConfigChecksum shifts for existing deployments, since the defaults land in the config before it's hashed.

Review only — no code changes were made or pushed. The build and test suite were not run in this environment (see verification caveat).
• branch fix/autoclaim-claimer-ethtxmanager-defaults

@arnaubennassar arnaubennassar changed the title fix(autoclaim): apply EthTxManager defaults to claimers fix(autoclaim): apply EthTxManager defaults and reduce log noise Sep 29, 2026
@arnaubennassar arnaubennassar changed the title fix(autoclaim): apply EthTxManager defaults and reduce log noise fix(autoclaim): EthTxManager defaults, start-block defaulting, and log noise Sep 29, 2026
Comment thread autoclaim/bridgedetector/startblock.go Fixed
…back, not genesis

Both bridge detectors' start-block fields (L1ToL2BridgeDetector.StartBlock,
L2ToLxBridgeDetector.StartL1Block) were plain uint64, so 0 was indistinguishable
from "unset" and meant "scan from genesis". A production deployment that omitted
them scanned ~6.9M Sepolia blocks (~19h) twice, and the L2-to-Lx side silently
fell back to full history whenever it could not resolve a baseline LER -- the
same zero-value-means-something-dangerous class as the GasPriceMarginFactor fix
already in this PR.

Both fields become *uint64 (nil = unset, explicit value incl. 0 used verbatim),
with a new StartLookback duration (default 24h) used to resolve an unset value
by binary-searching L1 block timestamps for "now - lookback" and clamping to
[minBlock, head] -- never below the L1 bridge sync's/RollupManager's own
genesis, never above head. This avoids estimating a block number from an
average block-time constant, which has been observed elsewhere in this
codebase to be wrong enough to skew a multi-hour ETA by over ten hours.

The L2-to-Lx detector's silent fall-back to full history when it cannot derive
an initial LER (predates the first L1 info tree leaf, or resolves to a zero
LER) is replaced with a loud, per-source retry: it no longer fails the whole
poll (that would let one bad source block every other source), just skips and
retries that one source every poll with a WARN, until the operator fixes the
config or explicitly opts into full history via StartL1Block=0.

Also updates config/default.go, which previously baked StartBlock=0/
StartL1Block=0 into every rendered config -- leaving those lines in place would
have made every existing deployment "explicit genesis" forever and defeated
this fix entirely.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DJuYNKE9jRLRrCWTrrEJ2T
@arnaubennassar
arnaubennassar force-pushed the fix/autoclaim-claimer-ethtxmanager-defaults branch from 0332cac to 9b48c33 Compare September 29, 2026 16:31
@arnaubennassar arnaubennassar changed the title fix(autoclaim): EthTxManager defaults, start-block defaulting, and log noise fix(autoclaim): EthTxManager defaults and start-block defaulting Sep 29, 2026
arnaubennassar added a commit that referenced this pull request Sep 30, 2026
…-lx block cursor (#1882)

## 🔄 Changes Summary
- **Fix**: a source network permanently excluded from bridge service
resolution no longer pins the `L2ToLx` detector's shared L1 block-window
cursor, which previously stopped **all** L2-origin claim discovery.

`autoclaim/bridgedetector/l2_to_lx.go` keeps **one** block-window cursor
(`l2-to-lx`) shared by every source network. `processSource` returned
`sourceRetryLater` for *any* `fetcher.GetURL` error, and `PollOnce` then
held that shared cursor just before the earliest skipped source's verify
row — or, when that row sat at the very start of the window, `return
result, nil` **without saving the cursor at all**, re-scanning the
identical window on the next poll.

`bridgeservicefinder.GetURL` returns `ErrNetworkDisabled` for every
network listed in `Config.IgnoreNetworkIDs`. That error is a deliberate,
permanent exclusion — it can never clear. So a single ignored network
with a `VerifyBatches` row anywhere in the scanned range wedged the
shared cursor **forever**, and no L2-origin claim request could ever be
created for **any** source.

Observed in a deployment whose ignore list covers most registered
rollups: over 8 hours the only `skip source` line emitted named an
ignored network, and the request store contained L1-origin requests
exclusively — not one L2-origin request.

**The fix**: a permanently excluded source is `sourceUpToDate` ("nothing
to do") instead of `sourceRetryLater`. It no longer contributes to
`retryBlock` and no longer holds the window back. Its own `(source,
destination)` LER cursors are still left untouched, so when the network
is later un-ignored, its **next** verify row is fetched from the old
pair cursor and covers everything bridged in between. That recovery does
depend on the source publishing another verify row: the shared window
has already moved past the row seen while the source was excluded and
only rewinds by `overlapBlocks` (default 1), so it never returns to it.
To un-ignore a source that has **stopped** producing verified batches,
reset the `l2-to-lx` block-window cursor so its old verify row falls
inside the scanned range again. The trade is deliberate — the old
behaviour bought that re-observability at the cost of deadlocking every
source. A genuinely retryable `GetURL` failure (service not yet
announced, or unhealthy) keeps its existing hold-the-window behaviour,
unchanged.

Two smaller points:
- The permanence distinction is expressed in the fetcher contract rather
than read out of `bridgeservicefinder` by the detector.
`ClaimCandidatesFetcher` gains a detector-local `ErrSourceDisabled`, the
permanent counterpart of `ErrURLNotFound`; `ServiceFetcher.GetURL` — the
designated translation point — maps the finder's disabled error to it,
checked **before** the not-found branch that error wraps, and still
wrapping `ErrURLNotFound` since an excluded source is also a source with
no URL. `processSource` branches on the local sentinel and no longer
imports `bridgeservicefinder`, and both sentinels are documented on the
`GetURL` contract: the whole bug was a permanence distinction the
contract never expressed. A test pins the mapping, so a future refactor
cannot silently reintroduce the deadlock, and any non-finder-backed
fetcher can now express the same thing.
- An excluded source is now reported once per source rather than on
every poll. The exclusion never clears and the detector polls for the
lifetime of the process, so the repetition carried no information.

The doc comments on `sourceOutcome`, `PollOnce` and `processSource` said
"transient" where the code accepted any error; they are updated to match
the new semantics.

## ⚠️ Breaking Changes
- None. No config, API, CLI or storage change.

## 📋 Config Updates
- None.

## 🔌 API Updates
- None.

## ✅ Testing
- 🤖 **Automatic**: 6 new tests in `autoclaim/bridgedetector`:
- `TestL2ToLxDisabledSourceDoesNotHoldBlockCursor` — an excluded source
does not hold the cursor back; the window advances past its verify row.
- `TestL2ToLxTransientURLErrorStillHoldsBlockCursor` — a retryable URL
miss **still** holds the cursor (guards against regressing the existing
behaviour).
- `TestL2ToLxDisabledSourceDoesNotStarveHealthySource` — mixed window:
one excluded source plus one healthy source at a later block; the
healthy source processes and the cursor advances.
- `TestL2ToLxDisabledSourceAtWindowStartDoesNotPinCursor` — the
production regression: an excluded source's verify row at the very start
of the window must not take the "return without saving the cursor" path.
- `TestL2ToLxDisabledSourceIsReportedOncePerSource` — the report is
emitted once, not per poll.
-
`TestServiceFetcherGetURL/network_disabled_maps_to_the_permanent_sentinel`
— pins that the finder's disabled error arrives at the detector as
`ErrSourceDisabled`, never as a plain retryable `ErrURLNotFound`,
including the branch ordering that makes it so.
- **Confirmed to be genuine regression tests.** With the test files in
place and `l2_to_lx.go` reverted to its pre-fix state, 4 of the 5
behavioural tests fail:
  ```
--- FAIL: TestL2ToLxDisabledSourceDoesNotHoldBlockCursor (cursor 0x13 =
19, want 0x31 = 49)
--- FAIL: TestL2ToLxDisabledSourceDoesNotStarveHealthySource (cursor
0x13 = 19, want 0x31 = 49)
--- FAIL: TestL2ToLxDisabledSourceAtWindowStartDoesNotPinCursor
(CursorAdvanced: should be true)
--- FAIL: TestL2ToLxDisabledSourceIsReportedOncePerSource (mock:
Unexpected Method Call)
--- PASS: TestL2ToLxTransientURLErrorStillHoldsBlockCursor (behaviour
intentionally unchanged)
--- PASS: TestServiceFetcherGetURL (pins existing behaviour by design)
  ```
All pass with the fix applied, alongside the pre-existing retry tests
(`TestL2ToLxNotSyncedSkipsSourceWithoutAdvancingCursor`,
`TestL2ToLxRetriesSkippedSourceOnLaterPollWithoutNewLER`,
`TestL2ToLxRetrySkipAtWindowStartKeepsStoredCursor`).
- `go test ./autoclaim/... ./bridgeservicefinder/...` — all packages
`ok`.
- `golangci-lint run --timeout 5m` — `0 issues.`
- 🖱️ **Manual**: none required.

## 🐞 Issues
- No tracking issue.

## 🔗 Related PRs
- #1875 also touches `autoclaim/bridgedetector/l2_to_lx.go` (and its
test file) in the same `processSource` `GetURL` error branch, so the two
will conflict textually on merge. The changes are logically independent:
#1875 only lowers the log level of the skip line, this PR changes the
control flow so the shared cursor is not held. This PR is based directly
on `develop` and contains nothing from #1875.

## 📝 Notes
- Design decision: the excluded source's per-pair LER cursors are
deliberately **not** advanced. The block window is only the trigger for
re-examining a source; the per-`(source, destination)` LER cursor is
what bounds the actual candidate fetch. Leaving it untouched means that
if the operator later un-ignores the network, its next verify row is
fetched from the old pair cursor and still covers every bridge exit made
while it was excluded — even though the block window has long since
moved past. The converse is the operational caveat above: a re-enabled
source that never publishes another verify row is never re-examined,
because the window does not rewind to the row it already passed.
Resetting the block-window cursor is the recovery for that case.
- The once-per-source reporting is a plain `map[uint32]struct{}` on the
detector, touched only from the single poll goroutine — deliberately not
a general log-dedup mechanism.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01DJuYNKE9jRLRrCWTrrEJ2T

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
arnaubennassar and others added 2 commits October 1, 2026 14:32
…mer-ethtxmanager-defaults

# Conflicts:
#	autoclaim/bridgedetector/l2_to_lx.go
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Comment thread autoclaim/bridgedetector/startblock.go Fixed
…p (CodeQL)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Comment thread autoclaim/bridgedetector/startblock.go Fixed
…o int64 (CodeQL)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@arnaubennassar
arnaubennassar enabled auto-merge (squash) October 1, 2026 14:14
@joanestebanr

Copy link
Copy Markdown
Collaborator

@claude review

@claude

claude Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Claude finished @joanestebanr's task in 1m 19s —— View job


Review of #1875: EthTxManager defaults and start-block defaulting

The diagnosis is solid and the PR is well documented. The latest commits also resolved the CodeQL uint64 -> int64 alerts: the timestamp is now logged as a plain uint64, so there is no conversion left. The previous review's concerns about sentinel fields and the key rename are still open, and I found a few new issues in the start-block work.

I read the diff only. go build, go test and lint were not run in this environment. I also couldn't read the zkevm-ethtx-manager module source to confirm its mapstructure tags (the shell command was denied). The repo's own docs/autoclaim.md and the autoclaim e2e fixtures already use WaitReceiptMaxTime, which supports the PR's claim.


🔴 Required

1. Explicit 0 is still silently rewritten for the sentinel fields.

  • applyEthTxManagerDefaults (autoclaim/config/config.go:~183-195) replaces SafeStatusL1NumberOfBlocks == 0, FinalizedStatusL1NumberOfBlocks == 0 and EstimateGasMaxRetries == 0. It runs on the unmarshalled struct, so it can't tell "omitted" from "wrote 0 on purpose".
  • The repo's own example at docs/autoclaim.md:569-570 sets SafeStatusL1NumberOfBlocks = 0 and FinalizedStatusL1NumberOfBlocks = 0. After this PR those become 5 and 10, with no log line.
  • That contradicts the PR's "never touches a value the operator explicitly set".
  • Either narrow the defaulted set to the fields whose zero value is always broken (FrequencyToMonitorTxs, GasPriceMarginFactor and the two receipt durations), or keep the override and fix the docs example. In both cases, log each substitution at info level.

2. ErrBlockNotProcessed still aborts the whole poll, and auto-resolution makes it likelier.

  • l1infotreesync.GetLatestL1InfoLeafUntilBlock (processor.go:~235) returns ErrBlockNotProcessed when the L1 info tree sync hasn't reached blockNum. translateError does not map it to ErrNotFound, so initialFromLER (l2_to_lx.go:~731) wraps it as a generic error. That error propagates out of processSource and aborts the entire PollOnce.
  • That is the blast-radius problem the PR fixed for ErrCannotResolveInitialLER, and the same hazard is still open here.
  • It was rare with a hand-pinned old block. It is now the default case. On a fresh datadir the resolved block is head - 24h, and the L1 info tree sync starts far behind that. Every poll errors until the sync catches up, and no source is processed in the meantime.
  • Treat ErrBlockNotProcessed like the other "retry later" cases for that source, probably with an info-level log.

🟡 Suggested

3. The WARN on ErrCannotResolveInitialLER fires every poll.

  • It is logged once per source per poll (PollInterval defaults to 3s) with no throttling (l2_to_lx.go:~522). This PR exists because of a ~1,500 line/s log flood from a healthy-looking pod, so a permanent per-poll WARN is the same pattern at a lower rate.
  • Throttle it, for example by logging once per source when the state changes, or at most every N minutes.
  • With auto-resolution this can also be a normal transient: the resolved block can be earlier than the first leaf of an L1 info tree sync configured with a recent InitialBlock, so the source retries forever.

4. The resolved start block is not persisted, so it moves forward on every restart.

  • ResolveStartBlock recomputes now - lookback at each startup.
  • For L1-to-L2 this is harmless, because the durable cursor wins.
  • For L2-to-Lx, any source or destination pair first seen after a restart gets its baseline from the new, later block. Bridges between the old and new baselines are never claimed, and they were inside the window the operator expected. The warning log says "bridges before block N", but N changes across restarts.
  • At minimum, document that the window is relative to process start. Persisting the first resolved value would be better, and it overlaps with the datadir-wipe follow-up you listed.

5. Validate StartLookback.

  • Validate() doesn't reject a negative StartLookback. In resolveBlockAtLookback a negative value makes uint64(lookback.Seconds()) wrap, so targetTime lands in the future and the search silently returns head.
  • Add StartLookback > 0 to Validate(), for the enabled detectors.

6. resolveBlockAtLookback doesn't actually clamp to minBlock when head.Number <= minBlock.

  • It returns head.Number, which is below minBlock. The doc comment says [minBlock, head].
  • This only happens on a misconfigured floor, but the comment and the code disagree.

7. Constant names don't match the Go fields they are assigned to.

  • DefaultEthTxManagerWaitReceiptMaxTime is assigned to cfg.GetReceiptMaxTime, and ...WaitReceiptCheckInterval to cfg.GetReceiptWaitInterval (config.go:~176-180).
  • Name them after the Go fields.
  • The constants are exported but used only inside the package and its tests, so consider unexporting them.

8. The GetReceiptMaxTime → WaitReceiptMaxTime key fix covers only config/default.go. The old spelling remains in:

  • tools/force_ger_update/example-config.toml:47-48, which is a live config for a shipped tool that embeds ethtxmanager.Config and has no ApplyDefaults. It has the same non-binding bug.
  • tools/force_ger_update/README.md:93-94
  • docs/ethtxmanager.md:12-13
  • the commented-out entries in test/config/*.template, test/e2e/kurtosis/*.template.toml and test/e2e/envs/**/aggkit-config.toml

Fix them in this PR, or list them as an explicit follow-up in the description.

9. Add a test that loading the default config yields 250ms / 1s for AggOracle.EVMSender.EthTxManager's receipt timings.

  • Nothing asserts this today, which is why the key mismatch survived.
  • 250ms max time with a 1s check interval also looks inverted: the budget expires before the first poll. Worth checking against the module's receipt loop now that these values bind.

10. Document the new EthTxManager defaults in the claimer table in docs/autoclaim.md. EthTxManager is still listed as required with no mention that eight fields are now filled in.


🟢 Optional

  • Defaults are applied to disabled claimers too. That is harmless, just unconditional.
  • Use require.Equal with a float64 instead of require.InDelta(..., 0) for the margin-factor assertions in autoclaim/config/config_test.go.
  • InternalConfigChecksum will change on upgrade with no config edit behind it, because the defaults are applied before the config is hashed.
  • ClaimerConfig.Validate requires EthTxManager.StoragePath but not Etherman.URL, which is equally fatal at ethtxmanager.New. This is a separate gap.
  • The PR mixes two unrelated fixes, as the description admits. The start-block half (~400 lines, new algorithm) is the better candidate to split out if reviewers want a smaller diff.

🚀 What's New

  • autoclaim/config.Config.ApplyDefaults() fills each [[AutoClaim.Claimers]] entry's unset EthTxManager fields with the same values AggOracle uses. It is called from config/config.go after unmarshalling and before AutoClaim.Validate().
  • L1ToL2BridgeDetector.StartBlock and L2ToLxBridgeDetector.StartL1Block become *uint64. When absent, a new bridgedetector.ResolveStartBlock binary-searches L1 headers for the block at now - StartLookback, clamped to [minBlock, head]. Explicit values, including 0, are used verbatim.
  • L2ToLx.initialFromLER no longer silently falls back to full history for a non-zero StartL1Block. It returns ErrCannotResolveInitialLER, and processSource turns that into a per-source WARN and retry-later.
  • docs/autoclaim.md is updated for the new start-block behavior.

🐛 Bug Fixes

  • A claimer with no EthTxManager config no longer gets FrequencyToMonitorTxs = 0, which busy-spun the tx monitor, or GasPriceMarginFactor = 0, which made every transaction underpriced. Any margin factor <= 0 is coerced to 1.
  • config/default.go used GetReceiptMaxTime and GetReceiptWaitInterval, which don't match the ethtxmanager.Config mapstructure tags, so AggOracle's receipt timings were never applied. They are renamed to WaitReceiptMaxTime and WaitReceiptCheckInterval.
  • Unset bridge-detector start blocks no longer default to genesis, which scanned about 6.9M Sepolia blocks.
  • The module-level deadlock in zkevm-ethtx-manager#139 is not fixed. A claimer already wedged needs manual recovery.

📋 Config Updates

New key (both detectors), defaulted in ApplyDefaults:

[AutoClaim.L1ToL2BridgeDetector]
StartLookback = "24h"   # used only when StartBlock is absent
[AutoClaim.L2ToLxBridgeDetector]
StartLookback = "24h"   # used only when StartL1Block is absent
  • StartBlock and StartL1Block have no default now. The = 0 lines are removed from the rendered template.
  • Claimer EthTxManager fields now defaulted when unset:
Field Default
FrequencyToMonitorTxs 1s
WaitTxToBeMined 2s
WaitReceiptMaxTime 250ms
WaitReceiptCheckInterval 1s
GasPriceMarginFactor 1 (<= 0 coerced)
SafeStatusL1NumberOfBlocks 5
FinalizedStatusL1NumberOfBlocks 10
EstimateGasMaxRetries 1
  • The AggOracle template keys are renamed to WaitReceiptMaxTime and WaitReceiptCheckInterval.

⚠️ Breaking Changes

No schema or API break, but these behavior changes happen on upgrade:

  1. A deployment that omitted StartBlock or StartL1Block now starts from about 24h back instead of genesis. Bridges older than that are never autoclaimed. Explicit values, including 0, are unaffected.
  2. A claimer that explicitly sets SafeStatusL1NumberOfBlocks = 0, FinalizedStatusL1NumberOfBlocks = 0 or EstimateGasMaxRetries = 0 gets 5, 10 or 1 instead, with no log (🔴 Unify L1 syncers #1).
  3. AggOracle's receipt timings go from 0 to 250ms and 1s, now that the keys bind.
  4. For L2-to-Lx with a non-zero StartL1Block, a source whose initial LER can't be derived is skipped with a repeated WARN instead of silently scanning full history.
  5. InternalConfigChecksum changes for existing deployments.

--- • branch fix/autoclaim-claimer-ethtxmanager-defaults

…r, key renames)

- stop defaulting Safe/Finalized/EstimateGasMaxRetries (explicit 0 is meaningful)
- treat ErrBlockNotProcessed as retry-later per source; throttle retry-later logs
- reject negative StartLookback; unexport default constants and name after Go fields
- rename GetReceipt* keys to WaitReceipt* in remaining docs/templates/tool config
- document EthTxManager defaults and restart-relative start window

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@joanestebanr joanestebanr left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@arnaubennassar
arnaubennassar merged commit c173ea4 into develop Oct 1, 2026
33 checks passed
@arnaubennassar
arnaubennassar deleted the fix/autoclaim-claimer-ethtxmanager-defaults branch October 1, 2026 15:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants