Skip to content

fix(chainlink): incremental WHERE must be OR, not AND, across a FULL … - #10048

Open
burugupallyprem-coder wants to merge 2 commits into
duneanalytics:mainfrom
burugupallyprem-coder:fix/chainlink-incremental-or
Open

burugupallyprem-coder wants to merge 2 commits into
duneanalytics:mainfrom
burugupallyprem-coder:fix/chainlink-incremental-or

Conversation

@burugupallyprem-coder

@burugupallyprem-coder burugupallyprem-coder commented Sep 28, 2026 •

Copy link
Copy Markdown

The bug

14 of the chainlink_<chain>_*_request_daily models filter both sides of a FULL OUTER JOIN in the incremental WHERE, joined with AND:

FROM {{ ref('chainlink_arbitrum_fm_fulfilled_transactions') }} fulfilled
FULL OUTER JOIN {{ ref('chainlink_arbitrum_fm_reverted_transactions') }} reverted ON
    reverted.block_time = fulfilled.block_time AND
    reverted.node_address = fulfilled.node_address
{% if is_incremental() %}
  WHERE
    {{ incremental_predicate('fulfilled.block_time') }}
    AND {{ incremental_predicate('reverted.block_time') }}
{% endif %}

incremental_predicate(col) expands to col >= <cutoff>. A node-day with no reverted transaction has reverted.block_time = NULL, NULL >= cutoff is NULL, and the row is dropped. So on every incremental run the FULL OUTER JOIN degrades to an INNER JOIN, and the only rows that survive are node/timestamp pairs that have both a fulfilled and a reverted transaction at the identical block_time.

That coincidence is rare. With incremental_strategy='merge' on (date_start, node_address), nothing is deleted — the table simply stops receiving new rows and continues to look healthy.

The models' own output columns say unmatched rows were expected:

COALESCE(fulfilled.node_address, reverted.node_address) AS node_address

The fix

AND → OR, in 14 files. With OR, a fulfilled-only row passes on the first disjunct (TRUE OR NULL = TRUE) and a stale pair still fails (FALSE OR FALSE = FALSE).

This is not a new idea — 13 models in the same family already do it this way. Every ocr_request_daily and automation_request_daily, on every chain, uses OR. Normalised for the product name, chainlink_arbitrum_fm_request_daily.sql and chainlink_arbitrum_ocr_request_daily.sql are identical files apart from the alias, a post_hook, and this one word. This PR makes the 14 match the 13.

Measured on your production tables

Queried on Dune, 28 September 2026:

model fulfilled input latest reverted input latest output latest fulfilled rows arriving after the output froze
fm_request_daily — AND 2025-12-08 2026-09-28 2024-06-04 17,257
vrf_request_daily — AND 2026-09-27 2026-09-28 2026-07-01 172
ocr_request_daily — OR 2026-09-22 2026-09-28 2026-09-28 0

Scope and what this PR does not fix

14 files, one word each: fm_request_daily and vrf_request_daily on arbitrum, avalanche_c, bnb, ethereum, fantom, gnosis, optimism and polygon.

One thing deliberately left alone: the join is on (block_time, node_address), which is not unique in either input — both upstream models declare unique_key = (tx_hash, tx_index, node_address), one row per transaction. A node with F fulfilled and R reverted transactions in the same block produces F×R rows and both counts report F×R. That is present on full refresh too and is a separate change; I have not measured how often it fires and did not want to mix it into a one-word fix. Happy to open it as its own issue.

How I found it

I maintain a static analyser for grain and join defects in analytical SQL, and spellbook is one of the repositories I run it against. It flagged this shape, and I then checked it by hand against your own OR models and by running the compiled CTE. Everything above is verifiable from the repo and from public Dune tables.

…OUTER JOIN

A node-day with no reverted transaction has reverted.block_time = NULL, so NULL >= cutoff is NULL and the row is dropped. Every incremental run degrades the FULL OUTER JOIN to an INNER JOIN. With incremental_strategy='merge' those node-days are never written.

The other 13 models of the same family -- ocr_request_daily and automation_request_daily on every chain -- already use OR. This makes the 14 match the 13.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@cursor

cursor Bot commented Sep 28, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

@github-actions
github-actions Bot marked this pull request as draft September 28, 2026 04:23
@github-actions github-actions Bot added WIP work in progress dbt: daily covers the Daily dbt subproject labels Sep 28, 2026
@burugupallyprem-coder

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

github-actions Bot added a commit that referenced this pull request Sep 28, 2026
@burugupallyprem-coder
burugupallyprem-coder marked this pull request as ready for review September 28, 2026 04:44
@github-actions github-actions Bot added ready-for-review this PR development is complete, please review and removed WIP work in progress labels Sep 28, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dbt: daily covers the Daily dbt subproject ready-for-review this PR development is complete, please review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant