fix(polling_source): eliminate per-batch schema inference, fix zero-row crash (ENG-952) - #261
Merged
eywalker merged 10 commits intoAug 28, 2026
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes an ENG-952 regression in PollingSource where per-batch Arrow nullability inference could crash the polling loop when a zero-row batch followed a batch containing nulls. The change makes schema/nullability a per-source invariant by establishing a canonical Arrow schema once and applying it consistently to subsequent batches.
Changes:
- Establish
_canonical_arrow_schemaonce perPollingSourceinstance (declared-schema path usesimpl.schema(); otherwise infer once from the first non-empty batch) and apply canonical nullability to every batch via name-based casting. - Skip zero-row frames before canonical schema establishment on the infer-once path to avoid locking in an all-non-nullable schema.
- Add 5 async regression tests covering the crash scenario, warning behavior, and empty-first-batch behavior; document the resolved design issue (PS4) and add supporting spec/plan artifacts.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
src/orcapod/core/sources/polling_source.py |
Adds per-source canonical Arrow schema handling and avoids zero-row nullability re-inference that caused SchemaInconsistencyError. |
tests/test_channels/test_polling_source.py |
Adds ENG-952 regression tests validating stability across zero-row batches and correct warning behavior for schema inference. |
DESIGN_ISSUES.md |
Logs PS4 as resolved with a concise description of the root cause and fix. |
superpowers/specs/2026-08-27-eng-952-polling-source-zero-row-batch-fix.md |
Captures the design rationale and intended behavior for the canonical-schema approach. |
superpowers/plans/2026-08-27-eng-952-polling-source-zero-row-batch-fix.md |
Provides an implementation/test plan matching the applied fix and added coverage. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Contributor
Author
|
Thanks for the review! Copilot raised no inline comments and found no issues across all 5 changed files. No code changes are needed in response to this review round — the PR is ready as-is. |
…g from inline test impls Follow the established inline-class pattern used throughout the file: test impl classes define only identity, schema, poll, fetch, and close. Also add a descriptive failure message to the _accumulated_stream assertion so failures self-describe rather than raising an AttributeError on the next line.
…ng zero-row crash
…t non-empty batch
…ove stale type: ignore
…ed for _batches API ITL-617 replaced _accumulated_stream with an append-only _batches list. Zero-row batches produce empty ArrowTableStream entries in _batches, so the assertion must count total rows across all batches rather than batch-list length. Also rename DESIGN_ISSUES PS4 entry to PS5 to avoid collision with the ITL-617 PS4 entry that was added to main.
kurodo3
Bot
force-pushed
the
eywalker/eng-952-fix-upstream-pollingsource-dies-when-a-poll-returns-a-zero
branch
from
August 28, 2026 00:37
42db3c0 to
1eb1d43
Compare
eywalker
approved these changes
Aug 28, 2026
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Contributor
Author
|
Thanks for the approval, @eywalker! No changes needed — the PR is ready to merge whenever you'd like. |
eywalker
deleted the
eywalker/eng-952-fix-upstream-pollingsource-dies-when-a-poll-returns-a-zero
branch
August 28, 2026 02:00
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_build_stream_from_dfcalledinfer_schema_nullableon every batch. A zero-row batch hasnull_count == 0for all columns, so every field was inferrednullable=False._validate_combining_schemasthen raisedSchemaInconsistencyErroragainst the accumulated stream's nullable schema — killing the source on the first empty poll._canonical_arrow_schemaexactly once per source, never re-infer per batch.impl.schema()returns a non-NoneSchema, derive the Arrow schema from Python type annotations (T | None→nullable=True). No inference, no warning.impl.schema()returnsNone, infer from the first non-empty batch and emit aWARNINGprompting the caller to declare a schema. Zero-row frames before schema establishment returnNone(skipped) so they cannot lock in a spurious all-non-nullable schema.TestPollingSourceZeroRowBatch.DESIGN_ISSUES.md.Fixes ENG-952
Test plan
uv run pytest tests/test_channels/test_polling_source.py::TestPollingSourceZeroRowBatch -v— 5 new tests, all passuv run pytest tests/test_channels/test_polling_source.py -v— 66 tests, all passuv run pytest tests/ -x -q— 4774 passed, 93 skipped, 2 xfailed, 0 failures🤖 Generated with Claude Code