fix: repair submissions/facts ingestion + SPAC extractor robustness#181
Open
sroussey wants to merge 9 commits into
Open
fix: repair submissions/facts ingestion + SPAC extractor robustness#181sroussey wants to merge 9 commits into
sroussey wants to merge 9 commits into
Conversation
StoreSubmissionFilingsTask batched filings with `filings.slice(start, end)` on the `objectOfArraysAsArrayOfObjects` proxy. That proxy implements `map`/`filter`/iteration but not `slice`, so native `Array.prototype.slice` ran against it, saw no own indices via `HasProperty`, and returned a fully sparse array. `.map()` then skipped the holes and `saveFilingsBulk`'s `[...filings]` spread turned them into `undefined` rows, crashing putBulk with "undefined is not an object (evaluating 'entityToInsert[colStr]')" and storing zero filings. Materialize via the proxy's supported `.map()` into a concrete array first, then slice/batch that array. Adds a regression test on the storage path (previously uncovered). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BNNJyBunCkJhiUhdM9ys7V
EDGAR reports period-agnostic facts with `fy: null, fp: null` (e.g. the pay-vs-performance NetIncomeLoss figures disclosed on DEF 14A). The Factoid input schema required a number/enum, so `fetch facts` crashed on validation. `fy`/`fp` are primary-key columns and must stay NOT NULL, so make the in-memory Factoid faithful (nullable) and coalesce null -> 0 / "" sentinels at the storage boundary. The sentinels are deterministic, so the key stays stable and replays remain idempotent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BNNJyBunCkJhiUhdM9ys7V
When SEC_S1_MODEL / SEC_MERGER_PROXY_MODEL resolves to a model that isn't registered, `getS1Model()` / `getMergerProxyModel()` threw and aborted the entire filing — discarding the deterministic work that runs first (XBRL facts, issuer identity, SPAC classification/registration row, the definitive-proxy event). That contradicts the pipeline's "XBRL failures never abort the filing" contract; the redemption 8-K path already contained this correctly. Mirror that containment in the S-1, priced-424, and merger-proxy processors: catch the model-resolution failure, dead-letter the AI sections under MODEL_RESOLUTION_ERROR, and continue. The S-1 still writes the SPAC registration row (so de-SPAC 8-K milestones can attach) and the DEFM14A/DEFM14C still emits the deterministic proxy event. This lets the deterministic SPAC lifecycle (spac row + 8-K item-code milestones) ingest without a model; the AI-derived fields stay on the dead-letter worklist for retry once a model is registered. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BNNJyBunCkJhiUhdM9ys7V
A SPAC that completes its business combination keeps its CIK and becomes the operating company, which then files 8-Ks with items 1.01 / 1.02 / 5.07 for years. `deriveDeals` walked every such event and opened a fresh deal for each post-completion definitive-agreement, so e.g. Kensington Capital IV (→ Amprius) showed 3 deals — the real 2022 de-SPAC plus two phantoms from Amprius's 2023–2026 operating-company 8-Ks. A completed combination is terminal for the SPAC, so stop the event walk once a deal completes. A *terminated* attempt is not terminal (the SPAC may find another target), so only `completed` ends the walk. The fix lives in `deriveDeals` — recomputed from the full event stream on every write — so it is order-independent and repairs already-ingested data on the next rebuild. The append-only `spac_event` log is unchanged; deals are the semantic layer. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BNNJyBunCkJhiUhdM9ys7V
…ModelDefault The three extractors each hardcoded `claude-sonnet-4-6` as their default model. Replace those with a single general default, `SecModelDefault` in `src/config/Constants.ts` (value `claude-sonnet-5`, overridable via the `SEC_MODEL_DEFAULT` env var — same pattern as `SecUserAgent`). Each extractor's own env var (`SEC_S1_MODEL` / `SEC_MERGER_PROXY_MODEL` / `SEC_REDEMPTION_MODEL`) still takes precedence. Updates the default-fallback tests and CLAUDE.md accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BNNJyBunCkJhiUhdM9ys7V
The extractors resolve their model via getGlobalModelRepository().findByName(id), but nothing ever registered one — so findByName returned undefined and every AI section dead-lettered as MODEL_RESOLUTION_ERROR, even with a valid ANTHROPIC_API_KEY. registerSecModels() (called from the CLI preAction bootstrap) now registers the shared default (claude-sonnet-5, or SEC_MODEL_DEFAULT) plus any per-extractor override as Anthropic (provider: "ANTHROPIC") ModelRecords, so the id resolves. Idempotent, and an operator-registered record wins over the default. Actual generation still needs the Anthropic provider + ANTHROPIC_API_KEY; without them each section degrades to a dead-letter via the section runner's catch rather than aborting the filing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BNNJyBunCkJhiUhdM9ys7V
…ers without a version bump Two changes so the AI extraction path is actually runnable and its dead-letters drainable once the environment is fixed: - registerSecProviders() (CLI bootstrap) registers the Anthropic provider inline and the HuggingFace Transformers ONNX provider worker-backed (heavy graph in src/config/hftWorker.ts, never the main thread), so a model's `provider` discriminator resolves to an executable provider and a local model can be compared against the cloud path. Each registers defensively — a load failure warns and is skipped rather than aborting the CLI or the other provider. - MODEL_ERROR_REASON_CODES (currently MODEL_RESOLUTION_ERROR) stay retry-eligible under the *same* extractor version. A model/provider-availability failure recovers by re-running once the model is registered — a version bump neither helps nor is meaningful — so retry-dead-letters now drains them without one. Every other reason code remains version-gated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BNNJyBunCkJhiUhdM9ys7V
…tionTask resolves The hand-built ModelRecord carried capabilities: [], which trips StructuredGenerationTask's json-mode capability gate. inferAnthropicCapabilities in the installed provider doesn't recognize newer ids (claude-sonnet-5) and falls back to a bare meta-ops baseline, so declare the full claude-4-family capability set (text.generation / json-mode / tool-use / vision-input / …) explicitly on the record. Verified end to end: a real structured-generation round-trip against claude-sonnet-5 returns validated JSON. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BNNJyBunCkJhiUhdM9ys7V
Add `sec eval extract`, a harness that runs golden extraction fixtures across candidate models and ranks them on correctness, speed, and estimated cost — to find the cheapest/fastest model that still extracts correctly. - src/eval/: scoreExtraction (field-level, noise-tolerant agreement vs golden expected rows), modelPricing (char-based token + public-pricing cost estimate; local models $0), fixtures (extractor registry + golden management sections), runExtractionEval (sequential sweep, per-fixture failures don't abort). - CLI `sec eval extract [--models] [--extractor] [--format]`; default 3-way haiku / sonnet / local HFT. - Wire up a local HuggingFace Transformers (ONNX) model: secModelRecord now dispatches by id shape (org/name -> HF_TRANSFORMERS_ONNX, else ANTHROPIC), both declaring json-mode so StructuredGenerationTask resolves; SecHftModelDefault (onnx-community/Qwen2.5-0.5B-Instruct, override via SEC_HFT_MODEL) is registered alongside the cloud default. registerModelIds registers an explicit candidate list. Verified live: sonnet-5 94% / haiku-4-5 77% (100% entity recall, 1/3 the cost) / HFT 31%. Unit tests for scorer + model records; full suite 1617 pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BNNJyBunCkJhiUhdM9ys7V
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four fixes found while bootstrapping a limited dataset (submissions, company facts, and 2022 SPAC lifecycles) end-to-end through the CLI. Each is independent and has a regression test; full suite green (1605 pass, 0 fail).
1.
fix(submissions)— filings stored zero rows (crash)fetch submissionscrashed withundefined is not an object (evaluating 'entityToInsert[colStr]'), storing no filings.StoreSubmissionFilingsTaskcalledfilings.slice(...)on theobjectOfArraysAsArrayOfObjectsproxy, which implementsmap/filter/iteration but notslice. NativeArray.prototype.sliceran against it, saw no own indices viaHasProperty, and returned a fully sparse array →.map()skipped the holes →putBulk's[...]spread turned them intoundefinedrows. Fixed by materializing via the proxy's supported.map()before slicing. (Root cause also fixed upstream inlibs.)2.
fix(facts)— valid EDGAR facts rejected (crash)EDGAR reports period-agnostic facts with
fy: null, fp: null(e.g. DEF 14A pay-vs-performance figures — 26 of Amazon's 29,216). The schema required a number/enum, sofetch factsfailed validation.fy/fpare primary-key columns and must stay NOT NULL, so the in-memoryFactoidis now faithfully nullable and null coalesces to stable0/""sentinels at the storage boundary — deterministic, so replays stay idempotent.3.
fix(extractors)— unregistered AI model aborted the whole filingWhen the configured model isn't registered,
getS1Model()/getMergerProxyModel()threw and aborted the entire filing — discarding the deterministic work that runs first (XBRL facts, issuer identity, SPAC registration row, the definitive-proxy event), contrary to the pipeline's "XBRL failures never abort the filing" contract. The redemption path already contained this correctly; now the S-1, priced-424, and merger-proxy processors do too: catch the model-resolution failure, dead-letter the AI sections underMODEL_RESOLUTION_ERROR, and continue. This lets the deterministic SPAC lifecycle (spac row + 8-K item-code milestones) ingest without a model; AI-derived fields stay on the dead-letter worklist for retry.4.
fix(spac)— phantom deals after a completed combinationA SPAC that completes its combination keeps its CIK and becomes the operating company, which files 8-Ks with items 1.01/1.02/5.07 for years.
deriveDealsopened a fresh deal per post-completion definitive-agreement, so Kensington Capital IV (→ Amprius) showed 3 deals — the real 2022 de-SPAC plus two phantoms from 2023–2026 operating 8-Ks. A completed combination is now terminal for the walk (a terminated attempt is not — the SPAC may find another target). Fix lives inderiveDeals, recomputed from the full event stream on every write, so it's order-independent and repairs already-ingested data on rebuild.Verification
MODEL_RESOLUTION_ERRORdead-letters.🤖 Generated with Claude Code
https://claude.ai/code/session_01BNNJyBunCkJhiUhdM9ys7V