Add SPAC profile extraction and narrative enrichment columns#180
Merged
Conversation
Adds the embarc-facing narrative enrichment fields the app needs from S-1 filings, split across two extractor concerns: SPAC profile (new AI section): - New "Prospectus Summary" segmenter section over the summary / proposed- business prose, gated on the deterministic SPAC classification. - New spacProfile extractor + schema emitting focus (a controlled sector vocabulary), focus_location, description, team, and url_spac. - New merge-preserved nullable columns on the spac / spac_history rows (focus, focus_location, description, team, details, url_spac, url_sponsor), threaded through recordRegistration under the existing as_of guard. url_sponsor + details are editorial/manual (columns only). - S-1 extractor bumped 1.3.0 -> 1.4.0. Leadership (management section): - ManagementOutputSchema gains age + bio; birth_year is derived from the stated age relative to the filing year (stable across time) and, with the bio, persisted on person_observations. Additive nullable columns are created from the TypeBox schema on fresh DBs and via a guarded ALTER TABLE migration on existing SQLite DBs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QxvaRk34eJVqyhK1Y3CVYG
Extends the merger-proxy extractor to capture a short target-company business description alongside target_name / pipe_amount. It is correlated onto the matching spac_deal by filing-date window (like target_name) and derived onto the consolidated spac row from the active deal, so a dead deal's description can't shadow the live one. - MergerDealOutputSchema gains an optional target_description (optional so a pre-1.1.0 replay still validates); merger-proxy extractor bumped -> 1.1.0. - New nullable target_description columns on spac_merger_extraction, spac_deal, spac, spac_history; tracked for history/ChangeLog; guarded ALTER migrations. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QxvaRk34eJVqyhK1Y3CVYG
Editorial / infrastructure additions for the remaining embarc fields: - investorpres_url / investorpres_date columns on the spac + spac_history rows, derived from the latest `investor_presentation` event's source_document_url (new SpacEvent type). Population is deferred to a dedicated 8-K Item 7.01 EX-99 exhibit extractor — the 8-K path today carries only item codes, no exhibit access — so the slot derives null until then. - portals.featured (nullable boolean): editorial curation flag, not SEC-sourced. - family_description side table (family_kind, normalized_name) with a repo and `sec canonical <sponsor|underwriter>-family describe/description` CLI. Kept version-independent (keyed by normalized name, not the resolver-minted family id) so a resolver dropPrevious never wipes the curated text. Additive nullable columns + the new table are created from schema on fresh DBs and via guarded ALTER TABLE / setupDatabase on existing ones. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QxvaRk34eJVqyhK1Y3CVYG
Correctness: - focus / focus_location: empty extracted arrays now serialize to null (not "[]"), mirroring spac_tickers, so a later filing that restates no tags can't clobber previously-extracted focus under the rollup's non-null-wins merge. - Portal.featured is now merge-preserved across CFPORTAL amendments/withdrawals (the savePortalAsOf build callback omitted it, clobbering the editorial flag). - birthYearFromAge clamps its result to the schema's [1900, 2100] range so a very old age on an early filing can't produce an out-of-range birth_year. Cleanup: - Hoist the shared registration args (baseReg) in processFormS1; the parse- failure and profile paths no longer duplicate the 7-field base object. - Dedup the migration column list (spac/spac_history share one const) and note the SQLite-only scope is consistent with the existing migration convention. - Extract the near-identical family describe/description CLI blocks into a shared registerFamilyDescribeCommands helper. - Remove the unused FocusTag export; run prettier over the changed files. Tests: empty-focus→null, and CFPORTAL featured preservation across amendment. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QxvaRk34eJVqyhK1Y3CVYG
There is no existing populated database to migrate — fresh SQLite/Postgres DBs create the new columns straight from the TypeBox schema via setupDatabase(), and tests run on the in-memory backend. The one-shot ALTER-TABLE migration was dead weight (and never ran in tests), so remove it. This also moots the SQLite-only vs Postgres migration asymmetry entirely. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QxvaRk34eJVqyhK1Y3CVYG
`featured` is editorial curation, not SEC data — it does not belong in the extracted dataset. embarc will source featured portals from its own JSON file instead. Removes the column from PortalSchema and the CFPORTAL merge-preserve. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QxvaRk34eJVqyhK1Y3CVYG
Contributor
There was a problem hiding this comment.
Pull request overview
This PR expands the SPAC ingestion/reporting pipeline to extract and persist AI-derived “business profile” narrative fields (sector focus, geography, description, team, website) from S‑1 prospectus summaries, and to enrich the SPAC report with additional narrative/event-derived columns (e.g., target description, investor presentation link/date). It also adds leadership bio enrichment (age→birth year, bio) and introduces a version-independent repository for editorial family descriptions.
Changes:
- Add SPAC narrative/enrichment columns across SPAC schema, rollup logic, and report history tracking (including investor presentation URL/date derived from events).
- Add S‑1 “Prospectus Summary” segmentation + new structured extractor (
extractSpacProfile) with end-to-end coverage. - Add leadership enrichment fields (
birth_year,bio) for management observations, and add an editorialFamilyDescriptionRepoplus CLI wiring.
Reviewed changes
Copilot reviewed 36 out of 36 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/storage/spac/SpacSchema.ts | Adds narrative/enrichment columns to the canonical spac row schema. |
| src/storage/spac/spacRollup.ts | Rollup now merges narrative patch fields and derives investorpres_* + target_description. |
| src/storage/spac/spacRollup.test.ts | Unit tests for narrative merge behavior and investor-presentation/target-description derivations. |
| src/storage/spac/SpacReportWriter.ts | Tracks and snapshots new narrative/enrichment fields; passes profile fields through registration writes. |
| src/storage/spac/SpacMergerExtractionSchema.ts | Extends merger extraction storage schema with target_description. |
| src/storage/spac/SpacHistorySchema.ts | Mirrors new narrative/enrichment columns into history snapshots. |
| src/storage/spac/SpacEventSchema.ts | Adds investor_presentation event type and clarifies exhibit URL usage. |
| src/storage/spac/SpacDealSchema.ts | Adds target_description to per-deal storage. |
| src/storage/spac/spacDealGrouping.ts | Correlates merger extraction target_description onto derived deals. |
| src/storage/spac/spacDealGrouping.test.ts | Tests deal correlation behavior for target_description. |
| src/storage/observation/PersonObservationSchema.ts | Adds birth_year and bio fields to person observations. |
| src/storage/observation/PersonObservationSchema.test.ts | Updates schema tests to include new person enrichment fields. |
| src/storage/observation/PersonObservationRepo.ts | Persists the new observation columns on write. |
| src/storage/canonical/FamilyDescriptionSchema.ts | New schema for version-independent editorial family descriptions. |
| src/storage/canonical/FamilyDescriptionRepo.ts | New repo to get/set/list/remove editorial family descriptions. |
| src/storage/canonical/FamilyDescriptionRepo.test.ts | Tests for editorial family description round-tripping and deletion. |
| src/sec/html/parseEdgarHtml.golden.test.ts | Updates expected S‑1 sections to include PROSPECTUS_SUMMARY. |
| src/sec/forms/registration-statements/s1/spacProfileSchema.ts | New controlled vocabulary + output schema for SPAC profile extraction. |
| src/sec/forms/registration-statements/s1/spacProfile.e2e.test.ts | End-to-end tests covering SPAC profile extraction + dead-lettering behavior. |
| src/sec/forms/registration-statements/s1/sectionSchemas.ts | Extends management section schema to include age + bio. |
| src/sec/forms/registration-statements/s1/sectionExtractors.ts | Adds extractSpacProfile; extends management + merger deal extraction instructions. |
| src/sec/forms/registration-statements/s1/mergerDealSchema.ts | Adds optional target_description field for merger-deal structured output. |
| src/sec/forms/registration-statements/s1/leadershipBio.e2e.test.ts | End-to-end test for birth-year derivation + bio persistence. |
| src/sec/forms/registration-statements/s1/DocumentTreeSegmenter.test.ts | Tests segmentation for the new prospectus summary section. |
| src/sec/forms/registration-statements/s1/DocumentSegmenter.ts | Adds PROSPECTUS_SUMMARY and heading patterns for segmenting summary/business prose. |
| src/sec/forms/registration-statements/Form_S_1.storage.ts | Bumps extractor version; adds birthYearFromAge; runs SPAC profile extraction and merges into registration write; ensures base row is created on parse failure. |
| src/sec/forms/proxies-information-statements/Form_DEFM14A.storage.ts | Bumps extractor version; persists target_description from merger proxy extraction. |
| src/sec/forms/proxies-information-statements/Form_DEFM14A.storage.e2e.test.ts | E2E coverage for merger-proxy target_description persistence. |
| src/resolver/EntityObserver.ts | Plumbs birth_year and bio through person claims into observations. |
| src/config/TestingDI.ts | Registers in-memory storage for the new family description repo and updates SPAC DI wiring. |
| src/config/setupAllDatabases.ts | Ensures family_description storage is setup as part of global DB init. |
| src/config/resetAllDatabases.ts | Ensures family_description is cleared during DB resets. |
| src/config/DefaultDI.ts | Registers the production storage binding for family_description. |
| src/commands/underwriterFamily.ts | Wires editorial describe/description subcommands for underwriter families. |
| src/commands/sponsorFamily.ts | Wires editorial describe/description subcommands for sponsor families. |
| src/commands/familyDescribe.ts | New shared CLI subcommand implementation for setting/getting family descriptions. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+74
to
+78
| // Narrative / enrichment (embarc-facing). Merge-preserved filing-sourced | ||
| // scalars under the same `as_of` guard as the other scalars; `focus` / | ||
| // `focus_location` / `details` hold JSON-encoded strings (mirroring the | ||
| // `spac_tickers` string[] pattern). `url_sponsor` has no reliable SEC source | ||
| // and is editorial/manual (column only). `target_description` (merger-proxy) |
Comment on lines
+65
to
+78
| function latestInvestorPres(events: readonly SpacEvent[]): { | ||
| url: string | null; | ||
| date: string | null; | ||
| } { | ||
| const pres = events | ||
| .filter((e) => e.event_type === "investor_presentation") | ||
| .sort( | ||
| (a, b) => | ||
| a.event_date.localeCompare(b.event_date) || | ||
| a.accession_number.localeCompare(b.accession_number) | ||
| ); | ||
| const latest = pres.length > 0 ? pres[pres.length - 1] : null; | ||
| return { url: latest?.source_document_url ?? null, date: latest?.event_date ?? null }; | ||
| } |
Addresses two Copilot review comments on #180: - SpacSchema: the narrative-columns comment lumped `details` in with the string[] JSON pattern; clarify that focus/focus_location are JSON string[] while `details` is a JSON key/value object. - spacRollup.latestInvestorPres: replace the sort-then-take-last with a single-pass max by (event_date, accession_number) — same result, no O(n log n). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QxvaRk34eJVqyhK1Y3CVYG
There is no persisted database yet, so the version-bump ceremony — which only exists to make old dead-letters retry-eligible after a prompt/schema change — is moot, and the runtime version is bootstrap-seeded to 1.0.0 regardless of the fallback constant. Revert the S-1 (was bumped to 1.4.0) and merger-proxy (1.1.0) DEFAULT_EXTRACTOR_VERSION to 1.0.0; the prompt/schema changes stand on their own and can be versioned once a populated DB exists. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QxvaRk34eJVqyhK1Y3CVYG
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds AI-powered extraction of SPAC business profiles from prospectus summaries and introduces narrative enrichment columns to the SPAC report. The changes enable the embarc UI to display curated SPAC sector focus, geographic focus, descriptions, team information, and websites alongside deal data.
Key Changes
SPAC Profile Extraction: New
extractSpacProfile()function parses the "Prospectus Summary" section to extract:focus: controlled-vocabulary business sector tags (FinTech, Healthcare, etc.)focus_location: geographic focus areas (free-form prose)description: narrative business purposeteam: management team blurburl_spac: company websiteNarrative Enrichment Columns: Added to
spactable:focus,focus_location,description,team,url_spac(filing-sourced, merge-preserved)target_description(derived from active merger deal)details,url_sponsor(editorial-only, manual curation)investorpres_url,investorpres_date(derived from investor-presentation events)Leadership Bio Enrichment: Extended management extraction to capture:
age: person's stated age (integer or null)bio: short biography summarizing background/experiencebirthYearFromAge()utility derives stable birth year from age + filing dateFamily Description Storage: New
FamilyDescriptionRepoand schema for editorial descriptions of sponsor/underwriter families, decoupled from resolver versions to survive version bumpsS-1 Extractor Version Bump: v1.3.0 → v1.4.0 to reflect profile section addition
Prospectus Summary Section: Added
PROSPECTUS_SUMMARYtoS1_SECTIONSandDocumentTreeSegmenterto recognize and segment the opening business summary proseSPAC Row Creation on Parse Failure: Base SPAC row (registration event + name/SIC) is now created even when HTML parsing fails, ensuring SPACs are tracked even if profile extraction cannot run
Merge-Preserving Rollup Logic:
buildSpacRow()merges narrative fields from patches without clobbering existing values; stale filings can fill null slots but never overwrite set valuesEnd-to-End Tests:
spacProfile.e2e.test.ts: validates profile extraction, dead-lettering on missing summary, and non-SPAC filteringleadershipBio.e2e.test.ts: validates age→birth_year conversion and bio persistenceDatabase Migration:
migrateSpacNarrativeColumns()adds missing columns to existing SQLite databases on setupCLI Commands: Added
sponsor-family describeandunderwriter-family describecommands to set editorial family descriptionsImplementation Details
extractStructured()ceremony as other sections, with dead-lettering onSECTION_NOT_FOUNDwhen the prospectus summary is absentfocusvocabulary is a controlled enum (87 tags) to keep the field queryable in embarcfocus,focus_location,details) to match existingspac_tickerspatternhttps://claude.ai/code/session_01QxvaRk34eJVqyhK1Y3CVYG