Add typecheck budget guard to catch type-instantiation regressions#563
Closed
sroussey wants to merge 4 commits into
Closed
Add typecheck budget guard to catch type-instantiation regressions#563sroussey wants to merge 4 commits into
sroussey wants to merge 4 commits into
Conversation
…ons (#555) * ci: add typecheck instantiation-budget guard + perf investigation writeup Adds scripts/typecheck-budget.ts, which type-checks each composite package in isolation and gates its (deterministic, machine-independent) instantiation count against committed per-package budgets in scripts/typecheck-budget.json. Wired into CI as the `typecheck-budget` job and exposed as `bun run typecheck:budget`. Also documents the typecheck slowdown investigation (timeline, bisected culprit commits, root-cause analysis, and prototyped fixes) in docs/technical/typecheck-performance-investigation.md. https://claude.ai/code/session_018SckHLTLmkxKCSWyb74Lke * perf(ai): replace FromSchema-derived types with concrete interfaces (AI Text slice) Proof-slice for the typecheck-cost work: the dominant `packages/ai` check cost is the `FromSchema<typeof Schema>` conditional-type machinery (json-schema-to-ts), not the Workflow/CreateWorkflow generics. Converting the plain `type X = FromSchema<typeof Schema>` aliases to concrete types removes that per-use instantiation cost. This slice converts the AI Text task family + the foundational model types (ModelConfig/ModelRecord). Each concrete type is the type checker's exact resolution of the original FromSchema type (verified by a clean full `tsc -b`), so it is equivalent to the prior derived type; the schema constants remain the runtime contract. Budget baseline ratcheted to match. Measured (pinned tsc, deps prebuilt, isolated `tsc -p`): - ai: 564,545 -> 520,877 instantiations (this 7-file slice) - full-ai conversion projects to ~320k instantiations / -39% check time, with a ~-14% ripple on packages/test (tracked in #556) Tradeoff: a hot-path `Equal<X, FromSchema<...>>` drift guard is intentionally omitted because it would recompute FromSchema and defeat the win; the schemas stay the runtime source of truth. Refs #556. https://claude.ai/code/session_018SckHLTLmkxKCSWyb74Lke * perf(ai): convert remaining FromSchema-derived task types to concrete types Completes the FromSchema->concrete migration started in the AI Text slice. Converts the remaining plain and composite (Omit<...> / WithImageValuePorts<...>) type aliases across packages/ai to concrete types, removing the json-schema-to-ts conditional-type instantiation cost from the AI task input/output types. Each concrete type is the type checker's exact resolution of the original FromSchema type (verified by a clean full `tsc -b`); the schema constants remain the runtime contract. ModelConfig/ModelRecord and ChatMessage are referenced by name rather than inlined for readability. Four vector/embedding outputs that involve branded typed-array types (TextEmbedding, ImageEmbedding, VectorSimilarity) are intentionally left on FromSchema since they do not print to a clean concrete form. Measured (pinned tsc, deps prebuilt, isolated `tsc -p`): - ai: 564,545 -> 92,750 instantiations (-84%), 4.29s -> ~2.0s check (-52%) - test (ripple): 1,427,138 -> 947,215 instantiations (-34%), 12.4s -> ~9.2s Budget baselines ratcheted to match. Closes #556. https://claude.ai/code/session_018SckHLTLmkxKCSWyb74Lke --------- Co-authored-by: Claude <noreply@anthropic.com>
… new constructor signature After commit c011234 removed the legacy `{ tabularRepository: ... }` option shim, copy-paste from these README examples produced a TS error against the new `{ storage: ITaskOutputStorage }` constructor signature. Replaced the eight stale `tabularRepository:` constructor calls across two READMEs with `storage: tabularTaskOutputStorage(...)` and pulled the helper into each example's import block.
* ci: authenticate and cache Hugging Face model downloads CI test jobs were downloading ONNX/GGUF models anonymously from the Hugging Face Hub on every run, which hits anonymous IP rate limits on shared GitHub runners (downloads denied). - Pass WORKGLOW_SECRETS_PASSPHRASE to the model-downloading vitest jobs (hft, nodellama; rag already had it) so the test preload decrypts the on-disk credential store and hydrates HF_TOKEN into process.env. transformers.js authenticates Hub downloads when HF_TOKEN is set, raising the rate limit against the account. - Cache the repo-root ./models directory (where the test cache dir is configured) per job so models are not re-fetched on every run. * ci: add encrypted hf-token credential --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a CI guard (
scripts/typecheck-budget.ts) that gates per-package TypeScript instantiation counts to catch type-level performance regressions at PR time. Includes investigation documentation of two recent slowdowns and refactors AI task types to inlineFromSchemacalls, reducing instantiation pressure.Key Changes
New typecheck budget guard (
scripts/typecheck-budget.ts):tsc --extendedDiagnosticsscripts/typecheck-budget.json--updateto re-baseline,--jsonto emit measurements,--toleranceto adjust headroomtypecheck-budgetjob in.github/workflows/test.ymlInvestigation documentation (
docs/technical/typecheck-performance-investigation.md):16a94b54(MCPallOfspreading, already fixed) and888da0e1(durableairegression from task-runner refactor)aicostAI task type refactoring (multiple files in
packages/ai/src/task/):FromSchema<typeof SomeSchema>with inlined type definitionsAiChatTask,AiChatWithKbTask,PoseLandmarkerTask,FaceLandmarkerTask,GestureRecognizerTask,HandLandmarkerTask,ObjectDetectionTask,FaceDetectorTask,ImageClassificationTask,ImageSegmentationTask,ImageToTextTask,ImageEmbeddingTask,TextEmbeddingTask,ToolCallingTask,HierarchyJoinTask,ChunkRetrievalTask,HierarchicalChunkerTask,ContextBuilderTask,TextChunkerTask,ChunkVectorUpsertTask, and othersFromSchemaimports and helper types (WithImageValuePorts,TypedArraySchemaOptions) where no longer neededModelSchema refactoring (
packages/ai/src/model/ModelSchema.ts):ModelConfigandModelRecordtype definitions instead of deriving fromFromSchemaDocumentation updates:
packages/task-graph/README.mdandpackages/task-graph/src/storage/README.mdto show newtabularTaskOutputStorage()wrapper in examplestabularTaskOutputStorageimport to code samplesCI improvements:
typecheck:budgetnpm script inpackage.jsonImplementation Details
The budget guard uses
tsc -bto warm-build all packages (emitting.d.tsfor dependency graphs), then measures each package in isolation withtsc -p <pkg> --extendedDiagnosticsafter removing itstsbuildinfoto force a full re-check. Instantiations is the headline metric—it is deterministic and machine-independent, unlike wall-clock time, making the gate stable across CI runners while still catching regressions like thetasksspike (289k → 6.77M) or theaidoubling (0.9s → 3.4s check time).The AI task type inlining is a targeted mitigation for the
888da0e1regression. While the root cause is diffuse in core generics and no surgical fix was found, inlining avoids repeated instantiation of the same schema-to-type inference at multiple task sites, reducing overall pressure on the type checker.https://claude.ai/code/session_01VgNMXhHCKrbY1Eaa4rfRe8