Skip to content

fix: recover forwarded Claude skills and scope session search - #248

Open
jmagar wants to merge 1 commit into
mainfrom
codex/post-deploy-transcript-search-20260918
Open

jmagar wants to merge 1 commit into
mainfrom
codex/post-deploy-transcript-search-20260918

Conversation

@jmagar

@jmagar jmagar commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator

Why

Production verification after #247 exposed two gaps that CI did not model:

  1. Forwarded Claude history is privacy-scrubbed. ai_transcript_path is a SHA-256 locator and raw is empty, so historical skill backfill cannot reopen the original JSONL. Modern package-qualified command envelopes still survive in normalized user message content, but the backfill previously ignored that evidence.
  2. The transcript-only FTS from migration 60 still evaluated common terms globally before relational tool / since filters. On production, search_sessions(query="error", tool="claude", since=2026-09-11, limit=5) still exceeded the gateway timeout at ~59s.

Fix

Forwarded Claude command recovery

  • Recover package-qualified <command-message> / <command-name> evidence directly from persisted normalized Claude user messages.
  • Require persisted metadata_json.event_kind == "user" before trusting forwarded command content.
  • Preserve the existing source-file fallback for structured attributionSkill / attributionPlugin evidence.
  • Preserve source_unavailable accounting when neither normalized command evidence nor the original source is recoverable.
  • Add a production-shaped regression with scrubbed metadata, SHA-256 transcript locator, empty raw payload, and vibin:review-pr command content.

Scoped session search

  • Add migration 61.
  • Rebuild ai_logs_fts with provider scope (message, ai_tool).
  • Add partial AI-only idx_logs_ai_timestamp_tool(timestamp, ai_tool) WHERE ai_tool IS NOT NULL.
  • Scope the FTS expression to message and, for safe provider identifiers, ai_tool.
  • For since, derive an exact-safe minimum rowid from the AI-only timestamp index and push that bound into FTS before common-term expansion.
  • Retain the existing relational l.ai_tool = ? and timestamp filters as correctness guards.
  • Do not interpolate unsafe provider values into FTS; they fall back to the parameterized relational equality filter.

The rowid floor is derived from actual rows satisfying the event-time bound, not inferred from ingestion order, so late-arriving historical transcripts remain discoverable.

Production evidence before this PR

  • Cortex fix: repair transcript skill evidence and search #247 deployed cleanly as 03ba04df.
  • Migration 60 completed on the ~160 GB production DB in 146.0s; service returned healthy with zero restarts.
  • Modern Claude skill_events since 2026-09-01: 0.
  • Production forwarded Claude rows preserve command envelopes in message but have scrubbed source/raw data.
  • Representative scoped search_sessions call: ~58.996s timeout.
  • Live query plan showed FTS driving globally, followed by rowid lookups and relational filtering.

Validation

TDD:

  • New forwarded/scrubbed Claude regression failed first with inserted=0, source_unavailable=1, then passed after the recovery path.
  • New migration 61 and scoped-session-search regressions failed before implementation, then passed.

Targeted matrix:

  • all skill_backfill tests
  • all search_ai_sessions tests
  • migration 51 damaged-schema recovery
  • schema-43 historical upgrade fixture
  • migration 61 provider-scoped FTS/index regression
  • schema/docs parity
  • unsafe tool-scope interpolation guard

Full isolated gate:

CORTEX_FULL_PRE_PUSH=1 cargo xtask pre-push

Main library suite:

3030 passed
0 failed
3 ignored

All integration/workflow suites also passed.

Deployment verification after merge

After hosted CI is green and this PR is merged:

  1. build/deploy the exact merge SHA to TOOTIE with the current image rollback preserved;
  2. observe migration 61 to healthy completion;
  3. run the production skill backfill and verify modern Claude claude_skill_command events become nonzero;
  4. rerun the exact pre-deploy search_sessions query and record before/after latency + result correctness;
  5. update U8-929 with landed production evidence.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant