feat(semantic): [alpha build] provider-aware typed embeddings, reranking, diagnostics, and eval harness

[Zireael]

Summary

Semantic search in AFT moves from a minimal embedding-and-cosine prototype to a provider-capability-aware retrieval subsystem with typed vectors, optional reranking, background lifecycle management, diagnostics, and evaluation tooling. This is a public preview — the feature is functional and tested (~93 new tests) but expects iteration based on real-world feedback.

What changed

The upgrade touches the full semantic pipeline — config, indexing, retrieval, diagnostics, and observability — without breaking the default fastembed experience.

Typed vector representations

Vectors are no longer opaque f32 blobs. Every stored vector carries explicit type metadata (DenseF32, Int8SourceDecoded, BinaryPacked) and is paired with its source kind so the correct distance metric is selected automatically. Binary packed vectors use Hamming search (native bitwise XOR + popcount) instead of cosine, which is both faster and semantically correct for quantized embeddings. This unlocks Perplexity's base64_binary and base64_int8 output modes alongside standard dense providers.

Provider capability profiles

Each embedding backend (fastembed, OpenAI-compatible, Ollama, Perplexity) declares what it supports: output encoding, distance metric, dimension range, max batch size. The config layer validates combinations at configure time — you cannot accidentally request binary vectors through a cosine-only provider. Profiles also carry fingerprint fields so switching providers triggers a clean index rebuild rather than silent corruption.

Fingerprint-driven index lifecycle

A SemanticIndexFingerprint captures every dimension that affects index correctness: backend, model, base_url, dimension, chunking_version, output_encoding, storage_strategy, vector kinds, normalization, and prompt hashes. diff() classifies changes as Rebuild (structural — re-embed everything), ClearQueryCache (query prompts changed — invalidate cached results only), or None. This replaces the previous "delete and hope" invalidation with precise, explainable rebuild decisions.

Non-blocking cold start

Index builds run in a background thread with cooperative cancellation (SemanticCancellationToken via AtomicU64 generation counter). The build checks the generation before each embedding batch and exits early when a reconfigure arrives. Priority ordering ensures high-value files (recently edited, high PageRank) get embedded first. Exponential backoff handles transient provider failures without blocking the session.

Stale-vector pruning

When files are edited, deleted, moved, excluded, or re-included, the index tracks which vectors are stale and prunes them during the next refresh cycle. Every vector record carries file/chunk ownership metadata (file path, version, chunk hash, index fingerprint) so pruning is traceable and deterministic.

File policy and docs chunking

A configurable file policy controls which files enter the index (include globs, exclude globs, max file size, max chunk count). The docs chunker splits Markdown and documentation files into semantic sections before embedding, improving recall for documentation-shaped queries.

Reranking pipeline

Optional reranking via any OpenAI-compatible /v1/rerank or chat-completion endpoint. The pipeline sends initial retrieval candidates to a reranker, parses the response (supporting multiple JSON shapes), and reorders results with safe fallback — if the reranker fails, the original cosine-similarity order is returned unchanged. Config fields: rerank.enabled, rerank.model, rerank.base_url, rerank.api_key_env, rerank.max_candidates.

Search pipeline metrics and diagnostics

Every aft_search call records timing, cache hits/misses, result counts, and reranker fallback events. Metrics are exposed through the status command and through JSONL diagnostic logs for offline analysis. The DiagnosticsOutputMode config controls verbosity in tool output (compact | verbose | off).

Semantic doctor

semantic_doctor is a health-check command that reports config summary, index summary, metrics summary, provider summary, and actionable suggestions. Use it to verify that the index is healthy, the provider is reachable, and the configuration is consistent.

Semantic eval harness

semantic_eval runs a JSONL-defined evaluation suite against the semantic index. Each case specifies a query, expected paths, expected symbols, and top-k. The harness computes recall@k and MRR (Mean Reciprocal Rank) for quantifying retrieval quality across config changes.

Status integration

The status command now includes semantic health metrics: lifecycle state, entry count, dimension, total queries, cache hit ratio, average query time, and provider info. The OpenCode TUI sidebar surfaces these alongside the existing index state.

Config trust boundary

backend, base_url, and api_key_env are user-only fields — project-level aft.jsonc cannot inject these. A hostile repository cannot redirect embeddings at an attacker-controlled endpoint or exfiltrate API keys. The plugin logs a warning when it strips a project-level setting.

Contextualized document-chunk embedding (partial)

Initial support for Perplexity-style document/chunk grouped embedding — chunks from the same source document are batched together rather than flattened. Oversized document handling and retry logic are still in progress (see roadmap).

How to test

Default fastembed (zero-config)

# Enable semantic search in your AFT config
# ~/.config/opencode/aft.jsonc or ~/.pi/agent/aft.jsonc:
{ "semantic_search": true }

# Start a session — index builds in background
# Run aft_search with a concept query:
aft_search({ "query": "authentication middleware" })

Verify: results appear with source: semantic or source: hybrid tags. Status shows [index: ready] after build completes.

Provider switching

// Switch to OpenAI-compatible
{
  "semantic_search": true,
  "semantic": {
    "backend": "openai_compatible",
    "model": "text-embedding-3-small",
    "base_url": "https://api.openai.com/v1",
    "api_key_env": "OPENAI_API_KEY"
  }
}

Verify: index rebuilds automatically on next session start. Status shows new provider/model.

Reranking

{
  "semantic_search": true,
  "semantic": {
    "backend": "openai_compatible",
    "model": "text-embedding-3-small",
    "base_url": "https://api.openai.com/v1",
    "api_key_env": "OPENAI_API_KEY"
  },
  "rerank": {
    "enabled": true,
    "model": "rerank-english-v3.0",
    "base_url": "https://api.cohere.com",
    "api_key_env": "COHERE_API_KEY"
  }
}

Verify: search results show reranker-sorted order. Disable reranker — results fall back to cosine order.

Semantic doctor

aft_search({ "query": "test" })  # trigger index build if cold
# Then check health via status command or semantic_doctor

Verify: health report shows ConfigSummary, IndexSummary, MetricsSummary, ProviderSummary.

Eval harness

// Create eval-cases.jsonl:
{"query": "authentication handler", "expected_paths": ["src/auth/middleware.ts"], "expected_symbols": ["authMiddleware"], "top_k": 10}
{"query": "database connection", "expected_paths": ["src/db/pool.ts"], "expected_symbols": ["createPool"], "top_k": 10}

Verify: returns recall@k and MRR scores.

Test coverage

~93 tests across 8 test sub-tasks covering:

• Config parsing and backward compatibility
• Fingerprint diff matrix (all field combinations → Rebuild/ClearQueryCache/None)
• File policy, docs chunking, and manifest handling
• VectorStore trait with DenseF32 and BinaryPacked implementations
• Binary packed-vector storage and Hamming search
• Lifecycle states, snapshots, and stale-vector pruning
• Search pipeline metrics, diagnostics, and DiagnosticsOutputMode
• Concurrency, race conditions, and cancellation token behavior
• Security trust boundary enforcement (project config stripping)
• Semantic doctor health report
• Semantic eval harness (JSONL parsing, scoring, recall/MRR)
• Reranking pipeline (parse multiple JSON shapes, fallback on failure)

Roadmap

Still in progress or planned for follow-up:

• aft-t6p.23: Complete contextualized document-chunk embedding (oversized docs, retry logic) — partially implemented
• aft-t6p.2.2: Configurable snippet truncation in reranking (currently hardcoded at 200 chars)
• aft-t6p.18: End-to-end verification across all backends
• aft-t6p.5: Configuration and operations documentation
• Performance benchmarking suite
• Migration tooling for index format upgrades

Architecture notes

Key new modules:

• crates/aft/src/semantic_rerank.rs — reranking pipeline with safe fallback
• crates/aft/src/semantic_diagnostics.rs — JSONL diagnostic logging
• crates/aft/src/semantic_doctor.rs — health-check report generation
• crates/aft/src/semantic_eval.rs — evaluation harness (JSONL parser, scoring)
• crates/aft/src/vector_store.rs — VectorStore trait with DenseF32 and BinaryPacked implementations
• crates/aft/src/commands/semantic_doctor.rs — doctor command handler
• crates/aft/src/commands/semantic_eval.rs — eval command handler

Modified significantly:

• crates/aft/src/semantic_index.rs — lifecycle management, fingerprint-driven invalidation, non-blocking build, stale pruning, typed vectors
• crates/aft/src/config.rs — provider profiles, rerank config, trust boundary fields
• crates/aft/src/commands/status.rs — semantic health metrics
• crates/aft/src/commands/semantic_search.rs — reranking integration, diagnostics output mode

---

^{Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.}

---

Summary by cubic

Provider-aware semantic search with typed embeddings, completed contextualized chunking, optional local model2vec, reranking, diagnostics (with JSONL), a doctor and eval harness, and an expanded Semble benchmark suite — now rebased on main, with Docker-based Rust validation for Windows, a manual multi-target build workflow, hardened reranker behavior, stricter config trust boundaries, a fixed deferred-file refresh on the snapshot/vector store, and search UX aligned with upstream (fallbacks, contract, and Partial index status).

• Bug Fixes

  • Hardened reranker: strip fenced JSON, stream and cap response body (2 MiB), guard out-of-bounds indices, de-duplicate results, and add SSRF validation for rerank_base_url.
  • Fixed cosine similarity edge cases (zero-norm → NaN) and applied consistent sort/cap across semantic-, lexical-, and hybrid-only paths.
  • Restored deferred-file refresh on the VectorStore snapshot path; background refresh respects semantic_files during incremental updates.
  • Normalized prompt templates and wired query-cache hits and warnings correctly into diagnostics; validated non-empty queries; corrected chunk end-line math.
  • Release builds now include semantic-model2vec and semantic-fts5 features across all platforms.

• New Features

  • Completed contextualized document-chunk embedding (Perplexity): oversized-doc splitting, retry with backoff, and build diagnostics.
  • Added JSONL diagnostics logging with retention, redaction options, and configurable output modes; warning de-dup to reduce noise.
  • Introduced semantic_doctor health report and extended status with semantic metrics (latency percentiles, zero-result/low-confidence rates, rerank state).
  • Optional local embeddings via model2vec-rs behind the semantic-model2vec feature; experimental FTS5 lexical backend behind semantic-fts5.
  • Configurable max_results_per_file cap; provider capability profiles and typed vectors include binary-packed + Hamming search.
  • Semble benchmarks: pilot runner, speed and token-efficiency tests, ablation, dual-pass rerank comparison, and a CI regression checker.
  • New manual “Build AFT Binary” workflow to build any branch with selected targets; Windows users can run Rust fmt/check/clippy/test via Docker.

^{Written for commit 4224315. Summary will update on new commits.}

Greptile Summary

This alpha PR graduates the semantic search subsystem from a minimal embedding prototype to a provider-capability-aware retrieval pipeline with typed vectors, reranking, background lifecycle management, JSONL diagnostics, a health-check doctor, and an evaluation harness. The core mechanics (fingerprint-driven index invalidation, binary-packed Hamming search, reranker safe-fallback, trust boundary enforcement) are structurally sound, but the new semantic_eval harness contains three correctness bugs that make its output misleading.

• Eval harness correctness: EvalCaseResult::index is set from 0-based enumerate() but documented as 1-based; parse_jsonl claims trailing commas are tolerated but serde_json::from_str rejects them; score_case includes hits beyond the top-k window in first_hit_rank, inflating MRR for configurations where relevant results sit just outside the recall window.
• Typed vector / reranking pipeline: provider capability profiles, binary-packed Hamming search, and reranker fallback logic all look correct; the OOB-index warning and RerankerFailure minimal-mode visibility gaps flagged in previous review rounds remain open.
• Trust boundary: backend, base_url, api_key_env, and model_path are user-only fields enforced in the TypeScript config layer; the new compress/trust.rs uses atomic rename writes and fail-closed semantics.

Confidence Score: 3/5

Safe to merge for the core search pipeline; the new eval harness ships with incorrect output that should be fixed before it is used to compare configurations.

The three bugs in semantic_eval.rs — 0-based index reported as 1-based, trailing-comma tolerance documented but not implemented, and MRR inflated by hits outside the recall window — collectively make the eval harness unreliable. Anyone who runs eval suites during the alpha period to compare providers or tuning knobs will get skewed MRR numbers and confusing case references. The core retrieval, reranking, and lifecycle code is structurally sound.

crates/aft/src/semantic_eval.rs has three correctness issues that affect the reliability of all eval measurements.

Important Files Changed

Filename	Overview
crates/aft/src/semantic_eval.rs	New eval harness with three logic bugs: 0-based index reported as 1-based, trailing-comma tolerance claimed but not implemented by serde_json, and first_hit_rank set from hits beyond k inflating MRR scores.
crates/aft/src/semantic_rerank.rs	New reranking pipeline with body-size cap, markdown-fence stripping, and safe fallback on failure; trailing-slash/double-v1 URL construction issue was previously flagged.
crates/aft/src/commands/semantic_search.rs	Reranking integration added with deduplication and OOB index handling; Partial index status no longer overwritten on search; previously flagged OOB warning gating on diagnostics_enabled still present.
crates/aft/src/semantic_diagnostics.rs	New diagnostics, metrics aggregation, and per-query JSONL logging; RerankerFailure is verbose-only with no minimal-mode warning (previously flagged, unresolved).
packages/opencode-plugin/src/config.ts	Previously missing reranking/diagnostics fields added to SemanticConfigSchema; enum values now aligned with Rust serde strings (previously flagged mismatches fixed).
crates/aft/src/vector_store.rs	New VectorStore trait with FlatF32 (cosine) and FlatBinaryHamming implementations; zero-norm pruning and orphan removal look correct.
crates/aft/src/compress/trust.rs	New trust-state module for project filter files; atomic write with rename, fail-closed on any error, well-tested with concurrent write and corruption tests.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    Q["aft_search(query)"] --> CM[choose_mode]
    CM -->|regex/literal| GS[handle_grep_search]
    CM -->|semantic/hybrid| HS[handle_semantic_or_hybrid_search]
    HS --> IDX{SemanticIndexStatus}
    IDX -->|Disabled/Failed| FB[fallback response]
    IDX -->|Building| BF[lexical-only fallback]
    IDX -->|Partial| PW[push PartialIndex warning]
    IDX -->|Ready| EQ[embed_query]
    PW --> EQ
    EQ --> VK[vector k-NN search]
    VK --> FH[fuse_hybrid_results max_results_per_file cap]
    FH --> RR{rerank_candidates}
    RR -->|Skipped| SC[score / format results]
    RR -->|ReRanked| OOB[filter OOB indices deduplicate append missing]
    RR -->|Failed| WARN[push RerankerFailure warning]
    OOB --> SC
    WARN --> SC
    SC --> DP[format_diagnostics_prefix output_mode: off/minimal/verbose]
    DP --> DIAG{diagnostics_enabled?}
    DIAG -->|yes| LOG[SearchMetricsCollector + JSONL logger]
    DIAG -->|no| RESP[search_response]
    LOG --> RESP

Loading

Comments Outside Diff (1)

1. packages/opencode-plugin/src/config.ts, line 37-54 (link)

   TypeScript enum values don't match the Rust serde strings — config will fail to deserialize

   Several new enum schemas use values that don't align with the Rust serde representation:

   • SemanticOutputEncodingEnum allows "binary", "ubinary", "int8", "uint8" but Rust OutputEncoding deserializes from "base64_binary" and "base64_int8".
   • SemanticStorageStrategyEnum allows "flat" and "binary_pack" but Rust StorageStrategy expects "native_f32" and "binary_packed".
   • SemanticInputModeEnum includes "chunk_extracts" and "contextualized" but Rust InputMode only has "flat_texts" and "document_chunks".
   • SemanticDistanceMetricEnum uses "dot" but Rust DistanceMetric expects "dot_product".
   • SemanticBackendEnum is missing the new "perplexity" variant added to Rust.

   A user who follows the TypeScript autocomplete and picks output_encoding: "int8" will pass TypeScript validation but receive a deserialization error (or silent fallback to default) from the Rust binary at runtime.

_{Reviews (19): Last reviewed commit: "fix: benchmark script CLI parsing and re..." | Re-trigger Greptile}

[cubic-dev-ai]

8 issues found across 14 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="crates/aft/src/lsp/child_registry.rs">

<violation number="1" location="crates/aft/src/lsp/child_registry.rs:223">
P2: Test child process performs non-fork-safe Rust operations after raw `fork()`, risking deadlocks and flaky CI failures under multithreaded test execution.</violation>
</file>

<file name="packages/opencode-plugin/src/config.ts">

<violation number="1" location="packages/opencode-plugin/src/config.ts:40">
P2: Semantic enum literals were renamed without backward-compatibility aliases or migration, breaking existing configs that use old values.</violation>
</file>

<file name="packages/aft-bridge/src/paths.ts">

<violation number="1" location="packages/aft-bridge/src/paths.ts:13">
P1: Failed root→harness migration silently drops existing version state and suppresses upgrade announcements</violation>
</file>

<file name="crates/aft/tests/integration/helpers.rs">

<violation number="1" location="crates/aft/tests/integration/helpers.rs:13">
P2: `panic!` will fail the test under nextest/Rust test harness semantics, so this helper does not actually skip root-only tests.</violation>
</file>

<file name="crates/aft/src/commands/semantic_search.rs">

<violation number="1" location="crates/aft/src/commands/semantic_search.rs:458">
P1: Lexical-only results can lose their original relevance ordering because sorting uses clamped display scores and falls back to alphabetical file/name tie-breaking instead of preserving the incoming lexical rank.</violation>
</file>

<file name="scripts/zir-aft-check.sh">

<violation number="1" location="scripts/zir-aft-check.sh:680">
P1: Checks now depend on host `git` and silently mutate repo-local git config (`core.autocrlf`), which can abort runs or cause unexpected persistent side effects</violation>
</file>

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[greptile-apps]

Want your agent to iterate on Greptile's feedback? Try greploops.

[cubic-dev-ai]

You're iterating quickly on this pull request. To help protect your rate limits, cubic has paused automatic reviews on new pushes for now—when you're ready for another review, comment @cubic-dev-ai review.

[greptile-apps]

Out-of-bounds reranker indices silently dropped when diagnostics_enabled: false

The guard if oob_count > 0 && diagnostics_enabled means that when the reranker returns indices beyond the result window (e.g. [0, 99, 1] against 5 candidates), the OOB indices are silently dropped and the affected results are appended in their original order — with no visible indication to the agent. Since diagnostics_enabled defaults to false, this is the common case. The hard-failure path immediately below carries an explicit comment "Always surface reranker failures — regardless of diagnostics_enabled," making the inconsistency clear: the OOB case should follow the same unconditional pattern.

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	cargo/​model2vec-rs@​0.2.1
	cargo/​ignore@​0.4.25 ⏵ 0.4.26

View full report

[socket-security]

Warning

Review the following alerts detected in dependencies.

According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.

Action	Severity	Alert  (click "▶" to expand/collapse)
Warn		Obfuscated code: cargo writeable is 90.0% likely obfuscated Confidence: 0.90 Location: Package overview From: ? → cargo/writeable@0.6.3 ℹ Read more on: This package | This alert | What is obfuscated code? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Packages should not obfuscate their code. Consider not using packages with obfuscated code. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore cargo/writeable@0.6.3. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.
Warn		Obfuscated code: cargo zerocopy is 90.0% likely obfuscated Confidence: 0.90 Location: Package overview From: ? → cargo/zerocopy@0.8.52 ℹ Read more on: This package | This alert | What is obfuscated code? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Packages should not obfuscate their code. Consider not using packages with obfuscated code. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore cargo/zerocopy@0.8.52. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

View full report

[greptile-apps]

RerankerFailure silently suppressed in the default Minimal output mode

format_warning_minimal returns None for RerankerFailure, so any reranker failure warning in diag_warnings — including the hard-failure case that is pushed unconditionally at semantic_search.rs:612 — produces no visible output in the default output_mode: minimal configuration. A user with rerank_enabled: true and every other setting at its default will receive silently reordered (or un-reordered) results with no indication that the reranker failed. Only users who explicitly set output_mode: "verbose" will see the warning text. RerankerFailure should produce at minimum a ⚠ reranker failed — results in original order string from format_warning_minimal, consistent with how StaleIndex and EmptyResults are treated.