Harden telemetry provenance and hybrid retrieval isolation

.github/ISSUE_TEMPLATE/design-partner.yml

@@ -0,0 +1,61 @@
+name: Hosted Team design partner
+description: Apply for the paid, single-tenant founding Team pilot.
+title: "[Design partner] "
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for considering the founding pilot. This issue is public. Do not include credentials, private repository names or URLs, customer data, or confidential content.
+  - type: dropdown
+    id: team_size
+    attributes:
+      label: Engineering team size
+      options:
+        - 1-4
+        - 5-10
+        - 11-30
+        - 31+
+    validations:
+      required: true
+  - type: textarea
+    id: workflow
+    attributes:
+      label: Retrieval workflow
+      description: What do your developers or coding agents need to find today, and where does the current workflow fail?
+    validations:
+      required: true
+  - type: textarea
+    id: source_categories
+    attributes:
+      label: Source categories
+      description: List categories only, such as private Git repos, internal Markdown, Notion, or vendor docs. Do not name private systems.
+    validations:
+      required: true
+  - type: dropdown
+    id: deployment
+    attributes:
+      label: Preferred deployment boundary
+      options:
+        - Customer-controlled account
+        - Isolated managed instance
+        - Need to review both
+    validations:
+      required: true
+  - type: textarea
+    id: success
+    attributes:
+      label: What would make the 30-day pilot successful?
+      description: Include a measurable workflow outcome if possible.
+    validations:
+      required: true
+  - type: checkboxes
+    id: terms
+    attributes:
+      label: Pilot terms
+      options:
+        - label: I understand the founding pilot is $1,500 fixed for 30 days with no automatic renewal.
+          required: true
+        - label: I can provide one technical owner and at least 25 real queries for evaluation.
+          required: true
+        - label: I have not included confidential information in this public issue.
+          required: true

DESIGN_PARTNERS.md

@@ -0,0 +1,75 @@
+# Hosted Team founding design partner
+
+docs-puller will always be a local-first open-source CLI first. The proprietary
+hosted Team service is for teams that want the same private-corpus retrieval
+without operating synchronization, tenancy, access control, and evaluation.
+
+I am recruiting **one founding design partner** for a paid, single-tenant pilot.
+This is a concierge engagement, not a claim that the general-purpose hosted
+service is publicly launched.
+
+## Offer
+
+- **Price:** $1,500 fixed for 30 days; no automatic renewal.
+- **Team:** up to 10 users.
+- **Corpus:** up to 3 private repositories or documentation spaces plus up to
+  10 vendor sources supported by the existing pull surface.
+- **Deployment:** isolated managed instance, or a customer-controlled account
+  when the security review requires it.
+- **Retrieval:** local/BM25 retrieval by default; optional embedding and LLM
+  reranking is BYOK and activated only with explicit data-boundary approval.
+- **Onboarding:** source inventory, access-boundary review, initial ingest,
+  scheduled refresh, and a customer-specific fixture of at least 25 important
+  queries.
+- **Operation:** weekly health and retrieval-quality review, one shared support
+  channel, and reasonable connector fixes inside the agreed source set.
+- **Exit:** Markdown corpus export, eval fixture and results, configuration
+  export, and hosted-data deletion within 7 days of written confirmation.
+
+The provisional post-pilot Team price is **$199/month for up to 10 users**, plus
+provider usage at cost when optional hosted model calls are enabled. That price
+is modeled and will not be represented as validated until a partner agrees to
+continue.
+
+## Good fit
+
+- A 5–30 person engineering team already using coding agents or AI-enabled IDEs.
+- Important knowledge is split across private repositories, internal docs, and
+  fast-changing vendor documentation.
+- A hosted third-party retrieval service is unacceptable or insufficiently
+  measurable, but the team does not want to operate its own ingestion plane.
+- One technical owner can provide access, validate 25+ real queries, and join a
+  weekly 30-minute review.
+
+## Success contract
+
+The kickoff records a baseline and target rather than promising a generic
+quality number. A successful pilot must demonstrate all of the following:
+
+1. agreed sources synchronize through the isolated tenant without cross-tenant
+   access;
+2. at least 25 customer-authored queries have human-validated expected paths;
+3. retrieval meets the agreed Hit@5 target or improves at least 15 percentage
+   points over the kickoff baseline;
+4. at least 3 team members complete 50 provenance-labeled real searches;
+5. the operator receives a health, freshness, cost, and retrieval report; and
+6. the team chooses to continue, export and self-host, or delete without lock-in.
+
+## Not included
+
+- public multi-tenant availability, enterprise SLA, or multi-region failover;
+- SAML/SCIM, custom legal or compliance certification, or regulated data unless
+  separately reviewed;
+- unlimited repositories, arbitrary new connectors, or data migration outside
+  the stated export;
+- provider keys supplied by the operator without an approved BYOK boundary; or
+- a claim that payment alone validates the future $199/month plan.
+
+## Apply
+
+Open a
+[design-partner intake issue](https://github.com/nstranquist/docs-puller/issues/new?template=design-partner.yml).
+The issue is public: describe the workflow and source categories, but do not
+include repository names, credentials, private URLs, or confidential content.
+Detailed access and security discussion happens privately after fit is
+confirmed.

README.md

@@ -12,6 +12,20 @@ source tree. `docs-puller version --json` reports the build identity, supported
 commands, and stable capabilities for adapters such as `ndev docs`; release
 automation can fail closed with `docs-puller version --expect v0.2.3`.
 
+## Hosted Team Design Partner
+
+The local CLI stays free and local-first. I am recruiting one founding design
+partner for the proprietary hosted Team service: a 30-day, single-tenant pilot
+for an engineering team that wants private docs and repositories continuously
+synchronized and searchable by people and coding agents.
+
+The pilot is **$1,500 fixed**, has no automatic renewal, and includes a measured
+retrieval baseline plus a full corpus export at exit. The provisional post-pilot
+price is $199/month for up to 10 users; it will be validated with the first
+partner before becoming a public plan. Read the complete scope and exclusions in
+[DESIGN_PARTNERS.md](DESIGN_PARTNERS.md), then use the
+[design-partner intake issue](https://github.com/nstranquist/docs-puller/issues/new?template=design-partner.yml).
+
 ## Install
 
 Requirements: Go 1.26+.
@@ -81,25 +95,42 @@ docs-puller embed --out ~/code/docs --migrate-legacy
 docs-puller search "how do I count tokens with Anthropic" --out ~/code/docs --rerank-llm --rerank-hybrid --rerank-k 10
 ```
 
-The embedding batcher retries per-input token cap failures and recursively splits batches when the provider rejects total batch tokens.
+The embedding batcher retries per-input token cap failures and recursively
+splits batches when the provider rejects total batch tokens. A successful
+whole-document run also removes vectors for deleted or renamed documents before
+rewriting the flat sidecar. Source-scoped hybrid search filters the flat and
+SQLite vector paths before top-K selection, preserving the source isolation of
+the BM25 query.
 
 ## Telemetry To Fixture
 
-Query logging is opt-in:
+Query logging is on by default for normal `search` calls and can be disabled
+per call with `--log-query=false` or globally with
+`DOCS_PULLER_QUERY_LOG=0`. Integrations should identify themselves:
 
 ```sh
-docs-puller search "deploy Azure Functions from the CLI" --out ~/code/docs --log-query --intent support
-DOCS_PULLER_QUERY_LOG=1 docs-puller search "react native list performance" --out ~/code/docs
+docs-puller search "deploy Azure Functions from the CLI" --out ~/code/docs \
+  --intent support --client terminal --run-context operator
+DOCS_PULLER_QUERY_CLIENT=my-agent DOCS_PULLER_RUN_CONTEXT=agent \
+  docs-puller search "react native list performance" --out ~/code/docs
 ```
 
+Contexts `operator`, `agent`, `mcp`, and `production` count as real dogfood;
+`eval`, `test`, `benchmark`, and `batch` are synthetic; older or unlabeled
+rows stay `unknown`.
+
 Curate observed queries into a candidate fixture:
 
 ```sh
 docs-puller telemetry log --limit 20
+docs-puller telemetry summary --json
 docs-puller telemetry fixture --intent support --out-file eval/support-candidates.yaml
 ```
 
-Telemetry-derived fixtures use the observed top hit as `expect` and include a note to verify before promotion.
+Telemetry-derived fixtures use the observed top hit as `expect` and include a
+note to verify before promotion. Fixture export defaults to real traffic so
+repeated eval queries cannot silently become production fixtures; pass
+`--traffic-class all` only for an explicit legacy audit.
 
 ## Versioned Pins
 

embed.go

@@ -453,6 +453,10 @@ func writeFlatEmbeddingIndex(db *sql.DB, out, model string) error {
 }
 
 func topFlatEmbeddingPaths(out, model string, queryVec []float32, k int, depthPenalty float32) ([]embeddingScoredPath, bool, error) {
+	return topFlatEmbeddingPathsForSource(out, model, queryVec, k, depthPenalty, "")
+}
+
+func topFlatEmbeddingPathsForSource(out, model string, queryVec []float32, k int, depthPenalty float32, source string) ([]embeddingScoredPath, bool, error) {
 	metaPath, vecPath := flatEmbeddingPaths(out, model)
 	data, err := os.ReadFile(metaPath)
 	if err != nil {
@@ -484,6 +488,9 @@ func topFlatEmbeddingPaths(out, model string, queryVec []float32, k int, depthPe
 	qNorm := vectorNorm(queryVec)
 	scored := make([]embeddingScoredPath, 0, meta.Count)
 	for i, entry := range meta.Entries {
+		if !embeddingPathMatchesSource(entry.Path, source) {
+			continue
+		}
 		start := i * stride
 		c := cosineSimilarityBlob(queryVec, qNorm, vecBytes[start:start+stride])
 		if depthPenalty != 0 {
@@ -498,6 +505,26 @@ func topFlatEmbeddingPaths(out, model string, queryVec []float32, k int, depthPe
 	return scored, true, nil
 }
 
+func embeddingPathMatchesSource(path, source string) bool {
+	normalize := func(value string) string {
+		return strings.ReplaceAll(filepath.ToSlash(value), "\\", "/")
+	}
+	return source == "" || strings.HasPrefix(normalize(path), normalize(source)+"/")
+}
+
+func filterEmbeddingVectorsBySource(vecs map[string][]float32, source string) map[string][]float32 {
+	if source == "" {
+		return vecs
+	}
+	filtered := make(map[string][]float32)
+	for path, vec := range vecs {
+		if embeddingPathMatchesSource(path, source) {
+			filtered[path] = vec
+		}
+	}
+	return filtered
+}
+
 func mmapReadOnly(path string) ([]byte, func(), error) {
 	f, err := os.Open(path)
 	if err != nil {
@@ -669,8 +696,17 @@ func runEmbed(o embedOpts) error {
 		pending = append(pending, d)
 	}
 	if len(pending) == 0 {
-		if err := writeFlatEmbeddingIndex(db, o.out, o.model); err != nil {
-			fmt.Fprint(os.Stderr, searchruntime.EmbeddingFlatIndexWriteWarning(err))
+		if !o.dryRun {
+			pruned, err := pruneStaleEmbeddings(db, o.model, o.source, docs)
+			if err != nil {
+				return searchruntime.EmbeddingStoreBatchError(err)
+			}
+			if pruned > 0 {
+				fmt.Fprintf(os.Stderr, "embeddings: pruned %d stale path(s) for model=%s\n", pruned, o.model)
+			}
+			if err := writeFlatEmbeddingIndex(db, o.out, o.model); err != nil {
+				fmt.Fprint(os.Stderr, searchruntime.EmbeddingFlatIndexWriteWarning(err))
+			}
 		}
 		fmt.Print(searchruntime.EmbeddingUpToDateMessage(skipped, o.model))
 		return nil
@@ -708,6 +744,13 @@ func runEmbed(o embedOpts) error {
 	if msg := searchruntime.EmbeddingOversizeDocsWarning(skippedOversize); msg != "" {
 		fmt.Fprint(os.Stderr, msg)
 	}
+	pruned, err := pruneStaleEmbeddings(db, o.model, o.source, docs)
+	if err != nil {
+		return searchruntime.EmbeddingStoreBatchError(err)
+	}
+	if pruned > 0 {
+		fmt.Fprintf(os.Stderr, "embeddings: pruned %d stale path(s) for model=%s\n", pruned, o.model)
+	}
 	if err := writeFlatEmbeddingIndex(db, o.out, o.model); err != nil {
 		fmt.Fprint(os.Stderr, searchruntime.EmbeddingFlatIndexWriteWarning(err))
 	}
@@ -797,6 +840,66 @@ func collectDocsForEmbedding(out, source string) ([]docToEmbed, error) {
 	return docs, nil
 }
 
+// pruneStaleEmbeddings removes whole-document vectors whose paths no longer
+// exist in the live corpus. A source-scoped embed only prunes that exact source;
+// an all-source embed reconciles the complete selected model. Other models are
+// never touched. This keeps the model-wide flat sidecar from carrying deleted
+// or renamed documents indefinitely.
+func pruneStaleEmbeddings(db *sql.DB, model, source string, docs []docToEmbed) (int, error) {
+	live := make(map[string]struct{}, len(docs))
+	for _, doc := range docs {
+		live[doc.path] = struct{}{}
+	}
+
+	rows, err := db.Query(`SELECT path FROM embeddings WHERE model = ?`, model)
+	if err != nil {
+		return 0, fmt.Errorf("list cached embeddings for stale-path pruning: %w", err)
+	}
+	var stale []string
+	for rows.Next() {
+		var path string
+		if err := rows.Scan(&path); err != nil {
+			rows.Close()
+			return 0, fmt.Errorf("scan cached embedding path: %w", err)
+		}
+		if !embeddingPathMatchesSource(path, source) {
+			continue
+		}
+		if _, ok := live[path]; !ok {
+			stale = append(stale, path)
+		}
+	}
+	if err := rows.Close(); err != nil {
+		return 0, fmt.Errorf("close cached embedding path rows: %w", err)
+	}
+	if err := rows.Err(); err != nil {
+		return 0, fmt.Errorf("iterate cached embedding paths: %w", err)
+	}
+	if len(stale) == 0 {
+		return 0, nil
+	}
+
+	tx, err := db.Begin()
+	if err != nil {
+		return 0, fmt.Errorf("begin stale embedding prune: %w", err)
+	}
+	defer tx.Rollback()
+	stmt, err := tx.Prepare(`DELETE FROM embeddings WHERE path = ? AND model = ?`)
+	if err != nil {
+		return 0, fmt.Errorf("prepare stale embedding prune: %w", err)
+	}
+	defer stmt.Close()
+	for _, path := range stale {
+		if _, err := stmt.Exec(path, model); err != nil {
+			return 0, fmt.Errorf("prune stale embedding %q: %w", path, err)
+		}
+	}
+	if err := tx.Commit(); err != nil {
+		return 0, fmt.Errorf("commit stale embedding prune: %w", err)
+	}
+	return len(stale), nil
+}
+
 func loadEmbedCache(db *sql.DB, model string) (map[string]int64, error) {
 	rows, err := embeddb.New(db).ListEmbeddingMtimesByModel(context.Background(), model)
 	if err != nil {
@@ -1239,12 +1342,20 @@ func applyHybridRetrieval(query string, bm25Hits []searchHit, o searchOpts) ([]s
 			CallSite:  searchruntime.DefaultHybridRetrievalCallSite,
 			Record:    recordAIUsage,
 		}),
-		FlatTopK:         searchruntime.NewHybridFlatTopKCall(topFlatEmbeddingPaths),
+		FlatTopK: searchruntime.NewHybridFlatTopKCall(func(outDir, model string, queryVec []float32, k int, depthPenalty float32) ([]embeddingScoredPath, bool, error) {
+			return topFlatEmbeddingPathsForSource(outDir, model, queryVec, k, depthPenalty, o.source)
+		}),
 		WarnFlatFallback: searchruntime.NewHybridFlatFallbackWarningCall(os.Stderr),
 		OpenIndex:        searchruntime.NewReadOnlyEmbeddingIndexOpenCall(openEmbeddingsDB),
-		LoadVectors:      searchruntime.NewHybridVectorLoadCall(loadHybridCache),
-		ScoreVector:      searchruntime.CosineSimilarity,
-		Title:            searchruntime.NewHybridTitleLoader(o.out, extractTitle),
+		LoadVectors: searchruntime.NewHybridVectorLoadCall(func(db *sql.DB, model string) (map[string][]float32, error) {
+			vecs, err := loadHybridCache(db, model)
+			if err != nil {
+				return nil, err
+			}
+			return filterEmbeddingVectorsBySource(vecs, o.source), nil
+		}),
+		ScoreVector: searchruntime.CosineSimilarity,
+		Title:       searchruntime.NewHybridTitleLoader(o.out, extractTitle),
 	})
 }