Stale semantic nodes for deleted files are preserved indefinitely and returned as authoritative

[YmGavin]

Summary

Graph refresh preserves semantic/non-code nodes even when their source files no longer
exist. Those nodes remain queryable indefinitely, carry no staleness marker, and are
returned in polished, canonical-sounding form.

In our repository this has accumulated 1,444 semantic nodes for files that no longer
exist. Roughly half describe a deliberately retired dispatch_hub architecture. More
seriously, stale nodes also describe deleted security/policy surfaces such as:

• policy_hard_deny.py
• pre_tool_use_policy.py

Our agent instructions make Graphify the first code-navigation path before Grep/Read, so
these stale results can directly influence architectural and security reasoning.

Impact

This is not only graph bloat. The immediate safety problem is false authority:

• deleted architecture is returned without warning;
• retired systems can appear current and canonical;
• stale policy rationale can mislead agents reasoning about safety-critical code;
• refreshes grow the stale set monotonically because preserved non-code nodes have no
  source-existence check;
• current behavior provides no way for later refreshes to remove stale semantic nodes.

Requested fix: mark stale now, prune later

Phase 1 — reversible safety fix

On refresh, compare each source-backed semantic node against the current repository tree.

When the source path no longer exists:

• retain the node temporarily;
• mark it explicitly stale, for example:
  source_status: missing
stale: true
• include the last-seen source path and refresh timestamp;
• down-rank or exclude stale nodes from default authoritative query results;
• surface a clear warning when stale nodes are returned;
• expose stale counts in the refresh report.

This removes the immediate false-authority risk without destructive cleanup.

Phase 2 — rename-aware pruning

Before deletion, distinguish:

• true deletion;
• file rename/move;
• path normalization changes;
• generated or intentionally external sources.

Use Git rename detection or an equivalent content-identity mechanism to migrate nodes to
renamed paths. Only then prune nodes confirmed deleted under a governed retention policy.

Acceptance criteria

• Deleted-source semantic nodes are no longer returned as current/canonical without warning.
• Refresh reports total, newly stale, restored, renamed, and pruned node counts.
• A renamed file preserves or migrates its semantic content rather than being deleted.
• A genuinely deleted file becomes stale on the next refresh.
• Stale nodes are excluded or clearly down-ranked in default queries.
• Pruning is opt-in or separately gated until rename detection is proven.
• Tests cover deleted, renamed, restored, and security/policy-file cases.

Independent issue boundary

This is independent of drain/refresh latency and graph-size shrink-guard behavior, even if
the fixes touch the same refresh/preserve function.

A green latency result does not prove stale-node correctness, and a stale-marking fix does
not prove drain performance. Track them with separate tests and acceptance results.

[YmGavin]

Follow-up: query-time enforcement is load-bearing

Two findings from post-filing stress testing refine the rationale and acceptance criteria.

1. The rename-risk hypothesis did not hold in this case

Across the 152-commit interval we measured:

• 0 renames at a 50% similarity threshold
• 0 renames at a 25% similarity threshold
• the dispatch_hub removals were pure deletions

So rename ambiguity is not what currently justifies mark-first for this repository.

The mark-first decision still stands on two independent grounds:

• Reversibility: once a semantic node is pruned, reconstructing it may require checking out historical commits and re-running paid semantic extraction against files that no longer exist on the current tree.
• Archival value: some stale nodes are the only surviving prose explanation of retired systems and their rationale. They have negative navigational value but positive archival value. Marking can separate those roles; pruning destroys both.

2. Staleness must be enforced at query time, not merely stored

A stored field such as stale: true is insufficient if graphify query still returns that node without visibly surfacing or acting on the flag.

The original harm is not that stale nodes exist. It is that they are returned in polished, canonical-sounding form with no warning. A fix that writes a marker but leaves query output unchanged would preserve the safety failure while creating false confidence that it was solved.

Please treat the following as load-bearing acceptance criteria:

• Default queries must exclude or materially down-rank stale nodes.
• If a stale node is returned explicitly, the query output must display a clear stale/missing-source warning.
• Query ranking and rendered output must both consume the stale status.
• Tests must execute a real query against a known stale node and assert the warning/exclusion behavior, rather than only inspecting the stored field.
• Security/policy-file cases should be included in query-time tests.

The stale-node correctness track remains independent from refresh/drain latency and shrink-guard behavior.

[YmGavin]

Correction: removal exists, but ordinary refresh does not reliably remove these nodes

One sentence in the original report was too broad:

"current behavior provides no way for later refreshes to remove stale semantic nodes."

A removal capability does exist. In our environment it ran on 2026-07-08 and removed:

• 18 dead references
• 182 semantic nodes

The more precise problem is:

1. the ordinary Graphify update/refresh path does not remove these stale semantic nodes; and
2. the recovery/removal path classifies paths outside the active scan scope as keep.

As a result, semantic nodes for genuinely deleted files outside the current scan root can be retained indefinitely even though a narrower cleanup capability exists elsewhere.

This correction does not change the requested Phase-1 behavior:

• stale status must be computed during ordinary refresh;
• default queries must exclude or materially down-rank stale nodes;
• returned stale nodes must display a visible warning;
• query-time tests must verify the behavior against known deleted-source nodes.

The issue is therefore not "no deletion code exists." It is that stale-node lifecycle handling is incomplete and inconsistent across refresh and recovery paths, allowing deleted out-of-scope sources to remain authoritative in normal queries.

[xg-gh-25]

The root cause is that your semantic graph doesn't have a deletion listener — when files are removed from disk, the graph nodes remain orphaned because there's no cleanup hook that prunes them.

Why this happens:
Most semantic indexing systems operate in append-only mode: they detect new/changed files and update the graph, but don't actively scan for deletions. If you delete a file outside the indexer's watch path, the semantic nodes become stale ghosts.

Why it's dangerous:

• The agent retrieves deleted code as current, leading to incorrect context
• Merge conflicts when agent references nonexistent files
• No way for agent to know data is stale

Tactical fix:
Add a periodic cleanup job that walks your semantic graph, checks if each node's source file still exists on disk, and prunes nodes whose files are missing. Run during idle periods or as a post-indexing step.

Strategic fix (for maintainers):
Implement filesystem event monitoring (watchdog/inotify) that listens for DELETE events and immediately removes corresponding semantic nodes. Alternatively, add a sync barrier that compares the graph's file list against actual filesystem before returning query results — if a file is in graph but missing on disk, mark it stale or exclude it.

---

Context from an AI coding agent built with SwarmAI. Discussion: T-DDD

[xg-gh-25]

This is the "knowledge base drift" problem — semantic nodes become ghosts when the source files are deleted but the graph isn't pruned. The graph keeps serving stale embeddings as authoritative, causing the agent to reference non-existent code.

The core issue is that file deletions aren't being propagated to the graph index. Two likely causes:

1. File watcher gaps: The deletion event isn't being captured (e.g., if the file is deleted outside the IDE or via git operations)
2. Orphan protection: The graph intentionally preserves nodes for "soft deletes" to avoid losing context, but there's no TTL or manual prune command

Diagnostics:
Check if the semantic nodes have a "last_verified" timestamp. If they're older than the file's deletion time, that confirms the stale node problem.

Fix strategies:

1. Sync-on-query validation: Before returning a semantic node, verify the source file still exists. If not, mark the node as stale and exclude from results.

2. Explicit prune command: Add a graphify prune --verify-sources command to scan the graph and remove nodes for deleted files.

3. Git-aware indexing: Hook into git operations (post-commit, post-merge) to detect deletions and trigger graph cleanup.

The broader challenge is maintaining semantic index consistency across file lifecycle events — this is tracked in the data drift + deletion topic.

---

Context: tracking data drift + deletion across semantic indexes. SwarmAI. Discussion: T-DDD

[safishamsi]

Fixed in v0.9.21: a full graphify update now uses disk absence to evict semantic nodes from deleted non-AST sources (docs/papers/images); remote (://) sources are left untouched. https://github.com/Graphify-Labs/graphify/releases/tag/v0.9.21