fix: apply similarity threshold on raw cosine before normalization

README.md

@@ -80,7 +80,7 @@ Available in the Configuration screen's **Note Graph** section:
 | Setting | Default | Effect |
 |---|---|---|
 | Enable AI-based semantic analysis | Off | Adds semantic similarity edges using Joplin AI |
-| Similarity threshold | 50% | Lower values surface more semantic edges |
+| Similarity threshold | 70% | Percentile cutoff on raw similarity; only the strongest (100 − value)% of candidate pairs become edges. Lower values surface more semantic edges |
 | Max semantic edges per note | 5 | Caps how many semantic connections each note keeps |
 | Enable LLM analysis | Off | Adds Pass B category labels and relationship explanations |
 | Retry AI embedding | Off | One-shot: re-runs AI-based semantic analysis, reusing cached embeddings |

docs/settings.md

@@ -10,7 +10,7 @@ normal.
 | Setting | Key | Type | Default | Effect |
 |---|---|---|---|---|
 | Enable AI-based semantic analysis | `noteGraph.aiAnalysisEnabled` | Boolean | `false` | Turns semantic edges on or off. Requires Joplin AI to be enabled with a ready embedding index (Configuration screen's AI page). |
-| Similarity threshold (%) | `noteGraph.similarityThreshold` | Integer, 0-100, step 5 | `50` | Minimum bonus-boosted similarity score for a semantic edge to appear, as a percentage. Lower = more edges. Only applies when AI analysis is enabled. |
+| Similarity threshold (%) | `noteGraph.similarityThreshold` | Integer, 0-100, step 5 | `70` | Percentile cutoff on raw similarity: only the strongest (100 − value)% of candidate pairs become edges. Lower = more edges. Only applies when AI analysis is enabled. |
 | Max semantic edges per note (top-K) | `noteGraph.maxEdgesPerNote` | Integer, 1-20, step 1 | `5` | Caps how many of each note's strongest semantic connections are kept. Only applies when AI analysis is enabled. |
 | Enable LLM analysis | `noteGraph.llmEnrichmentEnabled` | Boolean | `false` | Turns on Pass B: category labels and relationship explanations via Joplin AI chat. Requires AI-based semantic analysis to also be enabled. See [LLM enrichment](llm-enrichment.md). |
 | Retry AI embedding | `noteGraph.retryEmbedding` | Boolean | `false` | One-shot trigger, not a persistent toggle: ticking it immediately retries AI-based semantic analysis (for example, after cancelling it), then unticks itself. No-op if the graph panel hasn't been opened yet. |

docs/similarity-engine.md

@@ -61,9 +61,19 @@ miss/no-op rather than failing the embed, so a corrupt cache degrades to
 fixed pipeline over every candidate note pair:
 
 ```
-raw scores -> floor -> normalize -> add bonuses -> threshold -> top-K
+raw scores -> floor (absolute) -> percentile cutoff (raw) -> normalize -> bonuses -> top-K
 ```
 
+The **percentile cutoff is checked on the raw score, before normalization**.
+Min-max normalization always maps the batch's most-similar pair to exactly
+1.0, so a post-normalization cutoff could never reject it — even in a vault of
+completely unrelated notes, the closest pair would be scaled up and pass.
+Checking the raw cosine first gives the cutoff a relative meaning; it keeps
+only the top `(1 - threshold)` fraction of the surviving raw scores, which is
+what separates a batch whose scores are all compressed into a narrow band
+(for example e5's [0.7, 1]). Normalization then only ranks the pairs that
+already passed.
+
 ### 1. Raw scores
 
 - **Vaults of 300 notes or fewer** (`LARGE_VAULT_THRESHOLD`): plain O(n²)
@@ -81,20 +91,33 @@ raw scores -> floor -> normalize -> add bonuses -> threshold -> top-K
 
 Pairs scoring below `SEMANTIC_FLOOR` (0.3) on the **raw** scale are dropped,
 unless the two notes are already directly linked (those are kept and
-resolved later, at the threshold step). This has to happen before
-normalization: min-max normalization always stretches the best pair in the
-batch to exactly 1.0, even in a vault of totally unrelated notes, so a floor
-applied *after* normalization could never reject anything. Flooring the raw
-score is what gives 0.3 an absolute, not batch-relative, meaning.
-
-### 3. Normalize
-
-Surviving scores, including any sub-floor pairs kept for being directly
-linked, are min-max normalized to `[0, 1]` together. If the spread between
-the batch's min and max is under 0.1, normalization is skipped (there is
-nothing meaningful to stretch).
-
-### 4. Bonuses
+resolved later, at the cutoff step). This has to happen before
+normalization: min-max normalization would always stretch the best pair in
+the batch to exactly 1.0, even in a vault of totally unrelated notes, so a
+floor applied *after* normalization could never reject anything. Flooring the
+raw score is what gives 0.3 an absolute, not batch-relative, meaning, and it
+is the small absolute floor that keeps a tiny vault with little data from
+manufacturing edges out of weak scores.
+
+### 3. Percentile cutoff
+
+Pairs whose raw score is below the score at the configured percentile
+(`DEFAULT_THRESHOLD` = 0.7, user-adjustable) are dropped. The percentile is
+computed over the above-floor raw scores in this batch, so it keeps only the
+top `(1 - threshold)` fraction — e.g. the strongest 30% at the default 70%.
+Like the floor, this runs on the raw scale, *before* normalization, and it is
+batch-relative on purpose: an unrelated pair that happens to be a batch's
+closest cannot be normalized up to pass, and a batch whose scores all sit in
+a narrow band (the e5 failure mode) is still separated by relative rank.
+Normalization then only ranks the pairs that already cleared it.
+
+### 4. Normalize
+
+Surviving scores are min-max normalized to `[0, 1]` together. If the spread
+between the batch's min and max is under 0.1, normalization is skipped (there
+is nothing meaningful to stretch).
+
+### 5. Bonuses
 
 Three additive bonuses nudge the normalized score:
 
@@ -109,10 +132,9 @@ Three additive bonuses nudge the normalized score:
 They are excluded from the tag-overlap bonus because sharing them says
 nothing about content similarity.
 
-### 5. Threshold
-
-Pairs whose bonus-boosted score is below the configured threshold
-(`DEFAULT_THRESHOLD` = 0.5, user-adjustable) are dropped.
+Because the floor and the percentile cutoff are checked on the *raw* score,
+bonuses can rank pairs but can never manufacture an edge out of a weak
+semantic score.
 
 ### 6. Top-K
 

docs/troubleshooting.md

@@ -26,12 +26,16 @@ simply the expected result. See [Settings reference](settings.md).
 
 ## The graph has very few or no semantic edges
 
-- Check the **similarity threshold** setting; 50% is the default and can be
-  lowered to surface more edges.
+- Check the **similarity threshold** setting; 70% is the default and can be
+  lowered to surface more edges. The threshold is a *percentile* cutoff on raw
+  similarity, so it always keeps only the strongest (100 − value)% of the
+  candidate pairs, even when the embedding model compresses every score into a
+  narrow band.
 - A small vault, or a vault with genuinely unrelated notes, will produce
-  fewer edges by design: the raw-score floor (`SEMANTIC_FLOOR`, 0.3) exists
-  specifically to prevent tag or link bonuses alone from manufacturing an
-  edge out of a weak semantic score. See [Similarity
+  fewer edges by design: semantic edges require the raw cosine similarity to
+  clear the absolute floor (`SEMANTIC_FLOOR`, 0.3) *before* the percentile
+  cutoff runs, and no tag, link, or time bonus can manufacture one out of a
+  weak semantic score. See [Similarity
   engine](similarity-engine.md).
 - Confirm the embedding index state is `ready` or at least `indexing` with
   meaningful progress, not `preparing`.

src/manifest.json

@@ -2,7 +2,7 @@
 	"manifest_version": 1,
 	"id": "com.yugalkaushik.plugin-note-graph",
 	"app_min_version": "3.5",
-	"version": "1.0.0",
+	"version": "1.0.1",
 	"name": "Note Graph Plugin",
 	"description": "Visualizes your notes as an interactive graph, connecting them by links, tags and AI-detected semantic similarity.",
 	"author": "yugalkaushik",

src/services/__snapshots__/AnalysisController.golden.test.ts.snap

@@ -63,6 +63,24 @@ exports[`AnalysisController (full pipeline golden set) renders the enriched sema
         "type": "semantic",
       },
     },
+    {
+      "data": {
+        "id": "11111111111111111111111111111111::44444444444444444444444444444444::semantic",
+        "score": 0.15000000000000002,
+        "source": "11111111111111111111111111111111",
+        "target": "44444444444444444444444444444444",
+        "type": "semantic",
+      },
+    },
+    {
+      "data": {
+        "id": "22222222222222222222222222222222::44444444444444444444444444444444::semantic",
+        "score": 0.1,
+        "source": "22222222222222222222222222222222",
+        "target": "44444444444444444444444444444444",
+        "type": "semantic",
+      },
+    },
     {
       "data": {
         "id": "33333333333333333333333333333333::44444444444444444444444444444444::semantic",
@@ -79,7 +97,7 @@ exports[`AnalysisController (full pipeline golden set) renders the enriched sema
       "data": {
         "category": "Gardening",
         "community": 0,
-        "degree": 6,
+        "degree": 7,
         "id": "11111111111111111111111111111111",
         "label": "Alpha Note",
         "noteId": "11111111111111111111111111111111",
@@ -90,11 +108,11 @@ exports[`AnalysisController (full pipeline golden set) renders the enriched sema
       "data": {
         "category": "Gardening",
         "community": 0,
-        "degree": 4,
+        "degree": 5,
         "id": "22222222222222222222222222222222",
         "label": "Beta",
         "noteId": "22222222222222222222222222222222",
-        "size": 6,
+        "size": 7,
       },
     },
     {
@@ -112,11 +130,11 @@ exports[`AnalysisController (full pipeline golden set) renders the enriched sema
       "data": {
         "category": "Gardening",
         "community": 0,
-        "degree": 3,
+        "degree": 5,
         "id": "44444444444444444444444444444444",
         "label": "Delta",
         "noteId": "44444444444444444444444444444444",
-        "size": 1,
+        "size": 6,
       },
     },
   ],

src/services/settings/GraphSettings.test.ts

@@ -30,7 +30,7 @@ describe('GraphSettings', () => {
 					}),
 					'noteGraph.similarityThreshold': expect.objectContaining({
 						type: SettingItemType.Int,
-						value: 50,
+						value: 70,
 						minimum: 0,
 						maximum: 100,
 						public: true,
@@ -122,7 +122,7 @@ describe('GraphSettings', () => {
 
 			expect(result.threshold).not.toBeNaN();
 			expect(result.topK).not.toBeNaN();
-			expect(result).toEqual({ threshold: 0.5, topK: 5 });
+			expect(result).toEqual({ threshold: 0.7, topK: 5 });
 		});
 
 		it('clamps an out-of-range threshold and topK to the registered min/max', async () => {
@@ -144,7 +144,7 @@ describe('GraphSettings', () => {
 
 			const result = await getSimilaritySettings();
 
-			expect(result).toEqual({ threshold: 0.5, topK: 5 });
+			expect(result).toEqual({ threshold: 0.7, topK: 5 });
 		});
 
 		it('falls back to defaults instead of clamping to the minimum when a value is null, empty, or a boolean', async () => {
@@ -155,7 +155,7 @@ describe('GraphSettings', () => {
 
 			const result = await getSimilaritySettings();
 
-			expect(result).toEqual({ threshold: 0.5, topK: 5 });
+			expect(result).toEqual({ threshold: 0.7, topK: 5 });
 
 			(joplin.settings.values as jest.Mock).mockResolvedValue({
 				'noteGraph.similarityThreshold': false,
@@ -164,7 +164,7 @@ describe('GraphSettings', () => {
 
 			const secondResult = await getSimilaritySettings();
 
-			expect(secondResult).toEqual({ threshold: 0.5, topK: 5 });
+			expect(secondResult).toEqual({ threshold: 0.7, topK: 5 });
 		});
 	});
 

src/services/settings/GraphSettings.ts

@@ -55,7 +55,7 @@ export async function registerGraphSettings(): Promise<void> {
 			section: SECTION_NAME,
 			label: 'Similarity threshold (%)',
 			description:
-				'Lower value = more semantic edges. Only applies when AI analysis is enabled.',
+				'Percentile cutoff on raw similarity: only the strongest (100 - value)% of candidate pairs become edges. Lower value = more semantic edges. Only applies when AI analysis is enabled.',
 		},
 		[MAX_EDGES_PER_NOTE_KEY]: {
 			value: TOP_K,