diff --git a/README.md b/README.md index 55aa2e2..c1e5497 100644 --- a/README.md +++ b/README.md @@ -80,7 +80,7 @@ Available in the Configuration screen's **Note Graph** section: | Setting | Default | Effect | |---|---|---| | Enable AI-based semantic analysis | Off | Adds semantic similarity edges using Joplin AI | -| Similarity threshold | 50% | Lower values surface more semantic edges | +| Similarity threshold | 70% | Percentile cutoff on raw similarity; only the strongest (100 − value)% of candidate pairs become edges. Lower values surface more semantic edges | | Max semantic edges per note | 5 | Caps how many semantic connections each note keeps | | Enable LLM analysis | Off | Adds Pass B category labels and relationship explanations | | Retry AI embedding | Off | One-shot: re-runs AI-based semantic analysis, reusing cached embeddings | diff --git a/docs/settings.md b/docs/settings.md index 8aeee8a..ca0b028 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -10,7 +10,7 @@ normal. | Setting | Key | Type | Default | Effect | |---|---|---|---|---| | Enable AI-based semantic analysis | `noteGraph.aiAnalysisEnabled` | Boolean | `false` | Turns semantic edges on or off. Requires Joplin AI to be enabled with a ready embedding index (Configuration screen's AI page). | -| Similarity threshold (%) | `noteGraph.similarityThreshold` | Integer, 0-100, step 5 | `50` | Minimum bonus-boosted similarity score for a semantic edge to appear, as a percentage. Lower = more edges. Only applies when AI analysis is enabled. | +| Similarity threshold (%) | `noteGraph.similarityThreshold` | Integer, 0-100, step 5 | `70` | Percentile cutoff on raw similarity: only the strongest (100 − value)% of candidate pairs become edges. Lower = more edges. Only applies when AI analysis is enabled. | | Max semantic edges per note (top-K) | `noteGraph.maxEdgesPerNote` | Integer, 1-20, step 1 | `5` | Caps how many of each note's strongest semantic connections are kept. Only applies when AI analysis is enabled. | | Enable LLM analysis | `noteGraph.llmEnrichmentEnabled` | Boolean | `false` | Turns on Pass B: category labels and relationship explanations via Joplin AI chat. Requires AI-based semantic analysis to also be enabled. See [LLM enrichment](llm-enrichment.md). | | Retry AI embedding | `noteGraph.retryEmbedding` | Boolean | `false` | One-shot trigger, not a persistent toggle: ticking it immediately retries AI-based semantic analysis (for example, after cancelling it), then unticks itself. No-op if the graph panel hasn't been opened yet. | diff --git a/docs/similarity-engine.md b/docs/similarity-engine.md index 339466f..fd3a0b8 100644 --- a/docs/similarity-engine.md +++ b/docs/similarity-engine.md @@ -61,9 +61,19 @@ miss/no-op rather than failing the embed, so a corrupt cache degrades to fixed pipeline over every candidate note pair: ``` -raw scores -> floor -> normalize -> add bonuses -> threshold -> top-K +raw scores -> floor (absolute) -> percentile cutoff (raw) -> normalize -> bonuses -> top-K ``` +The **percentile cutoff is checked on the raw score, before normalization**. +Min-max normalization always maps the batch's most-similar pair to exactly +1.0, so a post-normalization cutoff could never reject it — even in a vault of +completely unrelated notes, the closest pair would be scaled up and pass. +Checking the raw cosine first gives the cutoff a relative meaning; it keeps +only the top `(1 - threshold)` fraction of the surviving raw scores, which is +what separates a batch whose scores are all compressed into a narrow band +(for example e5's [0.7, 1]). Normalization then only ranks the pairs that +already passed. + ### 1. Raw scores - **Vaults of 300 notes or fewer** (`LARGE_VAULT_THRESHOLD`): plain O(n²) @@ -81,20 +91,33 @@ raw scores -> floor -> normalize -> add bonuses -> threshold -> top-K Pairs scoring below `SEMANTIC_FLOOR` (0.3) on the **raw** scale are dropped, unless the two notes are already directly linked (those are kept and -resolved later, at the threshold step). This has to happen before -normalization: min-max normalization always stretches the best pair in the -batch to exactly 1.0, even in a vault of totally unrelated notes, so a floor -applied *after* normalization could never reject anything. Flooring the raw -score is what gives 0.3 an absolute, not batch-relative, meaning. - -### 3. Normalize - -Surviving scores, including any sub-floor pairs kept for being directly -linked, are min-max normalized to `[0, 1]` together. If the spread between -the batch's min and max is under 0.1, normalization is skipped (there is -nothing meaningful to stretch). - -### 4. Bonuses +resolved later, at the cutoff step). This has to happen before +normalization: min-max normalization would always stretch the best pair in +the batch to exactly 1.0, even in a vault of totally unrelated notes, so a +floor applied *after* normalization could never reject anything. Flooring the +raw score is what gives 0.3 an absolute, not batch-relative, meaning, and it +is the small absolute floor that keeps a tiny vault with little data from +manufacturing edges out of weak scores. + +### 3. Percentile cutoff + +Pairs whose raw score is below the score at the configured percentile +(`DEFAULT_THRESHOLD` = 0.7, user-adjustable) are dropped. The percentile is +computed over the above-floor raw scores in this batch, so it keeps only the +top `(1 - threshold)` fraction — e.g. the strongest 30% at the default 70%. +Like the floor, this runs on the raw scale, *before* normalization, and it is +batch-relative on purpose: an unrelated pair that happens to be a batch's +closest cannot be normalized up to pass, and a batch whose scores all sit in +a narrow band (the e5 failure mode) is still separated by relative rank. +Normalization then only ranks the pairs that already cleared it. + +### 4. Normalize + +Surviving scores are min-max normalized to `[0, 1]` together. If the spread +between the batch's min and max is under 0.1, normalization is skipped (there +is nothing meaningful to stretch). + +### 5. Bonuses Three additive bonuses nudge the normalized score: @@ -109,10 +132,9 @@ Three additive bonuses nudge the normalized score: They are excluded from the tag-overlap bonus because sharing them says nothing about content similarity. -### 5. Threshold - -Pairs whose bonus-boosted score is below the configured threshold -(`DEFAULT_THRESHOLD` = 0.5, user-adjustable) are dropped. +Because the floor and the percentile cutoff are checked on the *raw* score, +bonuses can rank pairs but can never manufacture an edge out of a weak +semantic score. ### 6. Top-K diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 067f521..1eecab2 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -26,12 +26,16 @@ simply the expected result. See [Settings reference](settings.md). ## The graph has very few or no semantic edges -- Check the **similarity threshold** setting; 50% is the default and can be - lowered to surface more edges. +- Check the **similarity threshold** setting; 70% is the default and can be + lowered to surface more edges. The threshold is a *percentile* cutoff on raw + similarity, so it always keeps only the strongest (100 − value)% of the + candidate pairs, even when the embedding model compresses every score into a + narrow band. - A small vault, or a vault with genuinely unrelated notes, will produce - fewer edges by design: the raw-score floor (`SEMANTIC_FLOOR`, 0.3) exists - specifically to prevent tag or link bonuses alone from manufacturing an - edge out of a weak semantic score. See [Similarity + fewer edges by design: semantic edges require the raw cosine similarity to + clear the absolute floor (`SEMANTIC_FLOOR`, 0.3) *before* the percentile + cutoff runs, and no tag, link, or time bonus can manufacture one out of a + weak semantic score. See [Similarity engine](similarity-engine.md). - Confirm the embedding index state is `ready` or at least `indexing` with meaningful progress, not `preparing`. diff --git a/src/manifest.json b/src/manifest.json index bac2a7c..38da157 100644 --- a/src/manifest.json +++ b/src/manifest.json @@ -2,7 +2,7 @@ "manifest_version": 1, "id": "com.yugalkaushik.plugin-note-graph", "app_min_version": "3.5", - "version": "1.0.0", + "version": "1.0.1", "name": "Note Graph Plugin", "description": "Visualizes your notes as an interactive graph, connecting them by links, tags and AI-detected semantic similarity.", "author": "yugalkaushik", diff --git a/src/services/__snapshots__/AnalysisController.golden.test.ts.snap b/src/services/__snapshots__/AnalysisController.golden.test.ts.snap index b691f25..79e5b85 100644 --- a/src/services/__snapshots__/AnalysisController.golden.test.ts.snap +++ b/src/services/__snapshots__/AnalysisController.golden.test.ts.snap @@ -63,6 +63,24 @@ exports[`AnalysisController (full pipeline golden set) renders the enriched sema "type": "semantic", }, }, + { + "data": { + "id": "11111111111111111111111111111111::44444444444444444444444444444444::semantic", + "score": 0.15000000000000002, + "source": "11111111111111111111111111111111", + "target": "44444444444444444444444444444444", + "type": "semantic", + }, + }, + { + "data": { + "id": "22222222222222222222222222222222::44444444444444444444444444444444::semantic", + "score": 0.1, + "source": "22222222222222222222222222222222", + "target": "44444444444444444444444444444444", + "type": "semantic", + }, + }, { "data": { "id": "33333333333333333333333333333333::44444444444444444444444444444444::semantic", @@ -79,7 +97,7 @@ exports[`AnalysisController (full pipeline golden set) renders the enriched sema "data": { "category": "Gardening", "community": 0, - "degree": 6, + "degree": 7, "id": "11111111111111111111111111111111", "label": "Alpha Note", "noteId": "11111111111111111111111111111111", @@ -90,11 +108,11 @@ exports[`AnalysisController (full pipeline golden set) renders the enriched sema "data": { "category": "Gardening", "community": 0, - "degree": 4, + "degree": 5, "id": "22222222222222222222222222222222", "label": "Beta", "noteId": "22222222222222222222222222222222", - "size": 6, + "size": 7, }, }, { @@ -112,11 +130,11 @@ exports[`AnalysisController (full pipeline golden set) renders the enriched sema "data": { "category": "Gardening", "community": 0, - "degree": 3, + "degree": 5, "id": "44444444444444444444444444444444", "label": "Delta", "noteId": "44444444444444444444444444444444", - "size": 1, + "size": 6, }, }, ], diff --git a/src/services/settings/GraphSettings.test.ts b/src/services/settings/GraphSettings.test.ts index 74d4a63..b038f60 100644 --- a/src/services/settings/GraphSettings.test.ts +++ b/src/services/settings/GraphSettings.test.ts @@ -30,7 +30,7 @@ describe('GraphSettings', () => { }), 'noteGraph.similarityThreshold': expect.objectContaining({ type: SettingItemType.Int, - value: 50, + value: 70, minimum: 0, maximum: 100, public: true, @@ -122,7 +122,7 @@ describe('GraphSettings', () => { expect(result.threshold).not.toBeNaN(); expect(result.topK).not.toBeNaN(); - expect(result).toEqual({ threshold: 0.5, topK: 5 }); + expect(result).toEqual({ threshold: 0.7, topK: 5 }); }); it('clamps an out-of-range threshold and topK to the registered min/max', async () => { @@ -144,7 +144,7 @@ describe('GraphSettings', () => { const result = await getSimilaritySettings(); - expect(result).toEqual({ threshold: 0.5, topK: 5 }); + expect(result).toEqual({ threshold: 0.7, topK: 5 }); }); it('falls back to defaults instead of clamping to the minimum when a value is null, empty, or a boolean', async () => { @@ -155,7 +155,7 @@ describe('GraphSettings', () => { const result = await getSimilaritySettings(); - expect(result).toEqual({ threshold: 0.5, topK: 5 }); + expect(result).toEqual({ threshold: 0.7, topK: 5 }); (joplin.settings.values as jest.Mock).mockResolvedValue({ 'noteGraph.similarityThreshold': false, @@ -164,7 +164,7 @@ describe('GraphSettings', () => { const secondResult = await getSimilaritySettings(); - expect(secondResult).toEqual({ threshold: 0.5, topK: 5 }); + expect(secondResult).toEqual({ threshold: 0.7, topK: 5 }); }); }); diff --git a/src/services/settings/GraphSettings.ts b/src/services/settings/GraphSettings.ts index 458a185..1395f54 100644 --- a/src/services/settings/GraphSettings.ts +++ b/src/services/settings/GraphSettings.ts @@ -55,7 +55,7 @@ export async function registerGraphSettings(): Promise { section: SECTION_NAME, label: 'Similarity threshold (%)', description: - 'Lower value = more semantic edges. Only applies when AI analysis is enabled.', + 'Percentile cutoff on raw similarity: only the strongest (100 - value)% of candidate pairs become edges. Lower value = more semantic edges. Only applies when AI analysis is enabled.', }, [MAX_EDGES_PER_NOTE_KEY]: { value: TOP_K, diff --git a/src/services/similarity/SimilarityEngine.test.ts b/src/services/similarity/SimilarityEngine.test.ts index 9bed0e1..2852744 100644 --- a/src/services/similarity/SimilarityEngine.test.ts +++ b/src/services/similarity/SimilarityEngine.test.ts @@ -102,11 +102,10 @@ describe('SimilarityEngine', () => { }); it('gives higher scores to more similar notes', async () => { - // With the floor applied to the raw score before normalize, whichever - // pair is weakest among the floor survivors normalizes to exactly 0 — - // b-c (raw ~0.589) plays that role here so it doesn't drag a-c down - // with it, letting both a-b and a-c clear the threshold with a-b - // still scoring higher. + // A loose percentile keeps all three above-floor pairs so the test + // isolates normalization: whichever pair is weakest normalizes to + // exactly 0 — b-c (raw ~0.589) plays that role here so it doesn't + // drag a-c down with it, letting a-b score higher than a-c. const notes = [makeNote('a', 'A'), makeNote('b', 'B'), makeNote('c', 'C')]; const embedded = [ embed('a', [1, 0, 0]), @@ -115,7 +114,7 @@ describe('SimilarityEngine', () => { ]; const engine = new SimilarityEngine(notes, embedded); - const pairs = await engine.compute(); + const pairs = await engine.compute(0.2); const abScore = pairs.find( (p) => @@ -279,7 +278,7 @@ describe('SimilarityEngine', () => { ]; const engine = new SimilarityEngine(notes, embedded); - const pairs = await engine.compute(); + const pairs = await engine.compute(0.2); const abWithLink = pairs.find( (p) => @@ -423,12 +422,14 @@ describe('SimilarityEngine', () => { describe('top-K filtering', () => { it('limits edges per note', async () => { + // 30° steps keep each note's two neighbours above the raw threshold + // (cos 30° ≈ 0.87), so top-K has something to limit. const notes = []; const embedded = []; for (let i = 0; i < 6; i++) { notes.push(makeNote(`n${i}`, `Note ${i}`)); embedded.push( - embed(`n${i}`, [Math.cos((i * Math.PI) / 3), Math.sin((i * Math.PI) / 3)]) + embed(`n${i}`, [Math.cos((i * Math.PI) / 6), Math.sin((i * Math.PI) / 6)]) ); } @@ -471,31 +472,57 @@ describe('SimilarityEngine', () => { }); describe('custom threshold and top-K overrides', () => { - it('applies a stricter caller-supplied threshold instead of DEFAULT_THRESHOLD', async () => { - const notes = [makeNote('a', 'A'), makeNote('b', 'B')]; - const embedded = [embed('a', [1, 0]), embed('b', [0.95, 0.3])]; + it('applies a stricter caller-supplied percentile, keeping fewer pairs than DEFAULT_THRESHOLD', async () => { + // Raw scores span 0.5 to ~0.93, all above the floor. The default 0.7 + // percentile keeps the top ~30% (a-b and c-d); a stricter 0.95 keeps + // only the single strongest pair (c-d). + const notes = [makeNote('a', 'A'), makeNote('b', 'B'), makeNote('c', 'C'), makeNote('d', 'D')]; + const embedded = [ + embed('a', [1, 0, 0]), + embed('b', [0.9, Math.sqrt(1 - 0.81), 0]), + embed('c', [0.7, 0, Math.sqrt(1 - 0.49)]), + embed('d', [0.5, 0.3, Math.sqrt(1 - 0.25 - 0.09)]), + ]; const engine = new SimilarityEngine(notes, embedded); const defaultPairs = await engine.compute(); - const strictPairs = await engine.compute(1.2); + const strictPairs = await engine.compute(0.95); - expect(defaultPairs).toHaveLength(1); - expect(strictPairs).toEqual([]); + expect(defaultPairs).toHaveLength(2); + expect(strictPairs).toHaveLength(1); + expect(strictPairs[0].source).toBe('c'); + expect(strictPairs[0].target).toBe('d'); }); - it('applies a looser caller-supplied threshold that admits a pair DEFAULT_THRESHOLD would reject', async () => { - // Raw cosine 0.35 (above SEMANTIC_FLOOR) plus the same-day temporal - // bonus (0.1) lands at 0.45 — below DEFAULT_THRESHOLD (0.5) but above - // a caller-supplied 0.4. - const notes = [makeNote('a', 'A'), makeNote('b', 'B')]; - const embedded = [embed('a', [1, 0]), embed('b', [0.35, Math.sqrt(1 - 0.35 * 0.35)])]; + it('applies a looser caller-supplied percentile, admitting pairs DEFAULT_THRESHOLD would reject', async () => { + // A looser percentile lowers the raw-score cutoff, admitting pairs + // the default rejects. a-c (raw 0.7) is below the default cutoff + // (~0.9) but above the looser one, so it appears only with the + // looser percentile. + const notes = [makeNote('a', 'A'), makeNote('b', 'B'), makeNote('c', 'C'), makeNote('d', 'D')]; + const embedded = [ + embed('a', [1, 0, 0]), + embed('b', [0.9, Math.sqrt(1 - 0.81), 0]), + embed('c', [0.7, 0, Math.sqrt(1 - 0.49)]), + embed('d', [0.5, 0.3, Math.sqrt(1 - 0.25 - 0.09)]), + ]; const engine = new SimilarityEngine(notes, embedded); const defaultPairs = await engine.compute(); - const loosePairs = await engine.compute(0.4); + const loosePairs = await engine.compute(0.2); + + const acInDefault = defaultPairs.some( + (p) => + (p.source === 'a' && p.target === 'c') || (p.source === 'c' && p.target === 'a') + ); + const acInLoose = loosePairs.some( + (p) => + (p.source === 'a' && p.target === 'c') || (p.source === 'c' && p.target === 'a') + ); - expect(defaultPairs).toEqual([]); - expect(loosePairs).toHaveLength(1); + expect(acInDefault).toBe(false); + expect(acInLoose).toBe(true); + expect(loosePairs.length).toBeGreaterThan(defaultPairs.length); }); it('applies a caller-supplied top-K instead of TOP_K', async () => { @@ -503,12 +530,13 @@ describe('SimilarityEngine', () => { // keeps it in its own top-K), so topK=0 is the only value that // unambiguously proves the override took effect: every note's own // kept list is empty, so no pair can survive from any side. + // 30° steps keep adjacent pairs above the raw threshold. const notes = []; const embedded = []; for (let i = 0; i < 6; i++) { notes.push(makeNote(`n${i}`, `Note ${i}`)); embedded.push( - embed(`n${i}`, [Math.cos((i * Math.PI) / 3), Math.sin((i * Math.PI) / 3)]) + embed(`n${i}`, [Math.cos((i * Math.PI) / 6), Math.sin((i * Math.PI) / 6)]) ); } @@ -521,6 +549,50 @@ describe('SimilarityEngine', () => { }); }); + describe('percentile cutoff', () => { + it('keeps only the top fraction of pairs when every score is compressed into a narrow high band', async () => { + // Mimics the e5 failure mode: all pairwise cosines sit in a narrow + // high band (0.77-0.99) so an absolute threshold would reject + // nothing. The percentile cutoff still prunes to the top ~30%. + const notes = []; + const embedded = []; + for (let i = 0; i < 6; i++) { + const angle = 10 + i * 8; + notes.push(makeNote(`n${i}`, `Note ${i}`)); + embedded.push( + embed(`n${i}`, [Math.cos((angle * Math.PI) / 180), Math.sin((angle * Math.PI) / 180)]) + ); + } + + const engine = new SimilarityEngine(notes, embedded); + const pairs = await engine.compute(); + + expect(pairs.length).toBeGreaterThan(0); + expect(pairs.length).toBeLessThan(15); + for (const p of pairs) { + expect(p.score).toBeGreaterThan(0); + } + }); + + it('keeps the absolute floor decisive in a tiny vault with little data', async () => { + // A 3-note vault where every pair is below SEMANTIC_FLOOR but + // clearly ranked (0.1, 0.2, 0.3-tie is avoided here). The percentile + // alone would admit the top pair; the small absolute floor keeps a + // vault this sparse from manufacturing any edge. + const notes = [makeNote('a', 'A'), makeNote('b', 'B'), makeNote('c', 'C')]; + const embedded = [ + embed('a', [1, 0, 0]), + embed('b', [0.2, Math.sqrt(1 - 0.04), 0]), + embed('c', [0.1, 0, Math.sqrt(1 - 0.01)]), + ]; + + const engine = new SimilarityEngine(notes, embedded); + const pairs = await engine.compute(); + + expect(pairs).toEqual([]); + }); + }); + describe('SEMANTIC_FLOOR and threshold ordering', () => { it('rejects a below-floor pair even with a shared tag', async () => { // a and b are nearly orthogonal (cosine ~0), share a tag but are not @@ -536,11 +608,12 @@ describe('SimilarityEngine', () => { expect(pairs).toEqual([]); }); - it('lets a direct link bypass the floor, but the boosted score must still clear the threshold', async () => { + it('lets a direct link bypass the floor, but the boosted score must still clear the cutoff', async () => { // a and b are nearly orthogonal (cosine ~0) but directly link to each other // and share a tag. The link bypasses SEMANTIC_FLOOR (a user-created edge - // isn't a false positive), but the resulting boosted score (~0.25) still - // isn't enough to clear DEFAULT_THRESHOLD (0.5). + // isn't a false positive), but there is no above-floor pair to compute a + // percentile from, so the cutoff falls back to SEMANTIC_FLOOR and the + // boosted score (~0.25) still isn't enough to clear it. const notes = [ makeNote('a', 'A', ['b'], ['shared']), makeNote('b', 'B', [], ['shared']), @@ -581,6 +654,53 @@ describe('SimilarityEngine', () => { }); }); + describe('threshold is applied to the raw score, before normalization', () => { + it('keeps the strongest pair of a weak batch, since the cutoff is relative to the batch', async () => { + // a-b (raw 0.45) is the batch's strongest above-floor pair, so the + // percentile cutoff admits it even though 0.45 is weak in absolute + // terms — relative ranking is what matters. b-c (raw 0.144) is + // below SEMANTIC_FLOOR and never becomes a candidate. + const notes = [makeNote('a', 'A'), makeNote('b', 'B'), makeNote('c', 'C')]; + const embedded = [ + embed('a', [1, 0, 0]), + embed('b', [0.45, Math.sqrt(1 - 0.45 * 0.45), 0]), + embed('c', [0.32, 0, Math.sqrt(1 - 0.32 * 0.32)]), + ]; + + const engine = new SimilarityEngine(notes, embedded); + const pairs = await engine.compute(); + + expect(pairs).toHaveLength(1); + expect(pairs[0].source).toBe('a'); + expect(pairs[0].target).toBe('b'); + }); + + it('cannot manufacture a semantic edge from a shared tag or direct link below the raw percentile cutoff', async () => { + // a-b is the batch's strongest pair (raw 0.9). b-c is weaker (raw + // 0.59) but above the floor and shares a tag and a direct link — + // the bonuses still can't bring it back once the raw percentile + // cutoff (~0.9) has rejected it, because the cutoff runs before + // bonuses. + const notes = [ + makeNote('a', 'A'), + makeNote('b', 'B', ['c'], ['shared']), + makeNote('c', 'C', [], ['shared']), + ]; + const embedded = [ + embed('a', [1, 0, 0]), + embed('b', [0.9, Math.sqrt(1 - 0.81), 0]), + embed('c', [0.8, -0.3, Math.sqrt(0.27)]), + ]; + + const engine = new SimilarityEngine(notes, embedded); + const pairs = await engine.compute(); + + expect(pairs).toHaveLength(1); + expect(pairs[0].source).toBe('a'); + expect(pairs[0].target).toBe('b'); + }); + }); + describe('large vault (search-based) path retry', () => { function makeLargeVault(): { notes: Note[]; embedded: EmbeddedNote[] } { const count = LARGE_VAULT_THRESHOLD + 1; diff --git a/src/services/similarity/SimilarityEngine.ts b/src/services/similarity/SimilarityEngine.ts index b4125f2..1c88dea 100644 --- a/src/services/similarity/SimilarityEngine.ts +++ b/src/services/similarity/SimilarityEngine.ts @@ -45,14 +45,21 @@ export class SimilarityEngine { /** * Orchestrates the full similarity pipeline: - * compute → floor (raw scores) → normalize → enrich → threshold → top-K. + * compute → floor (raw scores) → percentile cutoff (raw scores) → normalize → + * bonuses → top-K. * - * SEMANTIC_FLOOR is applied to *raw* scores, before normalization. Min-max - * normalization always maps the batch's most-similar pair to exactly 1.0, - * so a post-normalization floor can never reject it — even in a vault of - * completely unrelated notes. Flooring on the raw scale (where 0.3 has an - * absolute meaning) is what actually guarantees that tags alone can never - * manufacture an edge out of a weak semantic score. + * Both the floor and the cutoff are checked on the *raw* score, before + * normalization. Min-max normalization always maps the batch's most-similar + * pair to exactly 1.0, so a post-normalization threshold could never reject + * it — even in a vault of completely unrelated notes, the closest pair + * would be scaled up and pass. Checking the raw cosine first gives the + * cutoff a relative meaning; normalization then only ranks the pairs that + * already passed. SEMANTIC_FLOOR (also raw, before normalization) is the + * small absolute floor that guards tiny vaults with little data: a pair can + * never become an edge on a weak raw score, no matter how relatively close + * it is. The percentile cutoff keeps only the top (1 - threshold) fraction + * of the surviving raw scores, which is what separates a batch whose scores + * are all compressed into a narrow band (e.g. e5's [0.7, 1]). */ public async compute( threshold: number = DEFAULT_THRESHOLD, @@ -75,10 +82,16 @@ export class SimilarityEngine { return []; } - const normalized = this.normalize(aboveFloor, SEMANTIC_FLOOR); + const cutoff = this.percentileCutoff(aboveFloor, threshold); + const aboveThreshold = this.filterBelowThreshold(aboveFloor, cutoff); + + if (aboveThreshold.length === 0) { + return []; + } + + const normalized = this.normalize(aboveThreshold, SEMANTIC_FLOOR); const enriched = this.addBonusPoints(normalized); - const aboveThreshold = this.filterBelowThreshold(enriched, threshold); - const topPairs = this.selectTopK(aboveThreshold, topK); + const topPairs = this.selectTopK(enriched, topK); return topPairs; } @@ -122,8 +135,8 @@ export class SimilarityEngine { * property without invoking it (see JoplinNativeProvider.validateAiApi for why). * * Score-scale assumption: search relevance scores are treated as raw - * similarity scores and flow through the same floor → normalize pipeline - * as cosine scores. + * similarity scores and flow through the same floor → threshold → + * normalize pipeline as cosine scores. * * Failure handling: each note's search call is retried on transient * failures before being skipped; a partial candidate set is still @@ -305,7 +318,7 @@ export class SimilarityEngine { /** * Removes pairs whose *raw* score is below the safety floor — unless the * notes are directly linked, in which case they're kept and left for the - * threshold check later. Runs before normalization on purpose: the floor + * cutoff check later. Runs before normalization on purpose: the floor * guards against spurious tag-only edges, which requires an absolute * scale, not a batch-relative one. */ @@ -313,7 +326,25 @@ export class SimilarityEngine { return pairs.filter((p) => p.score >= floor || this.isDirectlyLinked(p)); } - /** Keeps only pairs whose bonus-boosted score clears the threshold. */ + private percentileCutoff(pairs: SimilarityPair[], percentile: number): number { + const scores = pairs + .filter((p) => p.score >= SEMANTIC_FLOOR) + .map((p) => p.score) + .sort((a, b) => a - b); + + if (scores.length === 0) { + return SEMANTIC_FLOOR; + } + + const clamped = Math.min(1, Math.max(0, percentile)); + const index = Math.min( + scores.length - 1, + Math.max(0, Math.ceil(clamped * scores.length) - 1) + ); + return scores[index]; + } + + /** Keeps only pairs whose raw score clears the cutoff, before normalization. */ private filterBelowThreshold(pairs: SimilarityPair[], threshold: number): SimilarityPair[] { return pairs.filter((p) => p.score >= threshold); } diff --git a/src/services/similarity/ThresholdPresets.ts b/src/services/similarity/ThresholdPresets.ts index c7e2ed7..9622183 100644 --- a/src/services/similarity/ThresholdPresets.ts +++ b/src/services/similarity/ThresholdPresets.ts @@ -1,7 +1,7 @@ /** Only a direct link bypasses this floor — tags alone can never create an edge below it. */ export const SEMANTIC_FLOOR = 0.3; -export const DEFAULT_THRESHOLD = 0.5; +export const DEFAULT_THRESHOLD = 0.7; export const TOP_K = 5;