Fix/member header via include

[geircodes]

fix(hover,definition): resolve MEMBER through one INCLUDE hop for hover and F12

Status: this took three passes to get right. Each earlier commit fixed a real bug, but none of
them alone fixed the reported symptom (EVL:Lic giving no hover) — see the two "Follow-up" sections
below for what each subsequent test run still got wrong, and the final one for what actually closed it.

What happened

Hovering (or F12-ing) a cross-file reference — a plain global variable, or a PRE:Field-style
reference — returns nothing at all in member modules that use a common project convention:
putting the actual MEMBER('program') statement inside a small generated shim file, reached via
INCLUDE('member.clw'), instead of writing MEMBER(...) directly in every member.

Reproduced with EVL:Lic in a member file whose own INCLUDE('member.clw') contains:

MEMBER('TargetProgram')

Both lsp_hover and lsp_definition return nothing for this reference.

Root cause

TokenHelper.findMemberHeaderToken(tokens) looks for a literal MEMBER token with a
referencedFile in the tokens it's given — i.e. only the current file's own tokens. When the
real MEMBER(...) statement lives in a separately-INCLUDEd shim file instead, this returns
undefined, and every caller's "check the MEMBER parent" step is silently skipped.

Six call sites depend on this, across two services:

• MemberLocatorService.ts (hover path): global-variable cross-file lookup, PRE:field lookup, class
  member lookup, warmMemberParent.
• SymbolFinderService.ts (F12/definition path): findPrefixedField, findGlobalVariable.

Fix

Both services gained a resolveMemberHeaderToken(tokens, dir, fromFile) helper: try
TokenHelper.findMemberHeaderToken on the file's own tokens first (unchanged, fast path); if that
finds nothing, follow the file's first statement — and only its first statement — into an
INCLUDE'd shim.

This shape comes from a language rule, not a heuristic: MEMBER (or PROGRAM) must be the first
statement of a compiled module — only comments may precede it. So when a file has no literal
MEMBER, the only legal shim form is that the file's first statement is the INCLUDE carrying it
(or, transitively, the shim's own first statement is another INCLUDE — legal but rare). A MEMBER
behind any later include would not compile. Consequences:

• a non-INCLUDE first statement is an instant miss with zero file reads — the common miss-path
  (the freeze class from Perf: undeclaredVar 24s / RVD 7s on a large program file - chain-index cold build + 51 unfiltered template-equate candidates #358/Perf: missingIncludes re-reads the MEMBER parent file once per include-check (getMemberParentDocument uncached, ~4s warm) #366/Perf: startup freeze is mostly MISSING YIELDS — per-recursion timeSlicer + un-yielded include/interface/map walks + Promise.all for interactive edits #367) now costs nothing;
• each hop reads exactly one file; MAX_SHIM_HOPS (3) bounds the chained-shim case and a
  visited set stops include cycles;
• no sweep over all INCLUDE tokens — which was also a correctness hazard, since it could infer a
  parent the compiler rejects.

SymbolFinderService's copy resolves the shim include redirection-first
(resolveViaProjectRedirection, threaded with the current file for the #328 owner-project-first
behavior) with the relative-path probe as fallback — the same order
MemberLocatorService.resolveFilePath uses — so a shim reachable only via a .red mapping resolves
identically for hover and F12.

The convention logic itself lives as TokenHelper statics (findShimIncludeToken,
normalizeMemberFilename) shared by both services.

All 6 call sites (MemberLocatorService.ts x4, SymbolFinderService.ts x2) go through this instead
of calling TokenHelper.findMemberHeaderToken directly.

Testing

New test file CrossFileMemberViaInclude.test.ts, mirroring the existing direct-MEMBER pin in
CrossFilePrefixField327.test.ts: a member file INCLUDEs a shim.clw containing
MEMBER('parent'), and both F12 and hover on a PRE:Field reference must resolve to the parent
PROGRAM's field through that indirection. Plus three first-statement-rule pins, each verified red
against the sweep implementation by reverting just the source fix:

• a chained shim (first-statement INCLUDE → INCLUDE → MEMBER) resolves;
• a MEMBER behind a non-first-statement INCLUDE does not resolve — pins that no
  all-includes sweep creeps back in;
• a self-including shim terminates without resolving (cycle guard).

npm run test:server: 2390 passing / 0 failing / 4 pending, rebased onto origin/version-1.0.2
(the merged #391 three-arg isVariableLookupCandidate signature re-applied throughout).

Scope

Four files: server/src/services/MemberLocatorService.ts, server/src/services/SymbolFinderService.ts,
server/src/utils/TokenHelper.ts (shared shim-convention statics),
server/src/test/CrossFileMemberViaInclude.test.ts (new).
Found while investigating a hover-links bug report, but this is a plain cross-file symbol-resolution bug unrelated to that feature, so it goes out as its own PR.

Follow-up #1: fromFile omission in redirection (real bug, but NOT what was blocking EVL:Lic)

Testing the fix above with a SolutionManager loaded surfaced a second issue:
resolveMemberHeaderToken's call to resolveFilePath(inc.referencedFile!, fromDir) (and several
pre-existing parent-path resolutions in the same file) omitted the third fromFile argument. Per the
#328 "owner-project-first" contract on resolveFilePath/resolveViaProjectRedirection: without
fromFile, redirection can't identify which project is asking and falls back to an unscoped walk
across every project's redirection parser, returning the first match in solution order rather than the
one actually owned by the file being hovered — a real correctness gap in a multi-project solution.

Fixed by threading the current file's path through as fromFile everywhere it was missing. This is a
legitimate, worth-keeping fix, but turned out not to be what was blocking the EVL:Lic repro — the
test solution used here has only one project, so there was no cross-project collision actually
happening. Diagnosed with a standalone test that initially bypassed the resolver's redirection path
entirely — it had no SolutionManager loaded, so it never even exercised the code path it was meant to
test. Lesson for follow-up #2.

Follow-up #2: extension-less MEMBER targets (this is what was actually blocking EVL:Lic)

EVL:Lic still gave no hover after follow-up #1. Rewriting the test to actually load a
SolutionManager (SolutionManager.create('...TargetProgram.sln')) before calling the resolver —
matching the production code path instead of accidentally exercising the no-solution fallback —
surfaced the answer directly:

resolveViaProjectRedirection('TargetProgram', ...)      => null
resolveViaProjectRedirection('TargetProgram.clw', ...)  => <project dir>\TargetProgram.clw

Clarion MEMBER('TargetProgram') conventionally omits the .clw extension (the compiler infers
it) — completely idiomatic, not an edge case. referencedFile is stored exactly as written by the
tokenizer. resolveViaProjectRedirection's lookup matches by extension mask (a .red file's *.clw = ... line), so an extension-less name never matches any rule; SolutionManager.findFileWithExtension
matches by exact source-file basename, which also misses. Both silently return nothing for the
idiomatic, extension-less form — which is presumably the common case in most Clarion codebases, not a
corner case.

This is also why the tokenizer's structurePrefix propagation for nested FILE,PRE(x) →
RECORD,PRE() → field was never actually the problem (a concern raised earlier in this
investigation) — the chain never got far enough to exercise it; it was dying one step earlier, on the
MEMBER→PROGRAM hop itself.

Fixed with a normalizeMemberFilename() helper in both services (append .clw only when there's no
extension already — deliberately scoped to MEMBER targets only, since INCLUDE/LINK/MODULE targets
always carry an explicit extension already) applied at every MEMBER-target resolution call site.
Updated CrossFileMemberViaInclude.test.ts's shim fixture from MEMBER('parent.clw') to
MEMBER('parent') — the explicit-extension form the original fixture used would never have caught
either of these bugs.

Verified end-to-end with a SolutionManager loaded:
findPrefixFieldTokenInChain('Evl', 'Lic', ...) now resolves to the dictionary include file, matching
the expected hover output. npm run test:server: all passing, 0 failing, verified again in the isolated
worktree.

Follow-up #3: field name collides with its own enclosing structure's name

EVL:Lic and EVL:Txt now resolved correctly, but EVL:Evl (a field named the same as its enclosing
FILE) showed Evl — UNKNOWN pointing at the FILE's own declaration line instead of the field. Two
independent bugs, both in the same small area:

1. findPrefixFieldInTokens matched by structurePrefix alone. StructureProcessor stamps
   structurePrefix on the declaring structure token itself, not just its fields, so a field that
   happens to share its structure's name matches the structure's own declaration token first (it
   appears earlier in document order).
2. A second, independent tokenizer quirk feeds the same symptom class: DocumentStructure pushes a
   structure onto its stack the moment the structure's own keyword token is seen, so a later
   Variable/StructurePrefix-type token on that same declaration line — e.g. the GLOB:Owner
   argument of OWNER(GLOB:Owner), or the Evl argument inside PRE(Evl) itself — gets mistagged
   isStructureField=true with the structure's own prefix too, even though it's an attribute
   argument, not a real field.

Fixed both with a targeted selector in findPrefixFieldInTokens — prefer a field match that is
isStructureField AND declared on a line strictly after its structureParent's own line — rather
than touching DocumentStructure's core structure-stack walker, which many other features
(completion, diagnostics, F12) depend on.

Also addressed the original ergonomic question that started this whole thread: a PRE:Field hover
(e.g. EVL:Lic) resolves through the same code path as a true global variable and showed a generic
"🌍 Global variable" label — losing exactly the context needed to tell a real global apart from a
structure field reached via its PRE prefix. Now labeled "🔷 Evl field", matching the "X Field:"
wording StructureFieldResolver already uses for dot-notation access (Evl.Lic) to the same field.

New test: PrefixFieldNameCollidesWithStructure.test.ts, reproducing both the name-collision and the
same-line-attribute-argument shapes directly. npm run test:server: all passing (2344 in the isolated
worktree's older base + the 2 new tests), 0 failing.

Confirmed via the test suite: EVL:Lic, EVL:Txt, and EVL:Evl all hover correctly now, each
showing exactly one result — EVL:Lic is a single lexical token (Clarion colon-prefix notation), so
there is exactly one thing to hover, unlike Evl.Lic (two separate dot-joined tokens, each
independently hoverable, which is why that form can show two different results depending on which half
the cursor is on). That's expected, not a gap — noted here so a future pass doesn't try to "fix" it.

[msarson]

Reviewed — the resolution logic is sound (one INCLUDE hop, no recursion, normalizeMemberFilename correctly handles the extension-less MEMBER('parent') form) and the tests are meaningful.

Holding on a perf concern on a branch this repo is historically sensitive to:

• On the hover/F12 miss-path, when a file has no literal MEMBER, the fallback loops over all INCLUDE tokens, loadDocument/readFile+tokenizing each until it finds a MEMBER or exhausts them. Since includes rarely contain a MEMBER, a miss sweeps and tokenizes every included file — an uncapped cold cross-file sweep on exactly the path we spent Perf: undeclaredVar 24s / RVD 7s on a large program file - chain-index cold build + 51 unfiltered template-equate candidates #358/Perf: missingIncludes re-reads the MEMBER parent file once per include-check (getMemberParentDocument uncached, ~4s warm) #366/Perf: startup freeze is mostly MISSING YIELDS — per-recursion timeSlicer + un-yielded include/interface/map walks + Promise.all for interactive edits #367 de-freezing. Please cap the include count / bail on large token counts / add a cheap early-out.
• Resolver asymmetry: MemberLocatorService resolves the shim via redirection-aware resolveFilePath, but SymbolFinderService uses plain path.resolve + fs.existsSync (no redirection). A shim reachable only via .red would resolve for hover but not F12. Please align them.

Note this also textually conflicts with #391 (now merged) on MemberLocatorService.ts — you'll need to rebase and re-apply the 3-arg isVariableLookupCandidate signature. Address the sweep + asymmetry and I'll re-verify against the real ap1.sln perf rig before merging.

[msarson]

Logic is sound — holding on a perf concern: an uncapped cold tokenize-all-includes sweep on the hover/F12 miss-path (the freeze class #358/#366/#367 fixed), plus a resolver redirection asymmetry. Will also need a rebase vs merged #391. Details above.

[msarson]

👋 Housekeeping: 1.0.1 is now released (live on the Marketplace), so version-1.0.1 has been merged to master and the branch removed. I've retargeted this PR to version-1.0.2 — the new active development branch.

This PR is still held on the review feedback above — no change to that. When you get a chance to revise, please rebase your branch onto version-1.0.2 and push. Thanks for the contribution and your patience! 🙏

[geircodes]

Reply for PR #388 (fix/member-header-via-include)

Revised and rebased onto version-1.0.2 (the merged #391 three-arg isVariableLookupCandidate signature is re-applied throughout).

The perf concern is gone at the root, not capped. A Clarion language rule makes the all-includes sweep unnecessary — and wrong: MEMBER (or PROGRAM) must be the first statement of a compiled module; only comments may precede it. So when a file has no literal MEMBER, the only legal shim form is that the file's first statement is the INCLUDE carrying it (or, transitively, the shim's own first statement is another INCLUDE — legal but rare). A MEMBER behind any later include would not compile, so the sweep wasn't just uncapped cold I/O on the miss-path — it could infer a parent the compiler rejects.

resolveMemberHeaderToken is now a first-statement walk:

• first statement isn't an INCLUDE → instant miss, zero file reads (the common miss now costs nothing);
• each hop reads exactly one file;
• MAX_SHIM_HOPS (3) bounds the chained-shim case, a visited set stops include cycles.

Resolver asymmetry fixed. SymbolFinderService's copy now resolves the shim include redirection-first (resolveViaProjectRedirection, threaded with the current file for the #328 owner-project-first behavior) with the relative-path probe as fallback — the same order MemberLocatorService.resolveFilePath uses — so a shim reachable only via a .red mapping resolves identically for hover and F12.

The convention logic itself (findShimIncludeToken, normalizeMemberFilename) now lives as TokenHelper statics shared by both services. (#395 carries the identical TokenHelper hunk, so the two merge cleanly in either order.)

Tests (all red against the previous sweep implementation, verified by reverting just the implementation):

• chained shim (INCLUDE → INCLUDE → MEMBER) resolves;
• a MEMBER behind a non-first-statement INCLUDE does not resolve — pins that no sweep creeps back in;
• a self-including shim terminates without resolving (cycle guard).

Full suite: 2390 passing.