Skip to content

feat(pkl): extract calls, branches, throws and decorators - #1426

Open
tthayer wants to merge 2 commits into
DeusData:mainfrom
tthayer:feat/pkl-call-extraction
Open

feat(pkl): extract calls, branches, throws and decorators#1426
tthayer wants to merge 2 commits into
DeusData:mainfrom
tthayer:feat/pkl-call-extraction

Conversation

@tthayer

@tthayer tthayer commented Aug 3, 2026

Copy link
Copy Markdown

Problem

Pkl was already wired end-to-end — grammar vendored, CBM_LANG_PKL, extension mapping, import edges — but the spec was shallow. pkl_call_types was empty_types, so on a real-world 172-file Pkl corpus the graph had 0 CALLS edges while imports worked fine.

branching_node_types, throw_node_types and decorator_node_types were empty too, so Pkl contributed nothing to complexity metrics.

Why this needs a custom extractor

Pkl has no dedicated call node. unqualifiedAccessExpr and qualifiedAccessExpr are the same node whether they are a call or a bare property read — the only discriminator is an argumentList child:

unqualifiedAccessExpr 'helper(a)'     unqualifiedAccessExpr 'host'
  identifier 'helper'                   identifier 'host'
  argumentList '(a)'                  <-- no argumentList

extract_pkl_callee gates on that child. It is dispatched from extract_callee_name with an unconditional return, not via extract_callee_lang_specific: returning NULL from the latter falls through to the generic "first identifier child" fallback, which would mint a CALLS edge for every property read in every Pkl file.

Callee resolution:

source callee
helper(a) helper
utils.fallback(a) utils.fallback (module-qualified; cbm.c shortens to the last dotted segment)
s.trim().toLowerCase() toLowerCase (a receiver that is itself a call has parens in its text, so it is not prefixed)
new Server { ... } Server (links to the class def)

Other spec slots filled

All node names confirmed against real parse trees from a probe linking only the Pkl grammar, not guessed:

  • class: + typeAlias
  • import: + importGlobClause, importExpr
  • branch: ifExpr, whenGenerator, forGenerator
  • throw: throwExpr
  • decorator: annotation

forGenerator is also registered in cbm_is_loop_node_type so for (x in xs) counts toward loop-nesting depth. The name is Pkl-unique, so no other grammar collides.

Validation

Repro battery — the Pkl test moves from the structural battery (dims 1-5) to the full callable battery (dims 1-8), plus an inline negative assertion that bare property reads do not become CALLS edges. repro_grammar_config_pkl PASS.

Full suite: 318 passed, 6 failed, 2 skipped — byte-identical to the pre-change baseline (verified by stashing and re-running). The 6 failures are pre-existing and unrelated (repro_issue480/514/546/581, repro_issue409, repro_ts_inherited_method).

Real corpus — 172 Pkl files of production infrastructure config, indexed clean (0 skipped, 0 parse failures):

before after
CALLS edges 0 367

Split: 270 to Function, 97 to Class (constructors). ~2.1 calls/file — proportionate; a broken gate would have produced tens of thousands.

Quality was spot-checked rather than inferred from the count: sampled Function callees are genuine function definitions invoked with arguments, and a bare property read that shares its name with one of those functions correctly produced no edge — the exact case the argumentList gate exists to catch.

Note for reviewers

The repro fixture deliberately keeps every call site inside a method body, because dim 7 treats Module-sourced in-body calls as the enclosing-func gap. Real-world Pkl does call at module level (many of the 97 constructor edges are such sites), so this is a fixture constraint, not a claim about the language.

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

  • It will not be closed for inactivity. No stale bot touches pull requests here.
  • It may still sit a while before a human reads it. That is on us, not on you.
  • Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

  • Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
  • Get CI green, or say which failures you believe are pre-existing.
  • Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
  • Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.

@tthayer
tthayer force-pushed the feat/pkl-call-extraction branch 2 times, most recently from f9cc2b9 to 54a64c8 Compare August 3, 2026 18:04
Pkl was wired end-to-end but shallow: on a real-world 172-file Pkl corpus
the graph had 386 IMPORTS edges and 0 CALLS edges, because pkl_call_types
was empty_types.

Pkl has no dedicated call node. `unqualifiedAccessExpr` and
`qualifiedAccessExpr` are the same node whether they are a call
(`helper(a)`) or a bare property read (`host`) — the only discriminator is
an `argumentList` child. extract_pkl_callee gates on that child, and is
dispatched from extract_callee_name with an unconditional return: falling
through to field-based or generic first-identifier resolution would mint a
CALLS edge for every property read in every Pkl file, since a bare access
expr's first child is an identifier.

Callee resolution:
  helper(a)               -> "helper"
  utils.fallback(a)       -> "utils.fallback"  (module-qualified; cbm.c
                             shortens to the last dotted segment)
  s.trim().toLowerCase()  -> "toLowerCase"     (a receiver that is itself a
                             call has parens in its text and is not prefixed)
  new Server { ... }      -> "Server"          (links to the class def)

Also fills the other empty slots in the Pkl spec, all confirmed against
real parse trees rather than guessed: typeAlias (class), importGlobClause /
importExpr (import), ifExpr / whenGenerator / forGenerator (branch),
throwExpr (throw), annotation (decorator). forGenerator is additionally
registered in cbm_is_loop_node_type so `for (x in xs)` counts toward
loop-nesting depth; the name is Pkl-unique so no other grammar collides.

The Pkl repro test moves from the structural battery (dims 1-5) to the
full callable battery (dims 1-8) and adds an inline negative assertion
that bare property reads do NOT become CALLS edges — the regression the
argumentList gate exists to prevent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Tony Thayer-Osborne <tony.thayerosborne@conductorone.com>
@tthayer
tthayer force-pushed the feat/pkl-call-extraction branch from 54a64c8 to 4e8d01d Compare August 3, 2026 18:29
@tthayer
tthayer marked this pull request as ready for review August 3, 2026 19:47
@tthayer
tthayer requested a review from DeusData as a code owner August 3, 2026 19:47
The Windows shard failed Step 0i (parallel suite scheduler contract) with
"cleanup failed: hang_after_summary: leader exited leaving live
descendants" while its sibling shard passed the same contract. Nothing in
the wave was actually leaking a process.

run-test-wave.py passed --kill-grace as the subprocess timeout for the two
external Windows helpers: taskkill.exe /T /F and the powershell.exe
Get-CimInstance descendant probe. Those are different quantities.
kill_grace budgets how long a doomed process may take to die; the helpers
also have to pay process spawn plus, for PowerShell, CIM startup. The
contract fixtures run with --kill-grace 1, and one second is not reliably
enough to launch either helper on a loaded runner.

The two timeouts then compounded. A timed-out taskkill is reported as
"could not prove process-tree cleanup", which raises out of the wave loop;
the finally-block cleanup re-enters with the leader already dead, so it
falls to the descendant probe, which times out on the same one-second
budget and takes its "cannot prove absence -> assume the worst" branch.
Phantom descendants, red wave.

Give helper invocations their own floor, max(kill_grace, 30). kill_grace
still governs every actual death wait, so nothing fails open: the
timeout-race contract still refuses with rc=2 over a genuinely surviving
descendant, now because PowerShell answered rather than because it timed
out. The production path already passed --kill-grace 15 and is unchanged
in behaviour.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Tony Thayer-Osborne <tony.thayerosborne@conductorone.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant