Skip to content

treesitter: fixtures for the directives an unreadable construct swallows - #95

Merged
rikvanriel merged 1 commit into
facebookexperimental:mainfrom
rikvanriel:scratch/riel/preproc-recovery
Sep 29, 2026
Merged

rikvanriel merged 1 commit into
facebookexperimental:mainfrom
rikvanriel:scratch/riel/preproc-recovery

Conversation

@rikvanriel

Copy link
Copy Markdown
Contributor

include/linux/dev_printk.h defines 41 function-like macros and the index holds one of them. The cause is not the header's #ifdefs: a macro defined once per branch is indexed today. It is that
static inline __printf(3, 4) -- an attribute macro between the storage class and the return type -- is not parseable, and the resulting ERROR node does not stop at the declaration. It runs to the end of the file and swallows every directive after it, so no preproc_function_def node exists for the extraction query to match. In that header the construct is on line 71 and the span is lines 1..290.

Nothing is logged when this happens. The file indexes successfully, 16,125 dev_dbg call sites have no target, and tree-wide 1,224 function-like #defines across 435 files do not exist in the index. Measured on Linux 50d05c7c76c9 with tree-sitter 0.26.11 and tree-sitter-c 0.24.2, which is part of the measurement: 0.26.13 drops 27 more on the same tree.

No behaviour change. These are the shapes, each reduced until it is the smallest source that still reproduces what the tree does, and each test states what the parse returns rather than what it ought to:

  • the seven lines the defect reduces to;
  • dev_printk.h's shape, including a body that continues over a line and a name defined once per branch;
  • a commented-out directive inside the swallowed span, which is how a recovery that blanks what an ERROR covers can invent a definition the file never declared;
  • a #define between enum members, which is lost for a different reason and whose span does not run on, so recovery must not claim it.

Two of these hold the line for whatever recovery lands next: what parses today is a baseline it may not lose -- a prototype turned dce_hwseq.h from 59 definitions into 35 by blanking a struct whose tokens the extraction depended on -- and a macro inside a comment is never a definition, because a count that goes up is not evidence that what was gained was declared.

A fixture for the fifth shape is missing on purpose. The truncation in include/drm/intel/pciids.h, a block comment immediately before a continuation backslash, did not reproduce in two hand-written reductions: a run of three such directives extracts all of them. It needs a reduction from the file itself, and a fixture that does not reproduce the mechanism is worse than none.

Test Plan:
cargo test -- 278 tests, five new, all passing: the three losses above and the two invariants.
cargo test -- --ignored -- three new tests fail with the recovery they gate absent, one per recoverable shape, which is what makes them a gate rather than a description.
cargo clippy --all-targets -- -D warnings is clean.

`include/linux/dev_printk.h` defines 41 function-like macros and the index
holds one of them. The cause is not the header's `#ifdef`s: a macro defined
once per branch is indexed today. It is that
`static inline __printf(3, 4)` -- an attribute macro between the storage
class and the return type -- is not parseable, and the resulting ERROR node
does not stop at the declaration. It runs to the end of the file and
swallows every directive after it, so no preproc_function_def node exists
for the extraction query to match. In that header the construct is on line
71 and the span is lines 1..290.

Nothing is logged when this happens. The file indexes successfully, 16,125
`dev_dbg` call sites have no target, and tree-wide 1,224 function-like
`#define`s across 435 files do not exist in the index. Measured on Linux
50d05c7c76c9 with tree-sitter 0.26.11 and tree-sitter-c 0.24.2, which is
part of the measurement: 0.26.13 drops 27 more on the same tree.

No behaviour change. These are the shapes, each reduced until it is the
smallest source that still reproduces what the tree does, and each test
states what the parse returns rather than what it ought to:

  - the seven lines the defect reduces to;
  - dev_printk.h's shape, including a body that continues over a line and a
    name defined once per branch;
  - a commented-out directive inside the swallowed span, which is how a
    recovery that blanks what an ERROR covers can invent a definition the
    file never declared;
  - a `#define` between enum members, which is lost for a different reason
    and whose span does not run on, so recovery must not claim it.

Two of these hold the line for whatever recovery lands next: what parses
today is a baseline it may not lose -- a prototype turned dce_hwseq.h from
59 definitions into 35 by blanking a struct whose tokens the extraction
depended on -- and a macro inside a comment is never a definition, because
a count that goes up is not evidence that what was gained was declared.

A fixture for the fifth shape is missing on purpose. The truncation in
include/drm/intel/pciids.h, a block comment immediately before a
continuation backslash, did not reproduce in two hand-written reductions:
a run of three such directives extracts all of them. It needs a reduction
from the file itself, and a fixture that does not reproduce the mechanism
is worse than none.

Test Plan:
cargo test -- 278 tests, five new, all passing: the three losses above and
the two invariants.
cargo test -- --ignored -- three new tests fail with the recovery they gate
absent, one per recoverable shape, which is what makes them a gate rather
than a description.
cargo clippy --all-targets -- -D warnings is clean.
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Sep 29, 2026
@rikvanriel
rikvanriel merged commit 9e601a3 into facebookexperimental:main Sep 29, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant