Skip to content

treesitter: say which files gave up their swallowed directives - #97

Merged
rikvanriel merged 1 commit into
facebookexperimental:mainfrom
rikvanriel:scratch/riel/preproc-visibility
Sep 29, 2026
Merged

rikvanriel merged 1 commit into
facebookexperimental:mainfrom
rikvanriel:scratch/riel/preproc-visibility

Conversation

@rikvanriel

Copy link
Copy Markdown
Contributor

Reading the macros an unreadable construct swallows is silent, in both directions. A file that gains 35 of them looks exactly like a file that gains none, and a file recovery refuses to read looks like a file with nothing to read -- which is the same silence that let the defect stand: one logging header held 41 directives and contributed one row, and nothing anywhere said so.

Say it, once per file, and only where there is something to say.

  • A file whose swallowed definitions were read names them by count.
  • A file where the blanked parse no longer reads a definition the file's own parse does is refused whole, and says that it was: whatever else that parse found is not trustworthy, so the swallowed macros stay missing and the file is worth a look rather than a shrug.
  • A file where every definition the blanked parse found sits where the file writes a comment or a string says that too. Refusing to invent a macro is a different fact about a file than having none to recover, and a count of what was refused rides along on the line for a file that recovered something as well.

Silence otherwise, because the population that enters recovery is an order of magnitude larger than the population it helps: on Linux 28a2bc7211da, 1,639 files are parsed a second time and 139 of them gain anything. Logging the attempt would bury the result.

Counting what was refused needed one fix behind it. The rows a second reading finds were deduplicated to one per name before the file was asked which of them are comments, and the longer body wins that contest: a #define inside a comment can therefore take the name of a real swallowed definition below it, which both loses a macro the file defines and reports the loss as a refusal. Ask the file first, then deduplicate, and a refused name is exactly a name no row of the file declares.

Test Plan:
Three tests read what the analyzer says through a thread-local subscriber, so they hold the log as a contract rather than describing it: the recovered count appears with the file, a refusal appears with its count, and a file the parser reads whole says nothing at all.

cargo test -- 294 tests, twenty-one in this module, none ignored. The phantom-takes-a-name test was run against a build that deduplicates before asking the file, and fails there.
cargo clippy --all-targets -- -D warnings is clean.

Volume over Linux 28a2bc7211da, reading every *.c and *.h with SEMCODE_DEBUG=info -- these go to stderr, and the default filter is error, so a plain run still shows none of them: 154 lines. 139 report what they recovered and their counts sum to 681, the tree-wide gain; 15 report a refusal, among them the AMD display header whose second reading drops 24 definitions the first one reads, include/linux/bpf.h and drivers/char/random.c. Not one file on this tree reports a refused definition, either alone or beside a gain: those two lines have fixtures and no instance here.

That the recovery lines are invisible by default is worth its own decision later. A count of what a whole run recovered and refused belongs in the indexer's summary rather than behind a debug filter, and that needs the analyzer to return what it did instead of only saying it.

Reading the macros an unreadable construct swallows is silent, in both
directions. A file that gains 35 of them looks exactly like a file that gains
none, and a file recovery refuses to read looks like a file with nothing to
read -- which is the same silence that let the defect stand: one logging
header held 41 directives and contributed one row, and nothing anywhere said
so.

Say it, once per file, and only where there is something to say.

  - A file whose swallowed definitions were read names them by count.
  - A file where the blanked parse no longer reads a definition the file's own
    parse does is refused whole, and says that it was: whatever else that
    parse found is not trustworthy, so the swallowed macros stay missing and
    the file is worth a look rather than a shrug.
  - A file where every definition the blanked parse found sits where the file
    writes a comment or a string says that too. Refusing to invent a macro is
    a different fact about a file than having none to recover, and a count of
    what was refused rides along on the line for a file that recovered
    something as well.

Silence otherwise, because the population that enters recovery is an order of
magnitude larger than the population it helps: on Linux 28a2bc7211da, 1,639
files are parsed a second time and 139 of them gain anything. Logging the
attempt would bury the result.

Counting what was refused needed one fix behind it. The rows a second reading
finds were deduplicated to one per name before the file was asked which of
them are comments, and the longer body wins that contest: a `#define` inside a
comment can therefore take the name of a real swallowed definition below it,
which both loses a macro the file defines and reports the loss as a refusal.
Ask the file first, then deduplicate, and a refused name is exactly a name no
row of the file declares.

Test Plan:
Three tests read what the analyzer says through a thread-local subscriber, so
they hold the log as a contract rather than describing it: the recovered count
appears with the file, a refusal appears with its count, and a file the parser
reads whole says nothing at all.

cargo test -- 294 tests, twenty-one in this module, none ignored. The
phantom-takes-a-name test was run against a build that deduplicates before
asking the file, and fails there.
cargo clippy --all-targets -- -D warnings is clean.

Volume over Linux 28a2bc7211da, reading every `*.c` and `*.h` with
`SEMCODE_DEBUG=info` -- these go to stderr, and the default filter is `error`,
so a plain run still shows none of them: 154 lines. 139 report what they
recovered and their counts sum to 681, the tree-wide gain; 15 report a
refusal, among them the AMD display header whose second reading drops 24
definitions the first one reads, `include/linux/bpf.h` and
`drivers/char/random.c`. Not one file on this tree reports a refused
definition, either alone or beside a gain: those two lines have fixtures and
no instance here.

That the recovery lines are invisible by default is worth its own decision
later. A count of what a whole run recovered and refused belongs in the
indexer's summary rather than behind a debug filter, and that needs the
analyzer to return what it did instead of only saying it.
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Sep 29, 2026
@rikvanriel
rikvanriel merged commit 0ddc62e into facebookexperimental:main Sep 29, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant