`getChildren()` drops the `<` token of a type argument list when immediately followed by another `<`

[overlookmotel]

🔎 Search Terms

getChildren, addSyntheticNodes, missing token, type arguments, <<, reScanLessThanToken, LessThanLessThanToken.

🕗 Version & Regression Information

• This is the behavior in every version I tried

⏯ Playground Link

No response

💻 Code

const ts = require('typescript');

const code = `type Bar = ReturnType<<T>(x: T) => number>;`;

const sf = ts.createSourceFile(
  'a.ts',
  code,
  ts.ScriptTarget.ESNext,
  /*setParentNodes*/ true,
);

(function print(node, depth = 0) {
  console.log(
    '  '.repeat(depth) +
      `${ts.SyntaxKind[node.kind]} [${node.pos},${node.end}]` +
      ` ${JSON.stringify(code.slice(node.getStart(sf), node.end))}`,
  );

  for (const child of node.getChildren(sf)) {
    print(child, depth + 1);
  }
})(sf);

🙁 Actual behavior

The TypeReference's children skip the < at [21, 22] entirely - there is a hole between the Identifier ending at 21 and the SyntaxList starting at 22:

TypeReference    [10,42] "ReturnType<<T>(x: T) => number>"
  Identifier     [10,21] "ReturnType"
  SyntaxList     [22,41] "<T>(x: T) => number"   <-- gap at [21,22]
  GreaterThanToken [41,42] ">"

The token for first < is absent. Walking getChildren() recursively to collect leaf tokens yields a token stream that does not cover the whole source text.

🙂 Expected behavior

A LessThanToken at [21, 22], so that the children of TypeReference are contiguous and the leaf tokens cover the source. TypeScript's own AST is correct and only getChildren() disagrees with it.

Babel emits this token.

Additional information about the issue

The parser behaves correctly. It calls reScanLessThanToken(), splits the <<, and produces a correct tree with correct positions.

The problem is in the services layer: NodeObject.getChildren() -> createChildren() -> addSyntheticNodes() in src/services/services.ts. The parser does not retain a child node for a type argument list's <, so getChildren() recovers such tokens by re-scanning the text in the gaps between the children it did retain - and discards any token that overruns the gap:

function addSyntheticNodes(nodes, pos, end, parent) {
  scanner.resetTokenState(pos);
  while (pos < end) {
    const token = scanner.scan();
    const textPos = scanner.getTokenEnd();
    if (textPos <= end) {          // <-- 23 <= 22 is false, so the `<` is dropped
      // ...
      nodes.push(createNode(token, pos, textPos, parent));
    }
    pos = textPos;
    // ...

Here it is called with pos = 21, end = 22. The scanner returns a single LessThanLessThanToken ending at 23, 23 <= 22 is false, the token is thrown away, pos advances to 23, and the loop exits with the gap unfilled.

There's an asymmetry with >. The scanner emits > one character at a time and only merges on re-scan (reScanGreaterToken), so Foo<Bar<Baz>> is unaffected. < is the only direction where a plain scan produces something longer than the parser wanted.

addSyntheticNodes scans with no knowledge of the re-scan decisions the parser made, so any token the parser split can be lost this way.

Suggested fix

In addSyntheticNodes, rather than discarding an overrunning token, clip it to end or re-scan it - the scanner already exposes reScanLessThanToken() for this purpose.

Scope

In practice << is the only sequence that triggers this, because reScanLessThanToken is the only splitting re-scan reachable outside JSDoc. It is called from exactly two places, both opening a type argument list - parseTypeArgumentsOfTypeReference and parseTypeArgumentsInExpression. The other splitting re-scans (reScanAsteriskEqualsToken, reScanQuestionToken) are JSDoc-only.

Affected forms include:

type X = ReturnType<<T>(x: T) => number>;         // `<` at [19,20] missing
type Y = ReturnType <<T>(x: T) => number>;        // `<` at [20,21] missing
type Z = ReturnType/* c */ <<T>(x: T) => number>; // `<` at [27,28] missing
const a = foo<<T>(x: T) => T>();                  // `<` at [13,14] missing

Relation to existing issues

Issue for the corresponding parse error was #23996, fixed in TypeScript 3.3.1 by #26653.

That PR changed src/compiler/parser.ts and src/compiler/scanner.ts only, so the code now parses - but src/services/services.ts was not given the same treatment, which is why getChildren() still loses the token.

#47410 tracks the parser positions #26653 did not reach. This issue is the services-side counterpart.

Downstream impact

typescript-eslint builds its ESTree token list by walking getChildren(), so the token is missing from every consumer's SourceCode - any lint rule using getTokenBefore, getTokensBetween, or similar sees a different token stream depending on whether the project uses the TypeScript or Babel parser.

typescript-eslint/typescript-eslint#12820

Contribution

I'd be very happy to make a PR to fix the JS code. However, I assume this bug may also manifest in TS 7 (Go), and I don't know what policy is on "old" TS and "new" TS diverging.

Or maybe this doesn't apply, as TS 7 don't yet expose tokens to user code, so whatever it does internally is not observable?

[RyanCavanaugh]

We're not making fixes to the TS6 API, but based on the linked PR it looks like there's a TS7 equivalent problem