fix(context): keep generics and JSX inside code when removing MDX tags

[JayOfTheKeyboard]

Problem

cleanMdxContent removes MDX component tags such as <AppOnly> with the pattern <\/?[A-Z][a-zA-Z]*\s*\/?>. It also runs inside fenced code blocks and inline code. There the same shape is a generic type or a JSX example, so the index stores broken code:

In the docs	In the package
createTRPCClient<AppRouter>({	createTRPCClient({
this.configService.get<DatabaseConfig>('database')	this.configService.get('database')
export type Person = Selectable<PersonTable>	export type Person = Selectable
List<String> names	List names
root.render(<App />)	root.render()

One tRPC example says "Pass AppRouter as a type parameter" right above a call that no longer has one.

I measured it by parsing four registry doc sets on 030918f and counting fenced code lines that contain a <Name> token:

Docs	Code lines with <Name>	Missing from the index on main	With this PR
NestJS (docs.nestjs.com/content)	289	289	0
tRPC (www/docs)	298	298	0
Kysely (site/docs)	44	44	0
Drizzle (src/content/docs)	88	88	0

Through the CLI (context add <trpc>/www/docs), 45 tRPC chunks contain createTRPCClient( and none contain createTRPCClient<AppRouter>. With this PR, 43 chunks have the type parameter. The 3 without it are calls the docs really write that way. initTRPC.context<Context> goes from 0 chunks to 20.

Fix

• cleanMdxContent now matches code first and keeps it whole. That covers fenced blocks (or ~~~) and inline code spans. A fence opens and closes only on a line of its own, and only the same fence closes it. So a inside a code line or a sentence does not pair with the next block. An unclosed fence runs to the end, as in CommonMark. Tags outside code are still removed.
• createSection no longer runs cleanMdxContent a second time. Markdown sections are already cleaned whole before they are split. A long code block split across parts has its opening fence in an earlier part, so the second pass stripped generics from the later parts. With only the first change, 47 of the 298 tRPC lines and 45 of the 289 NestJS lines were still missing. createSection now only trims.

The second change also reaches AsciiDoc and reStructuredText. Their sections went through cleanMdxContent only when they were long enough to split. So a long section lost its generics and a short one kept them. In the JUnit docs 7 of 16 such lines were lost, including tasks.withType<Test>(). Now none are.

Not changed

• Indented code blocks (four spaces) are not treated as code. MDX does not support them, and in .mdx files the same indentation is often nested JSX. Getting this right needs the parse tree, which is a bigger change.
• A <Name> in plain prose, outside backticks, is still removed, as before.
• One small trade-off. Drizzle has a few {/* ```sql ... ``` */} comments. The closing line ``` */} opens a fence (it would in CommonMark too), so the rest of that section counts as code. As a result, 6 of Drizzle's 6,066 bare MDX tag lines, all </Section>, now stay in the index. Main removed all of them.
• splitAtParagraphs still splits code blocks at blank lines. That is a separate chunking question.

Test

Five tests. All fail on 030918f and pass with the fix:

• build.test.ts: fenced (``` and ~~~) and inline code keep createClient<AppRouter>(), `Promise`, `` and `List`, while `` is still removed.
• build.test.ts: fences pair by line and length. This covers a inside a code line, a in a sentence, a four-backtick fence around a ``` block, and an unclosed fence.
• build.test.ts: a code block split across section parts keeps the generic in its last part.
• build.test.ts: turndown output from HTML keeps Optional<User> in <code> and List<User> in <pre><code>.
• build.test.ts: an AsciiDoc section long enough to split keeps List<String> in every part.

I broke each part of the fix in turn, and each time a test went red:

Mutation	Tests that fail
no code skipping (the old regex)	fenced/inline, fences, split block, HTML
createSection runs cleanMdxContent again	split block, AsciiDoc
no inline-code alternative	fenced/inline, HTML
backtick fences only (no ~~~)	fenced/inline
fence closed by any ``` instead of the same fence	fences
opener not anchored to a line start	fences
closer not anchored to a line of its own	fences
no unclosed-fence alternative	fences
tags never removed	fenced/inline, fences, and the existing "removes MDX component tags"

Validation

• pnpm install --frozen-lockfile, pnpm lint, pnpm build and pnpm test all pass: context 267 tests (262 before), registry 103.
• Speed: parsing 11.3 MB of Drizzle docs took 13.4 s with the fix and 19.0 s on main, because split parts are no longer cleaned twice.
• Changeset: .changeset/keep-code-generics.md (patch).

[changeset-bot]

🦋 Changeset detected

Latest commit: 3a700c6

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
@neuledge/context	Patch
@neuledge/registry	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR