ADFA-5296: Javadoc mode for kdoc-to-json, JDK API docs as JSON, and a Pebble HTML renderer - #31
Open
alexmmiller wants to merge 6 commits into
Open
ADFA-5296: Javadoc mode for kdoc-to-json, JDK API docs as JSON, and a Pebble HTML renderer#31alexmmiller wants to merge 6 commits into
alexmmiller wants to merge 6 commits into
Conversation
Setting "javadoc-mode": true in the plugin config replaces the Dokka-shaped
JSON output with JSON that mirrors the api/ tree the javadoc tool produces:
the same file layout, the same page sections, and the same member anchors.
Only JSON is written (plus javadoc's plain-text element-list); rendering
stays the downstream template engine's job.
Layout: <module>/<pkg/as/path>/<Outer.Nested>.json, package-summary.json and
module-summary.json, plus the global index files (index, allclasses-index,
allpackages-index, deprecated-list, constant-values, index-files/index-N,
element-list). Module directories appear only for a genuinely multi-module
run, matching javadoc's own modular/non-modular split. Links between pages
are relative to the page they appear on, as javadoc's are.
New files:
javadoc/JavadocPaths javadoc's path scheme and erased member anchors
(<init>(double,double), toArray(java.lang.Object[]))
javadoc/JavadocModelIndex one whole-run pass building the type graph in both
directions, so the hierarchy closures and
inherited-member groups a javadoc page needs are
precomputed rather than walked per page
javadoc/JavadocDocs block-tag extraction and doc-comment rendering
javadoc/JavadocDtos the page DTOs, kept separate from the Dokka-shaped
hierarchy so neither constrains the other
javadoc/JavadocMapper Dokka model -> javadoc pages
javadoc/JavadocRenderer writes the tree
JsonFilters omitFields/omitNulls, now shared by both renderers
Reconciling Dokka's model with javadoc's view needed four corrections:
- Dokka merges a private Java field and its accessors into one Kotlin-style
property; that is unfolded back so getWidth() is a method and the private
field is undocumented, as javadoc has it.
- Dokka reports an interface default method only as "not abstract"; the
default keyword is recovered from that.
- Dokka files @PARAM <U> under "<U>", not "U".
- Constant values arrive wrapped as IntegerConstant(value=4) and are
unwrapped to their literals.
The config key is spelled javadoc-mode, which needs both @JsonProperty and
@SerialName: Dokka parses pluginsConfiguration with Jackson, while
JsonRenderer's manual fallback uses kotlinx.serialization, and either
annotation alone leaves the key silently ignored on one of the two paths.
Tests: tests/test_javadoc_mode.sh (58 assertions) drives a new Java example,
examples/example-java-library, covering the layout, class/interface/enum/
annotation/exception pages, override and inherited-member derivation, the
global index files, and cross-page link scoping. Full suite: 15/15 scripts
pass. README section 10 documents the mode, the recommended
documentedVisibilities setting for javadoc parity, and the limitations that
come from Dokka's model (no JPMS, no annotation-element defaults, no records).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds scripts/java/, which reproduces the JDK's own api/ tree (the docs in
SourceDocs/JavaDocs/html/api) as JSON, and the JPMS support in the plugin that
the module-per-directory layout depends on.
Measured on JDK 17: a full run takes ~90 seconds and writes 4,988 JSON files.
Against the official docs it matches exactly on structure -- 60/60 modules,
224/224 packages, 4,672/4,672 types, none missing and none extra -- and 4,305
of those 4,672 types (92%) match javadoc's member anchors exactly. The 367
that differ are understood and documented in README section 11; neither
direction is a missing page.
scripts/java:
stage_jdk_sources.py unpacks lib/src.zip and keeps only what javadoc
documents. javadoc's rule turns out to be exact: a
package appears in api/ iff its module exports it
unqualified. For java.base that is 53 exports and 53
documented packages, nothing left over either side.
jdk-docs/ a Dokka project registering each module directory as
a source root; nothing is compiled, only analysed
build-jdk-json-docs.sh driver for the three steps
compare_with_javadoc.py parity checker: modules, packages, types, and (with
--members) the member anchors of every type
Plugin changes:
- JPMS modules are read from module-info.java, which Dokka's model does not
carry at all. That supplies the <module>/<package>/<Class>.json layout, the
module page's requires/exports/opens/uses/provides, and its description.
A source root holding a module-info.java *is* a module root, so nothing
else is affected. Source roots arrive as directories or as expanded file
lists depending on how Dokka was configured; both are handled.
- {@inheritdoc} is resolved by the plugin instead of by Dokka. Dokka's own
resolver (InheritDocTagResolver.resolveThrowsTag ->
PsiElementToHtmlConverter.toInheritDocHtml) recurses without bound on much
of the JDK and dies with a StackOverflowError; a larger stack only buys
time (-Xss64m fails after 42s, -Xss512m after 3m28s). The staging script
rewrites the tag to an inert marker and the mapper resolves it by walking
the same supertype chain "Overrides:" and "Specified by:" come from. All
3,214 occurrences across the JDK still resolve, and a *missing*
@param/@return/@throws is now inherited too, as javadoc does.
- Member anchors erase type arguments. Dokka builds a Java DRI from the PSI
canonical text, which keeps them, so addAll(java.util.Collection<? extends
E>) has to become addAll(java.util.Collection) to match a real javadoc
build. This alone took member parity from 80% to 92%.
Tests: test_javadoc_mode.sh grows to 65 assertions, covering module-info
parsing via a module-info.java added to examples/example-java-library. Full
suite 15/15 scripts pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… module tables Re-adds the whole pebble-renderer/ tree. It was committed in 327c485a, but the later squash to ec3633f was made after a `git reset`, which had turned those files back into untracked ones -- so `commit -a` skipped them and the branch lost all 25 files while they survived on disk. Nothing else was affected; the plugin-side work was carried over intact. Then closes the gaps found by diffing the rendered HTML against SourceDocs/JavaDocs/html/api section by section: module-graph.svg Missing on all 60 module pages -- the bug that started this. Generated from the JSON's requires. All 60 now have node sets identical to the originals (57/60 also match edge counts; graphviz applies a transitive reduction we don't). The rules were derived by testing candidates against all 60 rather than assumed, and both are narrower than they look: the graph is the `requires transitive` closure plus java.base -- a plain `requires` does not propagate readability, and java.base is drawn although no module-info.java declares it. inheritedNestedTypes Never populated, ~700 groups. Dokka does not copy a supertype's nested types down the way it does fields and methods, so there is no InheritedMember to read and the groups are walked out of the hierarchy instead. indirectExports javadoc's "Indirect Exports" table, 16 module pages: the exporting modules in the readability closure. 16/16 exact. indirectRequires javadoc's "Indirect Requires" table, 3 module pages: that closure minus the direct requires. 3/3 exact. tag labels ~2,400 notes rendered under their source name. @APinote now reads "API Note:", @implSpec "Implementation Requirements:", and so on. The data was always in `tags`; only the heading was wrong. Likewise "Enclosing interface:" now follows the enclosing type's kind. Module pages now have no section the originals have. Structural parity is unchanged at 60/60 modules, 224/224 packages, 4,672/4,672 types, and 453,862 links and images resolve at 99.88%. Two gaps are not reproducible from this source and are recorded as such: doc-files/ (93 files; javadoc copies them from the JDK build repo, and src.zip does not ship them) and serialVersionUID (private fields, shown only on serialized-form.html). Page kinds outside this pipeline -- class-use/, package-use, the tree pages, serialized-form, search -- are listed in README section 11 rather than silently absent. Full suite: 15/15 scripts pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…bles
Two empty columns, both visible on the first page you land on:
- The overview listed all 60 modules with a blank Description. moduleSummary
was reading the Dokka module's documentation, which is empty for a JPMS
module; the real description lives in module-info.java, as it does on the
module page itself.
- A module page's Exports (and Opens) table had a blank Description for every
package. JdModuleExport now carries the exported package's summary
sentence, which is what javadoc shows in that column.
Structural parity unchanged: 60/60 modules, 224/224 packages, 4,672/4,672
types; 453,875 links and images resolve at 99.88%. Full suite 15/15.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…he originals
Started from the overview listing all 224 packages as well as the 60 modules,
where javadoc lists only the modules, then diffed every other page type rather
than assuming that was the only case.
index.html Modules only. javadoc's overview lists packages only
for a non-modular run; the full package list already
has its own page.
constant-values.html 3,138 of the JDK's 3,463 constants -> 3,458. Every one
of the 325 absent was inherited from an *undocumented*
supertype: java.util.jar.JarEntry's 40 CEN*/END*
constants come from the package-private
java.util.zip.ZipConstants.
class pages Same root cause, and it also accounted for ~480 missing
member anchors. A member inherited from a type this run
does not document is now reported as declared, which is
what javadoc does -- an "inherited from" group pointing
at a page that does not exist is a dead end. Member
anchors 4,305 -> 4,327 of 4,672 types.
deprecated-list.html Section headings were the raw JSON keys ("classes",
"enumConstants"); now javadoc's titles, plus a contents
list.
package pages "Related Packages" was missing entirely. Present on 206
pages, 181 identical to the originals. The rule is
parent + children + siblings, but only while the result
stays at five or fewer -- that condition is javadoc's,
not an invention: java.nio.channels lists its siblings
while java.util.concurrent and java.lang.annotation
list none, because java.util and java.lang have too
many children. Five reproduces 181/190; no cut-off
reproduces 95.
allclasses-index Listed all 4,672 types; javadoc indexes the public API,
so the 167 protected nested types are now left out.
They keep their pages, reachable from their enclosing
class, exactly as in the originals.
JdTypeSummary gains `modifiers` and JdPackagePage gains `relatedPackages`, so
both decisions are made from data in the JSON rather than guessed in a template.
Two counts still differ, both small and both showing more rather than less:
allclasses-index has 4,506 against javadoc's 4,402, and the A-Z index 54,248
against 55,483. Neither reduced to a rule that held across all 60 modules, so
they are left alone rather than tuned to fit.
test_javadoc_mode.sh's annotation-element assertion was asserting the old
behaviour and is updated: in the small example java.lang.Object is not
documented, so its methods are pulled up there, while the JDK build documents
java.lang and keeps them as inherited groups.
Structural parity unchanged: 60/60 modules, 224/224 packages, 4,672/4,672
types; 455,440 links and images at 99.88%. Full suite 15/15, 66 javadoc-mode
assertions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
An opt-in Javadoc mode for the
kdoc-to-jsonDokka plugin, the tooling to run it over the JDK,and a Pebble renderer that turns its JSON into browsable HTML.
Setting
"javadoc-mode": truereplaces the plugin's Dokka-shaped output with JSON that mirrors theapi/tree thejavadoctool produces — same file layout, same page sections, same memberanchors. Only JSON is written (plus javadoc's plain-text
element-list); rendering stays adownstream concern.
Why
SourceDocs/JavaDocs/html/apiis the JDK 17 API documentation we ship, and it only exists asscraped HTML. This makes it reproducible as structured JSON, which is what the documentation
database and the Pebble templating in
WebServer.ktactually want to consume.Results against the official docs
Measured on JDK 17 against
SourceDocs/JavaDocs/html/api:What's in it
kdoc-to-json—javadoc-modeconfig flag plus thejavadoc/package: path/anchor scheme,a whole-run type index (hierarchy closures, inherited-member groups), doc-tag extraction, the
page DTOs, mapper and renderer. JPMS structure is read from
module-info.java, which Dokka'smodel has no notion of.
scripts/java/—stage_jdk_sources.py(unpackssrc.zip, keeps what javadoc documents),a Dokka project,
build-jdk-json-docs.sh, andcompare_with_javadoc.pyfor parity checking.pebble-renderer/— templates following the official page structure, a Java program thatinstantiates them, and
module-graph.svggeneration.Notes for a reviewer
Rules were derived empirically, not assumed. Several turned out narrower than they look, and
each was checked against all 60 JDK module pages rather than inferred from one example:
exportsit unqualified — forjava.basethat is53 exports and 53 documented packages, nothing left over either side.
requires transitiveclosure plusjava.base; a plainrequiresdoes not propagate readability, and
java.baseis drawn although nomodule-info.javadeclaresit. 60/60.
fewer. That cut-off reproduces 181 of 190 pages; no cut-off reproduces 95.
Two Dokka bugs are worked around, both found by running the JDK through it and both documented
in README §11:
{@inheritDoc}resolution recurses without bound on much of the JDK. A bigger stack only buystime (
-Xss64mdies at 42 s,-Xss512mat 3m28s). The staging script rewrites the tag to aninert marker and the plugin resolves it itself, walking the same supertype chain javadoc walks —
all 3,214 occurrences still resolve.
out as
addAll(java.util.Collection<? extends E>)where javadoc erases them. Fixing that alonemoved member parity from 80 % to 92 %.
Known gaps, all recorded in README §11 rather than left silent:
doc-files/(93 files, 251 links) — javadoc copies these from the JDK's build repository;src.zipdoes not ship them, so no pipeline readingsrc.zipcan produce them.class-use/,package-use, the tree pages,serialized-form,search. These are cross-reference/navigation rather than API data.allclasses-index4,506 vs 4,402; A-Z index54,248 vs 55,483). Neither reduced to a rule that held across all 60 modules, so they are left as
they are rather than tuned to fit.
Environment: the plugin is pinned to Kotlin 1.9.24, which cannot run on JDK 26 at all. Use a
JDK ≤ 21 to build. Dokka generates in a worker process, so
org.gradle.jvmargsdoes not size it —dokkaGeneratorIsolationinscripts/java/jdk-docs/build.gradle.ktsdoes.Testing
tests/run_all.sh— 15/15 scripts pass, including a newtest_javadoc_mode.shwith 66assertions driving a new Java example (
examples/example-java-library). No regressions in theexisting Kotlin-facing tests.
🤖 Generated with Claude Code