Skip to content

Issue/2183/2 no observable change - #2187

Merged
davidgamez merged 6 commits into
MobilityData:masterfrom
CarloMendola:issue/2183/2-no-observable-change
Oct 14, 2026
Merged

davidgamez merged 6 commits into
MobilityData:masterfrom
CarloMendola:issue/2183/2-no-observable-change

Conversation

@CarloMendola

Copy link
Copy Markdown
Contributor

Summary:

Second of the six PRs #2183 is split into, in the grouping requested here. Based directly on master and independent of the other PRs in the series. Three changes grouped because none of them changes anything observable: same reports, same bytes, same notices — they only stop holding things the run does not need.

1. Close the ZipInputStream used to look for a subfolder. GtfsInput.hasSubfolderWithGtfsFile and createFromUrlInMemory opened a ZipInputStream over the archive and never closed it. Each holds a file descriptor and an Inflater, which owns a zlib stream in native memory released only when the cleaner eventually runs. Harmless for one CLI invocation; a long-running service validating one feed after another leaks descriptors and resident memory. Both are now in try-with-resources.

2. Cache the notice documentation comments per notice class. NoticeSchemaGenerator.loadComments opens a classpath resource and parses JSON on every call, and NoticeView calls it once per notice. Generating the HTML report for a feed with many notices repeated the same resource read and JSON parse thousands of times and retained one identical NoticeDocComments instance — each with its own map — per notice. It is simultaneously the largest CPU cost of report generation and a per-notice memory cost. The result depends only on the notice class, so it is memoized in a ConcurrentHashMap<Class<?>, NoticeDocComments>, as you preferred over ClassValue.

The lookup reads with get and falls back to computeIfAbsent only on a miss, which is deliberate: computeIfAbsent locks the bin of an already cached key unless that key happens to be the first entry of its bin. Measured over the 53 notice classes of core, per lookup: ClassValue 2.5 ns, plain computeIfAbsent 6.1 ns, get first 3.9 ns single-threaded (0.65 / 2.34 / 1.05 ns per operation with four threads).

One thing worth a reviewer's eye: NoticeDocComments is mutable and the cached instance is now shared. No caller in the repository mutates it (NoticeView and generateSchemaForNotice only read), and the cache javadoc says the instances must be treated as read-only. If you would rather have a hard guarantee, making the type immutable would be a good follow-up.

3. Serialize the JSON reports straight to their destination. gson.toJson(report) built the whole report as a String and getBytes copied it into a byte array, so writing a report held two or three copies of it at the moment the heap is already at its peak; on a feed with many notices the report is tens of megabytes. The reports are now serialized directly into a Writer on the output file, and the stdout variant writes to System.out (a PrintStream is an Appendable, so its encoding and the trailing newline are unchanged). Since Gson signals a failing writer with the unchecked JsonIOException, the system-errors write now catches that alongside IOException, which is what the previous Files.write form did in effect.

Implementation report for the whole series: https://github.com/CarloMendola/gtfs-validator/blob/9f409204bcefff7387f05a3f70118fb03134443b/prompts/memory-optimization/MEMORY_OPTIMIZATION_REPORT.md

Expected behavior:

No user-visible change. The produced bytes are identical — Gson uses the same serialization path for both APIs — so report.json, system_errors.json and report.html are what master writes for the same feed. Verified on the CTA Chicago feed (95 MB zip, 413 MB of CSV): the notices object of report.json is identical to master's, and report.html differs only in the generation timestamp. Nothing to screenshot for that reason.

Tests:

  • GtfsInputTest — subfolder detection with and without a nested GTFS file, now over a closed stream.
  • NoticeSchemaGeneratorTest.loadComments_isCachedPerNoticeClass — the same instance comes back.
  • NoticeSchemaGeneratorTest.loadComments_noticeWithoutComments_returnsEmptyComments — the missing-resource path still works through the cache.
  • ValidationRunnerTest.exportReport_writesUtf8EncodedJsonReports — writes both JSON reports with a non-ASCII filename in the notices and asserts the file contents, covering the encoding of the new writer path.

Nothing under docs/ describes any of these internals, so no documentation change is needed.

Please make sure these boxes are checked before submitting your pull request - thanks!

  • Run the unit tests with gradle test to make sure you didn't break anything
  • Add or update any needed documentation to the repo
  • Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
  • Linked all relevant issues
  • Include screenshot(s) showing how this pull request works and fixes the issue(s) — no visible change: the reports are identical to master's

CarloMendola and others added 3 commits September 3, 2026 10:42
`hasSubfolderWithGtfsFile` and `createFromUrlInMemory` opened a `ZipInputStream`
over the archive and never closed it. Each of them holds a file descriptor and
an `Inflater`, which owns a zlib stream in native memory released only when the
cleaner runs. It is harmless for a single CLI invocation, but a long-running
service validating one feed after another leaks descriptors and resident memory,
and resident memory is what gets a container killed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`NoticeSchemaGenerator.loadComments` opened a resource and parsed JSON on every
call, and `NoticeView` calls it once per notice. Generating the HTML report for
a feed with many notices therefore repeated the same resource read and JSON
parse thousands of times and retained one identical `NoticeDocComments`
instance, each with its own map, per notice.

The result only depends on the notice class, so it is now memoized in a
`ConcurrentHashMap` keyed by that class: notice classes are loaded once and live
for the lifetime of the process, so nothing more elaborate is needed, and the
cache behaves identically on any runtime.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`gson.toJson(report)` built the whole report as a `String`, and `getBytes` then
copied it into a byte array, so writing a report held two or three copies of it
in memory at once, at the point where the heap is already at its peak. On a feed
with many notices the report is tens of megabytes.

The reports are now serialized directly into a `Writer` on the output file, and
the stdout variant writes to `System.out` (a `PrintStream` is an `Appendable`, so
its encoding and the trailing newline are unchanged). The produced bytes are the
same: Gson uses the same serialization path for both APIs.

Gson signals a failing writer with `JsonIOException`, which is unchecked, so the
system errors report now catches it alongside the `IOException` the file
operations throw, as the previous `Files.write` form did.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@CLAassistant

CLAassistant commented Sep 3, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Comment thread main/src/main/java/org/mobilitydata/gtfsvalidator/runner/ValidationRunner.java Outdated
davidgamez and others added 2 commits October 13, 2026 15:34
Review follow-up. Serializing the report straight to `System.out` made Gson
append every token to the `PrintStream` on its own, and each of those calls is
synchronized and encoded separately: a 7.8 MB report (100 000 sample notices)
took about 7.7 s to write, where `master` takes 0.17 s.

The report now goes through a 64 KB `BufferedWriter` that hands it to
`System.out` in large chunks, so `System.out` still does the encoding. The same
report takes 0.12 s, and the written bytes are identical to `master`'s,
including on a non-ASCII file name.

Adds a test for the stdout output, with a report larger than the buffer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@davidgamez davidgamez left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@davidgamez
davidgamez merged commit 5881eda into MobilityData:master Oct 14, 2026
103 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants