Repository navigation
Issue/2183/2 no observable change - #2187
Merged
davidgamez merged 6 commits intoOct 14, 2026
Merged
davidgamez merged 6 commits into
davidgamez merged 6 commits into
Conversation
`hasSubfolderWithGtfsFile` and `createFromUrlInMemory` opened a `ZipInputStream` over the archive and never closed it. Each of them holds a file descriptor and an `Inflater`, which owns a zlib stream in native memory released only when the cleaner runs. It is harmless for a single CLI invocation, but a long-running service validating one feed after another leaks descriptors and resident memory, and resident memory is what gets a container killed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`NoticeSchemaGenerator.loadComments` opened a resource and parsed JSON on every call, and `NoticeView` calls it once per notice. Generating the HTML report for a feed with many notices therefore repeated the same resource read and JSON parse thousands of times and retained one identical `NoticeDocComments` instance, each with its own map, per notice. The result only depends on the notice class, so it is now memoized in a `ConcurrentHashMap` keyed by that class: notice classes are loaded once and live for the lifetime of the process, so nothing more elaborate is needed, and the cache behaves identically on any runtime. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`gson.toJson(report)` built the whole report as a `String`, and `getBytes` then copied it into a byte array, so writing a report held two or three copies of it in memory at once, at the point where the heap is already at its peak. On a feed with many notices the report is tens of megabytes. The reports are now serialized directly into a `Writer` on the output file, and the stdout variant writes to `System.out` (a `PrintStream` is an `Appendable`, so its encoding and the trailing newline are unchanged). The produced bytes are the same: Gson uses the same serialization path for both APIs. Gson signals a failing writer with `JsonIOException`, which is unchecked, so the system errors report now catches it alongside the `IOException` the file operations throw, as the previous `Files.write` form did. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
davidgamez
reviewed
Oct 13, 2026
Review follow-up. Serializing the report straight to `System.out` made Gson append every token to the `PrintStream` on its own, and each of those calls is synchronized and encoded separately: a 7.8 MB report (100 000 sample notices) took about 7.7 s to write, where `master` takes 0.17 s. The report now goes through a 64 KB `BufferedWriter` that hands it to `System.out` in large chunks, so `System.out` still does the encoding. The same report takes 0.12 s, and the written bytes are identical to `master`'s, including on a non-ASCII file name. Adds a test for the stdout output, with a report larger than the buffer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Second of the six PRs #2183 is split into, in the grouping requested here. Based directly on
masterand independent of the other PRs in the series. Three changes grouped because none of them changes anything observable: same reports, same bytes, same notices — they only stop holding things the run does not need.1. Close the
ZipInputStreamused to look for a subfolder.GtfsInput.hasSubfolderWithGtfsFileandcreateFromUrlInMemoryopened aZipInputStreamover the archive and never closed it. Each holds a file descriptor and anInflater, which owns a zlib stream in native memory released only when the cleaner eventually runs. Harmless for one CLI invocation; a long-running service validating one feed after another leaks descriptors and resident memory. Both are now in try-with-resources.2. Cache the notice documentation comments per notice class.
NoticeSchemaGenerator.loadCommentsopens a classpath resource and parses JSON on every call, andNoticeViewcalls it once per notice. Generating the HTML report for a feed with many notices repeated the same resource read and JSON parse thousands of times and retained one identicalNoticeDocCommentsinstance — each with its own map — per notice. It is simultaneously the largest CPU cost of report generation and a per-notice memory cost. The result depends only on the notice class, so it is memoized in aConcurrentHashMap<Class<?>, NoticeDocComments>, as you preferred overClassValue.The lookup reads with
getand falls back tocomputeIfAbsentonly on a miss, which is deliberate:computeIfAbsentlocks the bin of an already cached key unless that key happens to be the first entry of its bin. Measured over the 53 notice classes ofcore, per lookup:ClassValue2.5 ns, plaincomputeIfAbsent6.1 ns,getfirst 3.9 ns single-threaded (0.65 / 2.34 / 1.05 ns per operation with four threads).One thing worth a reviewer's eye:
NoticeDocCommentsis mutable and the cached instance is now shared. No caller in the repository mutates it (NoticeViewandgenerateSchemaForNoticeonly read), and the cache javadoc says the instances must be treated as read-only. If you would rather have a hard guarantee, making the type immutable would be a good follow-up.3. Serialize the JSON reports straight to their destination.
gson.toJson(report)built the whole report as aStringandgetBytescopied it into a byte array, so writing a report held two or three copies of it at the moment the heap is already at its peak; on a feed with many notices the report is tens of megabytes. The reports are now serialized directly into aWriteron the output file, and the stdout variant writes toSystem.out(aPrintStreamis anAppendable, so its encoding and the trailing newline are unchanged). Since Gson signals a failing writer with the uncheckedJsonIOException, the system-errors write now catches that alongsideIOException, which is what the previousFiles.writeform did in effect.Implementation report for the whole series: https://github.com/CarloMendola/gtfs-validator/blob/9f409204bcefff7387f05a3f70118fb03134443b/prompts/memory-optimization/MEMORY_OPTIMIZATION_REPORT.md
Expected behavior:
No user-visible change. The produced bytes are identical — Gson uses the same serialization path for both APIs — so
report.json,system_errors.jsonandreport.htmlare what master writes for the same feed. Verified on the CTA Chicago feed (95 MB zip, 413 MB of CSV): thenoticesobject ofreport.jsonis identical to master's, andreport.htmldiffers only in the generation timestamp. Nothing to screenshot for that reason.Tests:
GtfsInputTest— subfolder detection with and without a nested GTFS file, now over a closed stream.NoticeSchemaGeneratorTest.loadComments_isCachedPerNoticeClass— the same instance comes back.NoticeSchemaGeneratorTest.loadComments_noticeWithoutComments_returnsEmptyComments— the missing-resource path still works through the cache.ValidationRunnerTest.exportReport_writesUtf8EncodedJsonReports— writes both JSON reports with a non-ASCII filename in the notices and asserts the file contents, covering the encoding of the new writer path.Nothing under
docs/describes any of these internals, so no documentation change is needed.Please make sure these boxes are checked before submitting your pull request - thanks!
gradle testto make sure you didn't break anything