fix: bound remaining mcp_servers.json reads and close fifth unlocked writer - #802
Open
thomwebb wants to merge 1 commit into
Open
fix: bound remaining mcp_servers.json reads and close fifth unlocked writer#802thomwebb wants to merge 1 commit into
thomwebb wants to merge 1 commit into
Conversation
…writer Second follow-up to PUP-605, prompted by Andrew Tilson's post-merge review comment on mpfaffenberger#761. Two gaps remained in mcp_servers.json handling specifically: - HIGH: config.py's load_mcp_server_configs() -- reached at every CLI startup via MCPManager.__init__ -> sync_from_config(), and again from agents/_builder.py -- still did an unbounded open()/f.read(). This is the exact PUP-605 MemoryError crash shape, unfixed, on a startup path, for the file mcp_servers_store.py's own docstring already warned could balloon json.load the same way configparser ballooned in the original crash. The sibling JSONAgent startup read was correctly bounded in mpfaffenberger#761; this one was an oversight. Fixed via atomic_io.read_bounded_bytes, matching every other call site. - MEDIUM-HIGH: a fifth mcp_servers.json writer in mcp_/config_wizard.py::run_add_wizard() (public API, exported from mcp_/__init__.py) was still doing an unlocked, unbounded read-modify-write with a fixed (collision-prone) temp filename and no fsync. mpfaffenberger#761 correctly collapsed the four writers flagged in mpfaffenberger#757's review into one locked mcp_servers_store.py, but this fifth writer was missed -- and a lock only provides mutual exclusion if every writer participates. Migrated to the same shared mcp_servers_store.upsert_mcp_server() the other four use. Also addressed, from the same review comment: - Bounded two lower-priority project-level mcp_servers.json reads (mcp_/project_config.py, command_line/mcp/trust_command.py) -- project config is repo-supplied input (whatever repo you cd into), so these were flagged as next-most-wanted after the two items above. - atomic_io.atomic_write_bytes now masks stat mode bits via stat.S_IMODE before chmod, rather than passing os.stat()'s raw st_mode (which includes file-type bits) straight through -- harmless today since chmod masks it anyway, but worth doing properly now that it is a shared primitive. - path_lock's docstring now documents that it is not reentrant (flock is scoped per open-file-description, not per-process/thread). Note: this review also flagged that aws_bedrock/azure_foundry's now-dead save_extra_models() should be documented or dropped -- moot here, since those plugins were relocated to the separate code_puppy_core_plugins package in the interim (27efe8f) and no longer exist in this repo. Added regression tests pinning both bounded-read fixes, and rewrote the wizard-save test to use a real tmp_path file instead of mocking open()/os.path.exists()/json.dump internals that no longer exist post- refactor -- that stale mocking had been silently leaking a real file write to /tmp/mcp_servers.json on every test run. Full suite (minus one unrelated, pre-existing untracked/broken test file): 7461 passed, 28 skipped, 1 xpassed. ruff check + format clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Jira: PUP-610
Second follow-up to PUP-605, per Andrew Tilson's post-merge review of #761
After #761 merged (extending bounded/atomic/locked I/O to
mcp_servers.json,extra_models.json,spinners.json, and agent JSON files), Andrew left a further comment identifying two remaining gaps specific tomcp_servers.json-- the exact file this whole line of fixes exists to protect.HIGH:
config.py'sload_mcp_server_configs()was still an unbounded read, and it's on the startup pathReached at every CLI startup via
MCPManager.__init__->sync_from_config(), and again fromagents/_builder.py. This is the original PUP-605MemoryErrorcrash shape, unfixed, on a startup path, for the very filemcp_servers_store.py's own docstring already warned could "balloonjson.loadthe same wayconfigparserballooned in the original PUP-605 crash." The siblingJSONAgentstartup read was correctly bounded in #761 -- this one looked like a straightforward oversight.Fixed via
atomic_io.read_bounded_bytes, matching every other call site.MEDIUM-HIGH: a fifth, previously unidentified
mcp_servers.jsonwritermcp_/config_wizard.py::run_add_wizard()(public API, exported frommcp_/__init__.py) was still doing an unlocked, unbounded read-modify-write with a fixed (collision-prone) temp filename and nofsync. #761 correctly collapsed the four writers flagged in #757's original review into one lockedmcp_servers_store.pymodule -- but this fifth writer was missed. A lock only buys mutual exclusion if every writer participates; this one unlocked writer could silently void the safety guarantee the other four now have, which is arguably worse than before the fix, since the other four now look safe.Fixed by migrating to the same shared
mcp_servers_store.upsert_mcp_server()the other four use.Also from the same review comment
mcp_servers.jsonreads (mcp_/project_config.py,command_line/mcp/trust_command.py) -- project config is repo-supplied input (whatever repo youcdinto), so these were flagged as next-most-wanted after the two items above.atomic_io.atomic_write_bytesnow masks stat mode bits viastat.S_IMODEbeforechmod, rather than passingos.stat()'s rawst_mode(which includes file-type bits) straight through. Harmless today sincechmodmasks it anyway, but worth doing properly now that it's a shared primitive.path_lock's docstring now documents that it is not reentrant (flockis scoped per open-file-description, not per-process/thread) -- a sentence to keep it that way rather than a real bug found today.One nit from the same comment -- documenting or dropping
aws_bedrock/azure_foundry's now-deadsave_extra_models()-- turned out to be moot: those plugins were relocated to the separatecode_puppy_core_pluginspackage in the interim (27efe8f5) and no longer exist in this repo.Testing
tests/test_config_full_coverage.py,tests/mcp/test_project_config.py).tests/mcp/test_config_wizard.pyto use a realtmp_pathfile instead of mockingopen()/os.path.exists()/json.dumpinternals that no longer exist post-refactor -- that stale mocking had been silently letting a real file write leak to/tmp/mcp_servers.jsonon every test run (cleaned up).maintip, including thecode_puppy_core_pluginsextraction and the session-persistence migration).ruff checkandruff format --checkboth clean.cc @AndrewTilson -- both flagged gaps should be closed now.