Python: every recipe bundle runs in the server's own process

[knutwannheden]

A mod run of a pip recipe bundle spawns one Python process per bundle and sends the whole LST to each. The facade deserializes the tree from Java, then re-serializes it per child — on jd/tenacity that relay is 1,303,024 messages written and as many parsed again, producing no edits.

The second process exists to keep two bundles' dependencies apart. A private module table does that in one process, so it isn't needed: sys.modules is global, so two versions of one package cannot both be sys.modules['dep'] at once, but they can take turns. Entering a bundle swaps its entries in and back out.

Which modules a bundle owns

Ownership is by where a module was loaded from, not by name. The engine is imported from the shared engine root, so it matches no bundle and every bundle sees the one instance — no list to maintain and no drift.

A dependency the engine declares is already imported and resolves to the engine's copy. That is the same answer a child gives, whose PYTHONPATH puts the engine roots first.

What it costs and what it buys

Two recipe bundles on incompatible versions of one C-extension dependency, in one process, both on the C path rather than falling back to pure Python:

openrewrite-conflict-a   msgpack (1, 0, 8)  via msgpack._cmsgpack   engine shared: True
openrewrite-conflict-b   msgpack (1, 1, 0)  via msgpack._cmsgpack   engine shared: True

On jd/tenacity with UpgradeToPython313, against a CLI built from this branch with the engine installed from a real wheel:

                 wall   fix.patch   python CPU   processes
main             167s   c0a2a2ab       51.5s     2  (facade 27.2s + child 24.4s)
this branch       49s   c0a2a2ab       26.9s     1

Installing a second bundle leaves it at one process, where main spawns two children. Single runs on a contended machine, so read the process counts rather than the timings.

The trade-off

A bundle that dies natively takes the server with it, where before it took only its own child:

main          child dies (SIGSEGV) -> server reports the error and keeps serving
this branch   server dies, exit -11, later requests fail with BrokenPipe

• That is the real cost of the change and it wants a decision, not a footnote. It also applies to Python: a lone recipe bundle runs in the facade's own process #8745, which hosts a lone bundle in-process for the same reason.

A venv built by another interpreter is now rebuilt rather than routed around, since there is no child on its own interpreter to fall back to — is_usable_venv checks the recorded version alongside the base it borrows its stdlib from.

What goes

facade.py, bundle_children.py, the _hub_* block in server.py and the per-child ref tables all existed to move a tree between processes. ChildConnection stays: java_rpc_client drives a Java peer over the same framing, so only the environment its bundle children needed (child_command, _child_env, _engine_roots) goes with them.

Tests

test_bundle_overlay.py pins the table itself:

• two bundles resolving one module name to different copies
• a module from outside every venv staying shared
• a lazy import inside a call resolving to the calling bundle
• the host's path and modules surviving unchanged
• recipes interleaved as the scheduler runs them, A then B on each file in turn, since the table swaps per visit rather than per file

Run against the built wheel rather than the source tree. A module deleted from the worktree still resolves from an installed copy in site-packages, so a source-tree run cannot see deletions at all; the wheel run is what caught java_rpc_client's use of ChildConnection.

• Relationship to Python: a lone recipe bundle runs in the facade's own process #8745

That PR hosts a bundle in-process only while it is the only one, keeping spawn-and-route from the second bundle on. This one removes the spawn path outright.

• They are alternatives, opened side by side to compare. Python: a lone recipe bundle runs in the facade's own process #8745 is the smaller change carrying the same crash trade-off at n = 1; this one pays it at every n and deletes about 1,600 lines.