[Vsintegration] Cut UI-thread marshalling and code model database lock contention

[hpetriffer]

Six changes to the VS integration, each measured against a before/after run on the same machine. They fall into two groups: how project references are resolved through the automation API, and how the code model database is written.

Everything started from the intellisense logs of ~150 real sessions, which pointed somewhere unexpected: the file-open pipeline itself is fast and does not degrade with session age (ParseAsync p50 stays 13-31 ms from minute 0 to hour 4, colorization 1-13 ms). What makes the IDE feel slow is UI-thread marshalling and lock contention from other subsystems.

Project references

OAProjectReference.SourceProject resolved its hierarchy on every read, through VsShellUtilities.GetHierarchy, with no caching, each read inside a JoinableTaskFactory.Run that marshals to the UI thread. BuildDependency.GetReferencedHierarchy already cached exactly that lookup on ProjectInfo.Hierarchy; SourceProject ignored it.

The logs show 2,388,244 of these lookups across ~150 sessions - 234,360 in a single 11-minute session, 8,928 in one second.

It now reuses ProjectInfo.Hierarchy and caches the resolved EnvDTE.Project on a new ProjectInfo.DteProject, shared by every reference node pointing at the same project. Invalidation goes through the existing ProjectInfo lifetime, driven from SolutionListenerForProjectReferenceUpdate on project close, unload, reload and solution close. That listener sees every project in the solution, unlike SolutionListenerForProjectEvents which filters to our own hierarchies - so foreign project references are covered too. That also closes a pre-existing hole where BuildDependency could keep using the hierarchy of an unloaded foreign project.

A separate commit makes SourceProject register the ProjectInfo when it is missing. Without that, the cache only worked when CreateBuildDependencies happened to run first, and the difference was large: two sessions on the same solution with identical code gave 88 calls / 0 shell lookups and 4,256 calls / 4,170 shell lookups.

On RadixWf.sln (226 projects, 3,283 project reference nodes, 88 distinct referenced projects):

	before	after
GetHierarchy calls on solution load	13,183	88
calls per referenced project	150	1.00
solution-load burst, wall clock	6.54 s	2.23 s

Code model database

The database is a single in-memory SQLite connection behind one global lock. Three things were happening inside it that did not need to be.

Backups chained. CommitWhenNeeded is called from every write path but only updated lastWritten once SaveToDisk had finished, so every caller inside the 5-minute window queued its own BackgroundWorker and they serialized on the lock. One August session shows five backups running back to back, 10:00:03 to 10:00:35 - half a minute with the code model unavailable. The interval is now claimed when the backup is scheduled, with an Interlocked guard behind it, and the lock is held only around BackupDatabase instead of around the file delete, the OpenFile and the VACUUM as well. SafeFileDelete also slept 100 ms before its first attempt, on every backup.

A per-file orphan sweep. UpdateFileContents ran Delete from IncludeFiles where Id not in (select IdInclude from IncludeFilesPerFile) - a full anti-join - for every file written, 39,380 times in a cold walk, and redundant with the bulk DeleteOrphanFiles() that already runs at database open. Now once per project walk.

No transactions at all. Every statement was its own implicit transaction, and AddTypes/AddMembers issue one INSERT per type and per member (~160 for a 159-entity file). UpdateFileContents now wraps its work in BEGIN IMMEDIATE/COMMIT with a ROLLBACK in a FINALLY. These are sent as plain SQL rather than via DbTransaction on purpose: a DbTransaction would have to be assigned to every command created down the call chain, and Microsoft.Data.Sqlite throws when one misses it - and this builds against both providers.

Plus XFile.SaveToDatabase() called RefreshGlobalUsings() - a query - after every single file. It now marks the lists stale and they rebuild on first read.

Cold walk of RadixWf.sln, 39,380 file writes in every run:

	original	after
total time holding the DB lock	2,088 s	197 s (-91%)
p50 per write	58 ms	3 ms
worst single write	5,164 ms	265 ms
writes blocking > 1 s	3	0
chained backups	5 in a row	none
wall clock	~18 min	2.8 min

Two corrections made along the way

RefreshGlobalUsings built its result by clearing the live lists and refilling them, while the walker writes from several threads - a reader could see the list half emptied. It now builds new lists and swaps them in. And its dirty flag starts TRUE, because previously the only thing that ever populated those lists was a file save, so opening a solution against an up-to-date database left them empty.

What has and has not been verified

Verified: the unload/reload invalidation, over three open/unload/reload/close cycles - 86 of 88 projects resolve exactly once per cycle, no growth across cycles, and never below 88 per cycle, which would have meant stale entries surviving into the next solution. No errors in any run. The rebuilt database is byte-comparable in size across before/after walks.

Please be aware of the limits:

• Every number is one sample per configuration, on one solution, in VS 2022 Experimental. The A/B pairs are same-machine with near-identical workloads (39,380 writes, ~157,570 parses in both), which is the strongest evidence short of a benchmark suite - but it is not one.
• Not tested on VS 18.
• Caching an EnvDTE.Project across reference nodes was suspected of causing VS crashes during testing and then cleared - four crash dumps all showed FileNotFoundException: AcsLib on the finalizer thread via Component.Finalize(), reproduced on unmodified dev with the changes stashed out. But it has not been positively proven safe against every automation consumer, and a maintainer may know of consumers I do not.

One thing worth fixing separately, because it makes this area hard to profile: Logger.StartAsync() calls XSettings.EnableAll(), and IntellisenseOptions.WriteToSettings() then copies the option-page defaults straight over it. In a fresh experimental hive that silently disables database and parser logging, which is easy to misread as "the work never ran".

🤖 Generated with Claude Code