Fix heap-use-after-free in threaded area/radius propagation sweeps (~50% of threaded area runs corrupt, then segfault) - #4
Open
huxnlk wants to merge 1 commit into
Conversation
PlotPropagation(), PlotLOSMap() and PlotPropagationRadius() dispatch std::async(std::launch::async, ...) workers with a raw pointer into a local std::vector (ranges/r/radii), but store the returned std::future in a namespace-scope futures vector that is never read. finishProgress()/finishThreads() only poll progress counters or join an unused threads vector, so a worker can be observed as "done" the instant it increments its counter, one dereference before it actually returns. The local ranges/r/radii vector is then destructed (freeing its backing storage) while that worker still has a pending dereference of its PropagationRange/PropagationRadius pointer - AddressSanitizer confirms this as a heap-use-after-free at src/models/los.cc:175 in rangePropagation. Fix: make futures local to each function that dispatches workers, and actually await every future with .get() before the ranges/r/radii vector can be touched again or destructed. This also removes the implicit reliance on a std::future destructor blocking, which never applied here because futures lived at namespace scope and was never destroyed between calls. Also fixes an independent defect in the same PlotPropagation() cleanup step: erasing ranges by index while iterating shifts every later element down and shrinks size() by one, silently skipping every other element and calling erase(end()) on the last iteration of an even-sized vector. Replaced with ranges.clear(), since every element is discarded anyway. Verified on commit 7f6242a: 10/10 consecutive threaded area-sweep runs now exit 0 with a correct Area boundaries line and a byte-identical coverage.ppm (25622765 bytes), where the unfixed binary crashed (SIGSEGV, exit 139) with a corrupted bbox in roughly half of runs. 6/6 runs under AddressSanitizer show zero heap-use-after-free errors, where the unfixed binary reproduced the same src/models/los.cc:175 UAF reliably under ASan.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Preamble
First, thanks for maintaining this. With
Cloud-RF/Signal-Servernowhistory-only, this fork is the practical home of the ITM/ITWOM tooling, and
the CMake build, the
srtm2sdfconverters and the antenna-pattern toolchainall worked out of the box for us.
I have been standing up a VHF/UHF coverage pipeline against commit
7f6242afand hit a data race in the threaded area sweep that silentlycorrupts the result about half the time. Full repro, a stack trace, and a
verified fix below.
Environment
7f6242afb3685ff31d9ad14062b80d692ee56327signalserverHDcmake ../src && make, gcc 13.3.0, cmake 3.28.3srtm2sdf-hd-pm 1(ITM)Built for 32 DEM tiles at 3600 pixels- confirmed on every run belowA data race in the threaded segment workers silently corrupts the coverage result, then segfaults
Severity: high. Roughly half of all threaded area runs produce a wrong
answer. The wrong answer is written to a complete, well-formed
.ppmandaccompanied by a printed summary line, so nothing about the output file itself
says it is bad.
What happens
Seven consecutive runs, byte-identical command line, same tiles, same
container, nothing else changed. The runs split into two clean populations:
Area boundariesbboxThe three clean runs agree exactly. A
-nothreadsserial run of the samecommand gives 49.848 km2, matching them to 0.001 km2. So when the race
does not fire, the threaded sweep is correct.
When it does fire, three things happen together:
consistently ~75% of the correct value, i.e. about three segments' worth
out of the four in
Map segments: 4.Area boundaries:line is printed with nonsense longitudes:-5.910478 | 1619.495330 | -6.719922 | -1308.504670. Valid longitude is-180 to 180 and this AOI is ~155-156. Note the fractional parts
(
.495330,.504670) are exactly those of the site longitude, so theselook like the correct value plus a large garbage integer, not random
memory.
Then the process exits 139 (SIGSEGV) - after a complete
.ppmhas beenwritten (25,622,765 bytes, identical size in all seven run directories,
crashed or not).
Root cause
AddressSanitizer resolves this to a heap-use-after-free at
src/models/los.cc:175, insiderangePropagation:Reading
src/models/los.ccat this commit,PlotPropagation()(and, wefound once we looked, the same pattern in
PlotLOSMap()andPlotPropagationRadius()):std::vector<PropagationRange>(ranges/r/radii) fullybefore starting any thread.
std::async(std::launch::async, rangePropagation, ...)per segment - a raw pointer into the vector's own backing array is
handed to each worker. The returned
std::futureis stored infutures, astd::vector<std::future<void *>>declared at namespacescope (not local to the function, and shared across all three call
sites).
rangePropagation()casts the pointer back toPropagationRange *vand dereferences it in a loop; line 175 is one such dereference, in the
loop's own exit condition.
PlotPropagation()callsfinishProgress()to "wait"for the workers. That function polls progress counters
(incremented inside each worker's loop body) on a 500 ms sleep cycle,
and returns as soon as the running total matches the expected total. It
never calls
.get(),.wait(), or otherwise touchesfuturesor thestd::asyncreturn values - its own comment says "then finish out anyrunning threads", but that step is not implemented.
(
PlotLOSMap()andPlotPropagationRadius()callfinishThreads()instead, which only joins the separate, always-empty
threadsvector -an equivalent no-op, since nothing is ever pushed into
threads.)loop condition and running its cleanup/return, it can be observed as
"done" the instant its last counter increment lands - while it still
has one more dereference of
vto make before it actually exits.futuresnever being read also removes the other safety net: astd::futurefromstd::async(std::launch::async, ...)normallyblocks in its destructor until the thread completes - but only if it is
destroyed or waited on. Living at namespace scope,
futuresis neverdestroyed between calls, so that implicit wait never fires either.
PlotPropagation()erases everyelement of
rangesand, at function return, destructs the vector -freeing its backing storage. If a worker's exit was still pending, it
now dereferences freed memory at line 175. That is exactly what ASan
caught: freed by the main thread, read by a worker thread.
A second, independent defect in the same cleanup step:
Erasing index
ishifts every later element down and shrinks.size()byone, but
istill advances by one next iteration - so this skips everyother element, and on an even-sized vector its last iteration calls
erase(end()), which is undefined behaviour.The fix
This PR:
futureslocal to each ofPlotLOSMap(),PlotPropagation()andPlotPropagationRadius()instead of namespace-scope, so it can never beshared or left stale across calls.
for (auto &f : futures) { f.get(); }afterfinishProgress()/finishThreads()in all three functions, so everyworker is actually awaited before its backing vector can be touched
again or destructed.
PlotPropagation()withranges.clear()(every element is being discarded anyway, so this alsosidesteps the same-class bug rather than just patching the loop bounds).
I deliberately kept the diff to these three call sites and did not touch
finishProgress()/finishThreads()themselves, or the identicalerase-by-index pattern in
PlotPropagationRadius()'sradiicleanup(same file, a few dozen lines further down)
Verification
Built commit
7f6242afplus this patch, Ubuntu 24.04, gcc 13.3.0, exactrepro command below.
Area boundariesline, byte-identical 25,622,765-bytecoverage.ppm.Before the fix, the same build/command/tiles reproduced the crash in
roughly half of runs (fresh 7-run series immediately before this fix:
6 clean, 1 crashed with the corrupt bbox above - matching the original
table).
-g -fsanitize=address -fno-omit-frame-pointer,ASAN_OPTIONS=detect_leaks=0): zeroheap-use-after-free errors. The same ASan build without the fix
reproduces the
src/models/los.cc:175UAF reliably.