Fix Int overflow on 32-bit platforms (wasm32, watchOS) - #88
Fix Int overflow on 32-bit platforms (wasm32, watchOS)#88mansbernhardt wants to merge 3 commits into
Int overflow on 32-bit platforms (wasm32, watchOS)#88Conversation
|
Yes, valid point. The library was built primarily targeting 64-bit platforms but after the cross-platform effort invariably there are still 32-bit targets in this day and age. WASM64 is in the works but likely not viable any time soon. Forgive my possible naïveté, but |
That would actually be fantastic if you would like to. Preferably branch off main and punt it over as a new PR even if the actual tests fail. I've only recently added Android and now WASM build jobs to repository CI pipelines, but just haven't had time to look into how to get actual unit tests happening on CI. |
Alternative to the saturating fix in orchetect#88, per review feedback: use Int64 where the value genuinely needs 64 bits, rather than clamping. At `.max100Days` a subframe count exceeds Int32.max for every frame rate at the 80- and 100-subframe bases (smallest: 23.976fps@80 = 16_588_800_000), so on a 32-bit platform — wasm32, watchOS armv7k/arm64_32 — the `Int` form trapped on overflow. Because the bound is recomputed inside every wrapping add, that took ALL arithmetic on a `.max100Days` timecode with it, however small the operands. Widened, with no platform-conditional logic: - internal: `FrameCount.subFrameCount`, `framesToSubFrames`, `subFramesToFrames`, `FrameCount.init(subFrameCount:base:)`, and the `sfcNew` locals (which infer). - public: `TimecodeFrameRate.maxTotalSubFrames(in:base:)`, `maxSubFrameCountExpressible(in:base:)`, `Timecode.maxSubFrameCountExpressible`. Deliberately NOT widened: `maxTotalFrames`, which peaks at 1_036_800_000 (120fps @ 100 days) and fits a 32-bit Int; and the frames/subFrames components, which are bounded by it. Only the COUNT needs 64 bits. One narrowing remains, at `Timecode.rationalValue`: `Fraction` is Int-based, so a timecode beyond ~Int32.max subframes has no representable rational value on a 32-bit platform. That is a pre-existing limit of `Fraction`, not of the count, and is commented at the site. Unlike the saturating approach this makes `.max100Days` genuinely usable on 32-bit rather than merely non-trapping. Full suite passes: 506 tests in 55 suites.
83bd6d1 to
597a0c4
Compare
|
Not naïve at all — you're right, and it's the better fix. I've force-pushed it onto this branch (the saturating version is gone; shout if you'd rather have had it as a separate PR and I'll restore). No platform-conditional logic anywhere. The internal domain widens cleanly because the
Deliberately not widened: One narrowing remains, commented at the site: Also correcting something I wrote in the original description: I said the product exceeds Verified: full suite 506 tests in 55 suites on macOS, and 551 tests in 128 suites on wasm32 under wasmtime — the latter by pinning this branch into my own project, which was the only way I could actually execute your library's code on wasm. Which leads into the CI follow-up you asked for; PR coming, and it explains why running your suite there is not yet a two-line job. Unlike the saturating version, this makes |
| let outFrames = (subFrames - outSubFrames) / base.rawValue | ||
| static func subFramesToFrames(_ subFrames: Int64, base: SubFramesBase) -> (frames: Int, subFrames: Int) { | ||
| // The COUNT needs 64 bits; the resulting frames/subFrames do not — | ||
| // max total frames is ~1.04e9 even at 120 fps over 100 days. |
There was a problem hiding this comment.
This holds true at present time, but we can't assume 120fps will remain the highest frame rate provided by the library, nor should we assume that 100 days the greatest maximum upper bound that will be implemented. At 100 days, 120fps occupies 30 bits + 1 for the sign bit. 240fps occupies 31 bits + the sign bit. 480fps overflows a 32-bit signed Int.
Unit tests will invariably trip at any future point if greater frame rates or upper bounds are supported of course but it may be worth considering at this stage.
My feeling that there should be consistency with consumed and emitted types concerning total frame counts and total subframe counts across the library's public API surface. Inconsistency may be confusing to the consumer if a frame count is typed as You may have found them, but for cleanness and conciseness, deprecations all belong in a respective target's
In keeping with the concern for consistency, |
|
Agreed on consistency — a frame count typed One finding worth folding into the scope before anyone starts, which came out of the WASM CI work in #89: there is a third total-count domain in the same family — audio samples. That makes the job meaningfully larger than this PR, which is why I want to ask rather than assume: would you prefer to merge this one as approved and take the consistency work as a follow-up PR, or hold this one and do it all together? I am happy either way and will do the work regardless — it is your API and your call on how to stage it. My only reason for raising it is that this PR is already approved and fixes a hard trap on 32-bit, so there may be value in it landing on its own rather than waiting behind a larger refactor. If you would rather have one coherent change, say so and I will fold it all in here, Unrelated, in case it is useful: |
Yes, good point.
Well aware. GitHub CI has been very unreliable and often the runners and Actions backend cause random test failures and cancellations. It's a constant game of plugging leaks in the dam because the runners are a moving target and their composition never stays static for long. |
Int overflow on 32-bit platforms (wasm32, watchOS)
I think we can add it to this PR, as it's closely related in scope. The commit history is enough to allow in-situ rollbacks if needed. |
Second part of the consistency work requested in review: totals are Int64, per-component values stay Int. Frames (a TOTAL): FrameCount.Value's .frames/.split/.splitUnitInterval payloads, wholeFrames, maxTotalFrames, maxTotalFramesExpressible, Timecode.Stride, and the .frames(_:) source constructors. Samples (a TOTAL): .samples(_:sampleRate:) and samplesValue(sampleRate:). This is the domain with the least headroom — 24 hours at 48 kHz is 4_147_200_000, which overflows a 32-bit Int at the library's ORDINARY limits rather than at some hypothetical future frame rate. Deliberately left as Int, because they are components rather than totals: Components' h/m/s/f, FrameCount.subFrames, and FeetAndFrames (even 24h at 24fps is only ~129_600 feet). Note the Int companion overloads on .samples(_:) and .frames(_:). They are NOT redundant: Int is Swift's default integer-literal type, so with only Int64 and Double overloads present an ordinary expression such as ".samples(48000 * 2, sampleRate: 48000)" becomes ambiguous. They are also not deprecated, because a deprecated overload would then warn on ordinary literal use — worth confirming this is the tradeoff you want. Full suite: 506 tests in 55 suites.
FYI: I was right (#90). The runners are broken. There's nothing wrong with the package or the CI job itself. GitHub wastes so much of my time chasing false positives it's beyond belief. |
Third and final part of the consistency work requested in review. Fraction's numerator and denominator become Int64, along with the internal arithmetic that operates on them (normalize, reduce, greatestCommonDivisor, leastCommonMultiple) and Timecode.frameCount(of:). This also removes the one narrowing the earlier commits had to leave in place: Timecode.rationalValue previously converted a 64-bit subframe count down to Int because Fraction could not hold it, which meant a timecode beyond ~Int32.max subframes had no representable rational value on a 32-bit platform. That conversion is gone and the comment describing the limitation with it. As with .samples(_:) and .frames(_:), Fraction keeps Int companion initializers alongside the Int64 ones. Int is Swift's default integer-literal type, so Fraction(1, 30) should keep resolving without annotation; they are not deprecated because a deprecated overload would warn on ordinary literal use. Full suite: 506 tests in 55 suites, including the CMTime bridge.
|
Consistency work pushed — frames, subframes, samples and The rule I applied throughout: totals are Widened — Left as A nice side effect: widening Two things needing your call1. 2. No
If you want migration cover for those, it needs differently-named accessors — something like Worth noting the tests caught real 32-bit problems on the way through, not just type churn: several sample-count literals in |
|
Thanks very much.
Not strictly the case. func foo() -> Int { 0 }
@_disfavoredOverload
func foo() -> Int64 { 1 }
let x = foo() // infers Int, returns `0`
let y: Int64 = foo() // explicitly Int64, returns `1`The least breaking solution for the consumer would be to keep If there is a solution that can create the least disruption for the vast amount of consumers who are all working exclusively on 64-bit platforms that would be ideal. It doesn't make a ton of sense making disruptive changes to serve the needs of a tiny fraction of the consumer base. |
|
I tested But testing it turned up something that changes the shape of the decision, so rather than reply I went and measured both options.
|
| consistency refactor (currently on this PR) | minimal fix | |
|---|---|---|
| public API changes | frames, subframes, samples, Fraction, Stride |
none |
| test files changed | 3 | 0 |
| native suite | 506 pass | 504 pass |
| wasm32 | not yet run | 554 tests / 128 suites pass |
| fixes the 32-bit trap | yes | yes |
Branch: mansbernhardt:experiment/minimal.
The clamp is safe for the same reason the first version of this PR was: these values are only ever used as an upper bound, and a subframe count that large is itself unrepresentable in a 32-bit Int, so clamping still bounds the entire representable domain. On 64-bit it never engages.
Suggestion
Land the minimal fix to close the 32-bit trap with zero disruption to the 64-bit majority, and treat API consistency as its own deliberate change later — because consistency now unavoidably means choosing types for properties that can't be overloaded, which is an API decision rather than a mechanical refactor, and it deserves to be made on its own terms rather than as a side effect of a bug fix.
That said, this is your library and you've already said you'd like the consistency work here. The full refactor is pushed and green if you'd prefer it — just say which and I'll set the PR to match. I'd rather give you the measurements than argue for one.
|
Thanks for doing the exploratory work on this.
If the properties were converted to functions (
The simplicity of this approach without API changes makes the most sense at this point in time. My only hesitation is having values silently clamp instead of returning actual true values, if that behavior is not obvious at the callsite for consumers. As just one example, audio samples @ 48KHz overflows
Big-picture, yes - you're right. If we adopt There is one other possibility I might entertain at this junction before we ratify a solution. It wouldn't be entirely out of form to conditionally substitute |
|
Built and measured your fence idea rather than replying — it works, and it answers your clamping objection cleanly. Three options, all green
Branch: The shape is a single alias rather than fences at each signature, which keeps the duplication down: #if _pointerBitWidth(_64)
public typealias TimecodeTotalCount = Int
#elseif _pointerBitWidth(_32)
public typealias TimecodeTotalCount = Int64
#else
#error("Unsupported pointer width — TimecodeTotalCount needs a mapping for this platform.")
#endifApplied to total subframe counts and total sample counts — the two domains that provably overflow — plus the internal arithmetic between them. On a 64-bit build the alias is One cost worth knowing before you pick itInternal code has to be written alias-aware, and a 64-bit build will not catch mistakes. Because the alias is That's an ongoing maintenance tax rather than a one-off, and it makes the WASM CI job in #89 load-bearing rather than nice-to-have — without a 32-bit build in CI, this class of breakage lands silently. One thing I could not carry
Happy to set this PR to whichever of the three you prefer — say the word and it's one push. If it were mine I'd take the fence: it fixes the trap, gives true values everywhere, and costs 64-bit consumers nothing. But the maintenance tax above is real and you're the one who'll carry it. |
|
I'd probably move toward something more generic for an alias name like It's possible to swap in the alias for However, the more I dig into this the more it becomes evident there is no trivial way to do it cleanly without some form of compromise for the consumer. I'm increasingly leaning toward a new major version release where the entire codebase would adopt specific bitwidth types for public API consistently at every overflow pinch point, measured not just against current upper bounds but taking into consideration the potential for larger frame rates in future. |
|
On the pin, since it's the one practical thing outstanding on our side: we currently track this fork branch by Would you consider landing either the clamping or the fence variant as a 3.1.x patch in the meantime? Both are zero-public-API-change on 64-bit and green (504 native, 554 on wasm32), so neither pre-empts nor constrains the major version you're planning — they'd just close the 32-bit trap for anyone hitting it today, and let us move back onto a released tag. Entirely your call, and no urgency from our side; the pin is stable. Happy to wait for the major version if you'd rather do it once, properly. On the alias name — agreed that |
The bug
TimecodeFrameRate.maxTotalSubFrames(in:base:)computes its product directly inInt:With
extent == .max100Daysthat product exceedsInt32.maxfor every frame rate. The smallest case, 23.976 fps at 80 subframes, is alreadyso the multiplication traps on overflow on any 32-bit platform — wasm32, and watchOS
armv7k/arm64_32.Because the bound is recomputed inside every wrapping add (
sfcNew.clamped(to: 0 ... maxSubFrameCountExpressible)), this makes all arithmetic on a.max100Daystimecode trap on those platforms, no matter how small the operands are.Repro (wasm32)
Observed in a browser, wasm32 debug build:
Construction, comparison,
max(by:)and.realTimeValueall work; only arithmetic under.max100Daystraps.Worth noting this is easy to hit without ever choosing
.max100Daysdeliberately: our wrapper type sets it on everyTimecodeit constructs, so every timecode operation trapped once we started building for wasm32.The fix
Compute in
Int64, saturate on return:Int64.max, so the clamp never engages. The existing exact-value assertions inTimecodeFrameRate_Properties_Tests.properties()still hold.clamped(to:)range, or a>comparison against asubFrameCount. AsubFrameCountthat large is itself unrepresentable in a 32-bitInt, so saturating atInt.maxstill bounds the entire representable domain.Tests
Two regression tests in
TimecodeFrameRate Properties Tests.swift:maxTotalSubFramesDoesNotOverflowOn32Bit()— every frame rate × every subframe base at.max100Days; asserts the exact product on 64-bit andInt.maxon 32-bit, and thatmaxSubFrameCountExpressiblestays consistent.max100DaysArithmeticDoesNotTrap()— the wrapping add above.Full suite green locally: 506 tests in 55 suites passed.
One suggestion, happy to do it separately
The wasm CI jobs added in #87 (mine) run
swift buildonly. This defect compiles perfectly and traps at runtime, so a build-only job structurally cannot catch it — and the tests above would have, had the suite run under wasm32. If you'd like, I can follow up with a PR that runsswift teston the wasm jobs via wasmtime or Node.