Conversation
The divisors (hsamp << r), (vsamp << r), (hsamp << rpx) and (vsamp << rpy) used by the RPCL/PCRL/CPRL packet iterators can wrap around to zero on 32-bit builds. hsamp/vsamp come from the SIZ marker and may be any value in 1..255, while r/rpx/rpy can reach 29; e.g. hsamp=8 with rpx=29 yields 8 << 29 == 0 (mod 2^32), causing a division by zero (SIGFPE) in jpc_pi_nextrpcl as reported by OSS-Fuzz in jasper-software#420. Guard all four shifted divisors before their first use, mirroring the existing zero checks for xstep/ystep.
iliasabk
force-pushed
the
fix-pi-fpe-shifted-samp
branch
from
September 17, 2026 05:02
bf37982 to
34609b7
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the floating-point exception reported by OSS-Fuzz in #420 (
graphicsmagick:coder_JPC_fuzzer, FPE injpc_pi_nextrpclatjpc_t2cod.c:340).Root cause
In the RPCL/PCRL/CPRL packet iterators, the divisors
hsamp << r,vsamp << r,hsamp << rpxandvsamp << rpyare computed inuint_fast32_t. On 32-bit builds (asan_i386) this is 32 bits wide.hsamp/vsampcome from the SIZ marker and may be any value in 1..255 (nonzero is enforced injpc_cs.c), and the existing overflow check permitsr,rpx,rpyup to 29.For even sampling factors the shift can wrap to zero: e.g.
hsamp=8,rpx=29gives8 << 29 == 0(mod 2^32), andx % 0/JPC_CEILDIV(..., 0)raises SIGFPE. The existing zero checks only coverxstep/ystep, not these per-resolution-level divisors.Fix
Guard all four shifted divisors before their first use in each of the three affected iterators (
jpc_pi_nextrpcl,jpc_pi_nextpcrl,jpc_pi_nextcprl), returning -1 like the neighboring malformed-data checks. In the PCRL/CPRL iteratorsrpx/rpyare now computed beforetrx0/try0so the guard can precede the divisions (no semantic change).Verification
jpc_t2cod.ccompiles without warnings).8 << 29 == 0,128 << 25 == 0for thevsamppath) in 32-bit arithmetic; the new guard fires in both cases.Disclosure
I am an LLM-based coding agent (Devin). I am available to discuss this change and address review feedback.