Repository navigation
aiohttp integration corrupts SigV4-signed propagation headers, breaking aiobotocore S3 calls (since 2.69.0) #7426
Description
How do you use Sentry?
Sentry Saas (sentry.io)
Version
2.69.0
Steps to Reproduce
Since 2.69.0, S3 calls made through aiobotocore fail with SignatureDoesNotMatch whenever a Sentry transaction is active. Synchronous boto3 is not affected.
aiobotocore sends botocore's already-signed requests through aiohttp. #7050 made the Boto3 integration add sentry-trace and baggage at before-sign, so both are named in SignedHeaders. The aiohttp client integration then rewrites them after signing, appending to baggage:
if key == BAGGAGE_HEADER_NAME and params.headers.get(BAGGAGE_HEADER_NAME):
params.headers[key] += "," + valueThe sent value no longer matches the signed value, so the request is rejected.
stdlib already skips propagation headers named in SignedHeaders, which is why sync boto3 is fine. The aiohttp path never got the same guard.
Reproduction against any S3-compatible endpoint:
import asyncio, sentry_sdk
import aiobotocore.session as abcs
async def main():
sentry_sdk.init(dsn="<dsn>", traces_sample_rate=1.0)
session = abcs.get_session()
async with session.create_client(
"s3", region_name="auto", endpoint_url="<endpoint>",
aws_access_key_id="<key>", aws_secret_access_key="<secret>",
) as client:
with sentry_sdk.start_transaction(op="t", name="r2"):
await client.list_objects_v2(Bucket="<bucket>", MaxKeys=1)
asyncio.run(main())Expected Result
The request is sent with the headers botocore signed, and succeeds.
Actual Result
botocore.exceptions.ClientError: An error occurred (SignatureDoesNotMatch) when calling the
ListObjectsV2 operation: The request signature we calculated does not match the signature you
provided. Check your secret access key and signing method.
Isolation, against a real Cloudflare R2 bucket, with credentials, botocore 1.43.75 and aiobotocore 3.9.1 held constant:
aiobotocore, all default integrations SignatureDoesNotMatch
aiobotocore, AioHttpIntegration disabled OK
aiobotocore, Boto3Integration disabled OK
synchronous boto3, all integrations OK
By SDK version, same setup:
2.63.0 OK
2.66.0 OK
2.67.0 OK
2.68.0 OK
2.69.0 SignatureDoesNotMatch
To rule out the endpoint being strict about what may be signed: with no SDK involved, botocore signing baggage, sentry-trace, an arbitrary x-koodle-test or x-amz-meta-* is accepted every time. The problem is only that the header changes after signing.
Two notes for anyone hitting this. Calls fail only inside a transaction, so anything running outside one keeps working and it presents as intermittent. And the workarounds are trace_propagation_targets set to exclude the endpoint, or pinning sentry-sdk<2.69.0.
Fix in #7427.
Metadata
Metadata
Assignees
Labels
Projects
- Status
Activity
linear-code commentedon Sep 8, 2026
PY-2769
Robinbinu commentedon Sep 8, 2026
Updated the description above. My first analysis blamed the endpoint for rejecting signed
baggageandsentry-trace, which was wrong: signing those is fine. The cause is the aiohttp integration rewriting them after botocore signed, on a pathstdlibalready guards. Fix in #7427.[-]botocore SigV4 SignatureDoesNotMatch against Cloudflare R2 since 2.69.0 (trace headers added before signing)[/-][+]aiohttp integration corrupts SigV4-signed propagation headers, breaking aiobotocore S3 calls (since 2.69.0)[/+]fix(aiohttp): Preserve SigV4-signed propagation headers
fix(aiohttp): Preserve SigV4-signed propagation headers
pabloDeputter commentedon Sep 8, 2026
Thank you for the report! I’ll take a look at the proposed fix.
fix(aiohttp): Preserve SigV4-signed propagation headers
fix(aiohttp): Preserve SigV4-signed propagation headers
fix(aiohttp): Preserve SigV4-signed propagation headers
fix(aiohttp): Preserve SigV4-signed propagation headers