Implement Put Block From URL - #2681
Conversation
There was a problem hiding this comment.
Pull request overview
Adds support for Block Blob Stage Block From URL (Put Block From URL) in Azurite by downloading the source content via the existing blob download path (so SAS/range/source conditions can be applied) and persisting the staged data through the normal extent + uncommitted-block flow.
Changes:
- Implements
stageBlockFromURLinBlockBlobHandlerby fetching the source via HTTP and staging it as an uncommitted block. - Introduces a new
SourceConditionNotMet(412) storage error for unmet source conditional headers during staging. - Expands the block blob API test suite with stageBlockFromURL coverage (range, full copy, unmet condition, missing source).
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| tests/blob/apis/blockblob.test.ts | Adds tests validating stageBlockFromURL behavior (ranges, full copy, 412 on unmet source condition, 404 on missing source). |
| src/blob/handlers/BlockBlobHandler.ts | Implements the stageBlockFromURL handler: validates input, downloads source data, persists it, and returns MD5. |
| src/blob/errors/StorageErrorFactory.ts | Adds getSourceConditionNotMet() (412) error factory helper. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
Addressed the review in aa00b52:
|
|
Second round addressed in 381b126: the source fetch now pins to the local address the request arrived on (bracketing IPv6 literals) instead of hard-coded 127.0.0.1, so non-loopback |
|
Addressed in 8847aec: the pinned loopback fetch now sends the original source URL host as the |
|
Addressed in e665103: the same-instance comparison now lowercases the client-supplied Host header before comparing against the already-lowercased URL host, with a mixed-case Host regression test. |
e665103 to
d2654cb
Compare
|
Squashed after AI review succeeded. |
|
Andrew Gaul (@gaul) , could you please refresh your PR with main, and address the review comments if any to move this PR forward. |
d2654cb to
33059bb
Compare
Done. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.
Suppressed comments (1)
src/blob/handlers/BlockBlobHandler.ts:431
sourceContentMD5validation currently compares raw bytes and throwsInvalidOperationon mismatch. This diverges from other checksum validation paths in the codebase and fails to return the expectedInvalidMd5(wrong length) /Md5Mismatch(value mismatch) errors.
Consider validating that sourceContentMD5 is exactly 16 bytes and, on mismatch, throwing StorageErrorFactory.getMd5Mismatch(...) with base64-encoded values (same behavior as computeAndValidateTransactionalChecksums).
if (options.sourceContentMD5 !== undefined) {
if (
!Buffer.from(options.sourceContentMD5).equals(calculatedContentMD5)
) {
throw StorageErrorFactory.getInvalidOperation(
33059bb to
4f9ee08
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
src/blob/handlers/BlockBlobHandler.ts:399
- Range validation uses Number.parseInt(...), which loses integer precision above 2^53-1. For very large x-ms-source-range values this can mis-evaluate the end < start check and allow an invalid range through; the download path then ignores the malformed Range header and would silently stage the entire source blob. Use BigInt for the comparison so the validation works for the full 64-bit range syntax Azure accepts.
const rangeMatch = /^bytes=(\d+)-(\d*)$/.exec(options.sourceRange);
if (rangeMatch === null ||
(rangeMatch[2] !== "" &&
Number.parseInt(rangeMatch[2], 10) <
Number.parseInt(rangeMatch[1], 10))) {
Done although 2^53 range is unreasonable for a testing framwork. |
01e9382 to
f0eb07c
Compare
f0eb07c to
12d4df3
Compare
CodeQL comment is correct
Stage the block by fetching the copy source over loopback so that SAS authentication, x-ms-source-range, and the x-ms-source-if-* conditions are enforced by the existing download path, then persist it through the same extent flow as Put Block. Only sources on the same Azurite instance are supported, matching copyFromURL. The response carries the MD5 of the staged content and source condition failures return 412 SourceConditionNotMet as on the real service. Reject malformed x-ms-source-content-md5 and x-ms-source-content-crc64 up front so the errors name the header the caller sent, then compare the surviving values against the fetched bytes with computeAndValidateTransactionalChecksums, the same path Put Block uses. Mismatches return Md5Mismatch or Crc64Mismatch, and supplying both returns BothCrc64AndMd5HeaderPresent. The response reports the computed CRC64 when no MD5 was supplied, as stageBlock does. Request the source body verbatim and never decompress it. A blob's Content-Encoding is stored metadata rather than a description of the wire framing, so the download echoes it back over the raw stored bytes; decoding here would stage the decompressed content instead of what the source holds, and would fail outright when the property does not match the bytes (issue Azure#646). The source fetch is pinned to the address and port the request arrived on, so over HTTPS it presents whatever certificate Azurite was started with. Skip verification for that self-request, which would otherwise reject the self-signed certificates Azurite is normally run with and make the operation unusable under --cert/--key, and translate transport-level failures into CannotVerifyCopySource rather than letting them escape as a bodiless 500. The same applies to a body that fails partway through being read. Validated with the blockblob test suite and end to end with S3Proxy's native multipart part copy, which previously fell back to streamed emulation on Azurite's 501. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
12d4df3 to
f823533
Compare
Replace rejectUnauthorized: false with an https.Agent that trusts exactly the certificate this server presents, read from the socket the request arrived on, so a substituted certificate is still rejected. Hostname verification stays off because the request targets the bound address rather than a name. Addresses the CodeQL disabling-certificate-validation alert. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Suppressed comments (1)
src/blob/handlers/BlockBlobHandler.ts:33
- The LOOPBACK_HTTPS_AGENT comment is internally inconsistent (“handshake per request” vs “over an idle socket … mid-reuse”), which makes it hard to understand why keep-alive is intentionally disabled here. Reword to clearly state the intent (shared Agent to avoid reallocation; keep-alive disabled to avoid idle socket reuse).
/**
* Agents for the loopback self-request stageBlockFromURL makes to read a copy
* source, keyed by the certificate they pin. Shared so requests reuse one
* Agent rather than allocating their own, not for socket reuse: keep-alive
* stays off, taking a loopback handshake per request over an idle socket that
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
422ed99
into
Azure:main
The azureblob store copies a part with Put Block From URL, which no Azurite implemented: the first attempt came back 501, the store remembered that, and every UploadPartCopy afterwards fell back to streaming the range through s3proxy. Azure/Azurite#2681 implements the operation, source range and x-ms-source-if-* conditions included, and it sits past the startFrom commit this pin already named, so move the pin on to it rather than build twice. The lane takes the native path now: the multipart copy tests stage their blocks from a URL, and a condition the source does not meet comes back 412 SourceConditionNotMet, which is the 412 s3proxy owes the client. The bump also picks up Azure/Azurite#2680, which authorizes a classic copy's source with a HEAD instead of downloading the whole blob only to discard it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Stage the block by fetching the copy source over loopback so that SAS authentication, x-ms-source-range, and the x-ms-source-if-* conditions are enforced by the existing download path, then persist it through the same extent flow as Put Block. Only sources on the same Azurite instance are supported, matching copyFromURL. The response carries the MD5 of the staged content and source condition failures return 412 SourceConditionNotMet as on the real service.
Validated with the blockblob test suite and end to end with S3Proxy's native multipart part copy, which previously fell back to streamed emulation on Azurite's 501.