Skip to content

Implement Put Block From URL - #2681

Merged
Akanksha Jain (jainakanksha-msft) merged 3 commits into
Azure:mainfrom
gaul:stage-block-from-url
Aug 25, 2026
Merged

Implement Put Block From URL#2681
Akanksha Jain (jainakanksha-msft) merged 3 commits into
Azure:mainfrom
gaul:stage-block-from-url

Conversation

@gaul

Copy link
Copy Markdown
Contributor

Stage the block by fetching the copy source over loopback so that SAS authentication, x-ms-source-range, and the x-ms-source-if-* conditions are enforced by the existing download path, then persist it through the same extent flow as Put Block. Only sources on the same Azurite instance are supported, matching copyFromURL. The response carries the MD5 of the staged content and source condition failures return 412 SourceConditionNotMet as on the real service.

Validated with the blockblob test suite and end to end with S3Proxy's native multipart part copy, which previously fell back to streamed emulation on Azurite's 501.

Copilot AI lite review requested due to automatic review settings July 25, 2026 22:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds support for Block Blob Stage Block From URL (Put Block From URL) in Azurite by downloading the source content via the existing blob download path (so SAS/range/source conditions can be applied) and persisting the staged data through the normal extent + uncommitted-block flow.

Changes:

  • Implements stageBlockFromURL in BlockBlobHandler by fetching the source via HTTP and staging it as an uncommitted block.
  • Introduces a new SourceConditionNotMet (412) storage error for unmet source conditional headers during staging.
  • Expands the block blob API test suite with stageBlockFromURL coverage (range, full copy, unmet condition, missing source).

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
tests/blob/apis/blockblob.test.ts Adds tests validating stageBlockFromURL behavior (ranges, full copy, 412 on unmet source condition, 404 on missing source).
src/blob/handlers/BlockBlobHandler.ts Implements the stageBlockFromURL handler: validates input, downloads source data, persists it, and returns MD5.
src/blob/errors/StorageErrorFactory.ts Adds getSourceConditionNotMet() (412) error factory helper.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/blob/handlers/BlockBlobHandler.ts
Comment thread src/blob/handlers/BlockBlobHandler.ts
Comment thread src/blob/handlers/BlockBlobHandler.ts Outdated
Copilot AI review requested due to automatic review settings July 25, 2026 22:51
@gaul

Copy link
Copy Markdown
Contributor Author

Addressed the review in aa00b52:

  1. Content-Length — the handler now rejects a nonzero Content-Length with 400 InvalidHeaderValue, since Put Block From URL carries no request body, with a test issuing a raw request with a body.
  2. x-ms-source-if-tags — deliberately not forwarded, with a comment in the handler explaining why: unlike the Copy Blob operations, Put Block From URL has no source tags condition. The field appears in the shared SourceModifiedAccessConditions TypeScript interface, but blockBlobStageBlockFromURLOperationSpec has no Parameters.sourceIfTags, matching the service REST contract, so the deserializer never populates it for this operation.
  3. SSRF via Host — the loopback fetch is no longer built from the caller-supplied URL. The handler takes the actual bound address and port from the request's connection socket and fetches scheme://127.0.0.1:<localPort> plus only the caller's path and query, so a forged Host header can no longer point the fetch at an arbitrary URL. The same-instance Host comparison remains for the 404-on-foreign-source behavior that copyFromURL has.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Comment thread src/blob/handlers/BlockBlobHandler.ts
Comment thread src/blob/handlers/BlockBlobHandler.ts
Copilot AI review requested due to automatic review settings July 25, 2026 22:59
@gaul

Copy link
Copy Markdown
Contributor Author

Second round addressed in 381b126: the source fetch now pins to the local address the request arrived on (bracketing IPv6 literals) instead of hard-coded 127.0.0.1, so non-loopback --blobHost binds work; and malformed x-ms-source-range values now fail with 400 InvalidHeaderValue up front instead of silently staging the whole source, with tests for both the malformed-range shapes and the reversed-bounds case.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comment thread src/blob/handlers/BlockBlobHandler.ts Outdated
Copilot AI review requested due to automatic review settings July 25, 2026 23:05
@gaul

Copy link
Copy Markdown
Contributor Author

Addressed in 8847aec: the pinned loopback fetch now sends the original source URL host as the Host header — so product-style sources resolve their account through blobStorageContext.middleware exactly as a direct request would — while the TCP connection stays pinned to the server's bound address. Added an end-to-end test where both the destination request and the copy source use product-style account.localhost URLs.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comment thread src/blob/handlers/BlockBlobHandler.ts Outdated
@gaul

Copy link
Copy Markdown
Contributor Author

Addressed in e665103: the same-instance comparison now lowercases the client-supplied Host header before comparing against the already-lowercased URL host, with a mixed-case Host regression test.

Copilot AI review requested due to automatic review settings July 25, 2026 23:11

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

@gaul

Copy link
Copy Markdown
Contributor Author

Squashed after AI review succeeded.

@jainakanksha-msft

Copy link
Copy Markdown
Member

Andrew Gaul (@gaul) , could you please refresh your PR with main, and address the review comments if any to move this PR forward.

Copilot AI review requested due to automatic review settings August 13, 2026 16:41
@gaul

Copy link
Copy Markdown
Contributor Author

Andrew Gaul (Andrew Gaul (@gaul)) , could you please refresh your PR with main, and address the review comments if any to move this PR forward.

Done.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/blob/handlers/BlockBlobHandler.ts:431

  • sourceContentMD5 validation currently compares raw bytes and throws InvalidOperation on mismatch. This diverges from other checksum validation paths in the codebase and fails to return the expected InvalidMd5 (wrong length) / Md5Mismatch (value mismatch) errors.

Consider validating that sourceContentMD5 is exactly 16 bytes and, on mismatch, throwing StorageErrorFactory.getMd5Mismatch(...) with base64-encoded values (same behavior as computeAndValidateTransactionalChecksums).

    if (options.sourceContentMD5 !== undefined) {
      if (
        !Buffer.from(options.sourceContentMD5).equals(calculatedContentMD5)
      ) {
        throw StorageErrorFactory.getInvalidOperation(

Copilot AI review requested due to automatic review settings August 13, 2026 17:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/blob/handlers/BlockBlobHandler.ts:399

  • Range validation uses Number.parseInt(...), which loses integer precision above 2^53-1. For very large x-ms-source-range values this can mis-evaluate the end < start check and allow an invalid range through; the download path then ignores the malformed Range header and would silently stage the entire source blob. Use BigInt for the comparison so the validation works for the full 64-bit range syntax Azure accepts.
      const rangeMatch = /^bytes=(\d+)-(\d*)$/.exec(options.sourceRange);
      if (rangeMatch === null ||
        (rangeMatch[2] !== "" &&
          Number.parseInt(rangeMatch[2], 10) <
          Number.parseInt(rangeMatch[1], 10))) {

@gaul

Copy link
Copy Markdown
Contributor Author

Suppressed comments (1)

Done although 2^53 range is unreasonable for a testing framwork.

Copilot AI review requested due to automatic review settings August 21, 2026 18:12

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Comment thread src/blob/handlers/BlockBlobHandler.ts Outdated
Copilot AI review requested due to automatic review settings August 21, 2026 18:34

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Comment thread src/blob/handlers/BlockBlobHandler.ts Fixed
@jainakanksha-msft
Akanksha Jain (jainakanksha-msft) dismissed their stale review August 24, 2026 06:40

CodeQL comment is correct

Stage the block by fetching the copy source over loopback so that SAS
authentication, x-ms-source-range, and the x-ms-source-if-* conditions
are enforced by the existing download path, then persist it through
the same extent flow as Put Block.  Only sources on the same Azurite
instance are supported, matching copyFromURL.  The response carries
the MD5 of the staged content and source condition failures return
412 SourceConditionNotMet as on the real service.

Reject malformed x-ms-source-content-md5 and x-ms-source-content-crc64
up front so the errors name the header the caller sent, then compare
the surviving values against the fetched bytes with
computeAndValidateTransactionalChecksums, the same path Put Block
uses.  Mismatches return Md5Mismatch or Crc64Mismatch, and supplying
both returns BothCrc64AndMd5HeaderPresent.  The response reports the
computed CRC64 when no MD5 was supplied, as stageBlock does.

Request the source body verbatim and never decompress it.  A blob's
Content-Encoding is stored metadata rather than a description of the
wire framing, so the download echoes it back over the raw stored
bytes; decoding here would stage the decompressed content instead of
what the source holds, and would fail outright when the property does
not match the bytes (issue Azure#646).

The source fetch is pinned to the address and port the request arrived
on, so over HTTPS it presents whatever certificate Azurite was started
with.  Skip verification for that self-request, which would otherwise
reject the self-signed certificates Azurite is normally run with and
make the operation unusable under --cert/--key, and translate
transport-level failures into CannotVerifyCopySource rather than
letting them escape as a bodiless 500.  The same applies to a body
that fails partway through being read.

Validated with the blockblob test suite and end to end with S3Proxy's
native multipart part copy, which previously fell back to streamed
emulation on Azurite's 501.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings August 24, 2026 18:24
Replace rejectUnauthorized: false with an https.Agent that trusts exactly
the certificate this server presents, read from the socket the request
arrived on, so a substituted certificate is still rejected.  Hostname
verification stays off because the request targets the bound address
rather than a name.  Addresses the CodeQL disabling-certificate-validation
alert.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

src/blob/handlers/BlockBlobHandler.ts:33

  • The LOOPBACK_HTTPS_AGENT comment is internally inconsistent (“handshake per request” vs “over an idle socket … mid-reuse”), which makes it hard to understand why keep-alive is intentionally disabled here. Reword to clearly state the intent (shared Agent to avoid reallocation; keep-alive disabled to avoid idle socket reuse).
/**
 * Agents for the loopback self-request stageBlockFromURL makes to read a copy
 * source, keyed by the certificate they pin. Shared so requests reuse one
 * Agent rather than allocating their own, not for socket reuse: keep-alive
 * stays off, taking a loopback handshake per request over an idle socket that

Comment thread src/blob/handlers/BlockBlobHandler.ts Outdated
Copilot AI review requested due to automatic review settings August 24, 2026 18:31
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Copilot AI review requested due to automatic review settings August 24, 2026 18:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

@jainakanksha-msft
Akanksha Jain (jainakanksha-msft) merged commit 422ed99 into Azure:main Aug 25, 2026
45 checks passed
@gaul
Andrew Gaul (gaul) deleted the stage-block-from-url branch August 25, 2026 15:22
Andrew Gaul (gaul) added a commit to gaul/s3proxy that referenced this pull request Aug 25, 2026
The azureblob store copies a part with Put Block From URL, which no
Azurite implemented: the first attempt came back 501, the store
remembered that, and every UploadPartCopy afterwards fell back to
streaming the range through s3proxy.  Azure/Azurite#2681 implements the
operation, source range and x-ms-source-if-* conditions included, and
it sits past the startFrom commit this pin already named, so move the
pin on to it rather than build twice.

The lane takes the native path now: the multipart copy tests stage
their blocks from a URL, and a condition the source does not meet comes
back 412 SourceConditionNotMet, which is the 412 s3proxy owes the
client.  The bump also picks up Azure/Azurite#2680, which authorizes a
classic copy's source with a HEAD instead of downloading the whole blob
only to discard it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants