Skip to content

Make chunk_size_mb control browser upload chunks - #132

Merged
jR4dh3y merged 3 commits into
hoplite/euhesperides-d9702ac5from
hoplite/euhesperides-d9702ac5--chunk-size
Oct 1, 2026
Merged

jR4dh3y merged 3 commits into
hoplite/euhesperides-d9702ac5from
hoplite/euhesperides-d9702ac5--chunk-size

Conversation

@usehoplite

@usehoplite usehoplite Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Problem

chunk_size_mb is documented as a setting but nothing reads it. The handler stored the value and never used it, and the browser always sent 10 MiB chunks. The default also disagreed with itself: 5 in the config loader, 10 in the code constant, and 5 in the bundled config.yaml. An operator behind a proxy with a small request-body limit had no way to shrink upload chunks.

Change

  • Add GET /api/v1/system/upload, which returns { "chunkSizeBytes": N } from chunk_size_mb.
  • The upload store reads it before the first upload and passes it to the existing chunkSize option. Workers that start together share one request, so they use the same size. If the request fails, they keep the 10 MiB fallback and the next upload asks again. An upload cancelled while the request is pending never starts.
  • Use one default, DefaultChunkSizeMB (10). The bundled config.yaml now says 10, so default deployments keep the chunk size browsers sent before.
  • Remove the unused chunk size from StreamHandler.
  • Update docs/api.md, docs/configuration.md and the changelog.

An existing deployment that copied the old config.yaml with chunk_size_mb: 5 now uploads in 5 MiB chunks. That is the value they set, but it doubles the request count for large files.

Verification

  • New handler test covers a configured value and the unset fallback.
  • go vet ./..., handler, config and cmd tests pass. bun run check, lint and build pass.
  • Ran the real server with chunk_size_mb: 3. GET /api/v1/system/upload returned {"chunkSizeBytes":3145728} with a token and 401 without one.
  • In a real browser against the stack-tip build with chunk_size_mb: 3, a 8,000,000-byte upload called GET /api/v1/system/upload and then sent 3 chunk requests (200, 200, 201). The stored file's SHA-256 matched the source.
  • Three files uploaded together made one GET /api/v1/system/upload, and all three stored files matched.
  • Not tested: a deployment behind a proxy with a small body limit, and the fallback path when GET /system/upload fails.

Stack: 2 of 6. Targets the branch of the PR below it.


Written by anthropic/claude-sonnet-5-5 in the Hoplite agent harness.

Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>
@vercel

vercel Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
box-box Ready Ready Preview Oct 1, 2026 8:02am UTC

@coderabbitai

coderabbitai Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: a3206309-c8a4-4569-b15c-f3803a24c7eb

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Comment on lines +131 to +138
private async loadChunkSize(): Promise<void> {
if (this.chunkSizeLoaded) return;
try {
this.chunkSize = (await getUploadConfig()).chunkSizeBytes;
this.chunkSizeLoaded = true;
} catch {
// Retry on the next upload.
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Concurrent uploads use different chunk sizes

When uploads start together, each worker fetches the chunk setting separately. If one request fails while another succeeds, the first upload uses the 10 MiB fallback even though the server specifies a smaller size. If a proxy limits request bodies below 10 MiB, that upload fails while the other can succeed. Share the in-flight configuration request so the workers use the same result.

Knowledge Base Used: Frontend data and real-time state

const controller = new AbortController();
this.controllers.set(item.uploadId, controller);

await this.loadChunkSize();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Cancellation waits for configuration

If a user cancels while the configuration request is pending, that request continues and keeps a worker slot occupied. When it finishes, processItem starts the upload flow without checking whether the signal was aborted. The subsequent progress and error updates can replace the upload’s cancelled state and report it as failed.

Knowledge Base Used: Frontend data and real-time state

@jR4dh3y
jR4dh3y added this pull request to stack #135 October 1, 2026 07:40
usehoplite Bot and others added 2 commits October 1, 2026 07:59
…s-d9702ac5--chunk-size

Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>
Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>
@jR4dh3y
jR4dh3y merged commit 897426d into staging/main Oct 1, 2026
8 checks passed
jR4dh3y added a commit that referenced this pull request Oct 1, 2026
* Fix auto_discover mounts being dropped at startup (#131)

* Fix auto_discover mounts being dropped at startup

Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>

* Make the auto_discover test pick a mounted directory and check it is returned

Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>

---------

Co-authored-by: usehoplite[bot] <288093033+usehoplite[bot]@users.noreply.github.com>
Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>

* Make chunk_size_mb control browser upload chunks (#132)

* Make chunk_size_mb control browser upload chunks

Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>

* Share one chunk size request across upload workers and respect cancel

Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>

---------

Co-authored-by: usehoplite[bot] <288093033+usehoplite[bot]@users.noreply.github.com>
Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>

* Resolve virtual paths in one place (#133)

Co-authored-by: usehoplite[bot] <288093033+usehoplite[bot]@users.noreply.github.com>
Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>

* Parse WebSocket and job data at the boundary (#134)

* Parse WebSocket and job data at the boundary

Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>

* Validate job progress range, add parser tests, log rejected frames briefly

Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>

---------

Co-authored-by: usehoplite[bot] <288093033+usehoplite[bot]@users.noreply.github.com>
Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>

* Move frontend state to runes and split the browse page (#136)

* Migrate jobs and websocket stores to runes

Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>

* Migrate settings store to runes and split out appearance helpers

Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>

* Migrate auth store to runes

Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>

* Split the browse page into composables

Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>

* Keep the WebSocket reconnect backoff when connect runs in an effect

Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>

---------

Co-authored-by: usehoplite[bot] <288093033+usehoplite[bot]@users.noreply.github.com>
Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>

* Fix the auth API docs, drop stale comments, add AGENTS.md (#137)

* Fix the auth API docs, drop stale spec comments, add AGENTS.md

Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>

* Add the frontend test command to AGENTS.md

Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>

* Correct the logout docs and scope the path rule

Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>

---------

Co-authored-by: usehoplite[bot] <288093033+usehoplite[bot]@users.noreply.github.com>
Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>

* Fix --dev failing config validation (#139)

* Fix --dev failing config validation

Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>

* Isolate the dev credentials test from inherited BoxBox settings

Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>

---------

Co-authored-by: usehoplite[bot] <288093033+usehoplite[bot]@users.noreply.github.com>
Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>

* Test the WebSocket reconnect backoff (#138)

* Test the WebSocket reconnect backoff

Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>

* Keep the connection mode on WebSocket retries and test the attempt limit

Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>

---------

Co-authored-by: usehoplite[bot] <288093033+usehoplite[bot]@users.noreply.github.com>
Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>

* Fix browsing discovered mount points

Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>

* Skip inaccessible auto-discovered mounts

Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>

---------

Co-authored-by: usehoplite[bot] <288093033+usehoplite[bot]@users.noreply.github.com>
Co-authored-by: Radhey Kalra <radheykalra901@gmail.com>

This branch was successfully deployed

1 active deployment
Preview — 20056e18 Deployed Oct 1, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant