Load public GitHub repositories by URL through a Cloudflare Worker

.gitignore

@@ -27,6 +27,7 @@ logs
 .env.*
 !.env.example
 *.local
+.dev.vars*
 
 # Editors
 .idea

.oxlintrc.json

@@ -457,10 +457,16 @@
   // React Three Fiber renders three.js objects as JSX, so its props
   // (position, args, intensity, attach, ...) are not DOM attributes. Scoped to
   // the renderer package and the app's Canvas only.
+  // A module Worker is its default export ({ fetch }); Cloudflare finds no
+  // handler otherwise. Scoped to the one entry file wrangler.jsonc names.
   "overrides": [
     {
       "files": ["packages/renderer/src/**/*.tsx", "apps/web/src/app.tsx"],
       "rules": { "react/no-unknown-property": "off" }
+    },
+    {
+      "files": ["apps/web/src/api/worker.ts"],
+      "rules": { "import/no-default-export": "off" }
     }
   ]
 }

AGENTS.md

@@ -10,9 +10,9 @@ Guidelines for AI agents and contributors working in this Turborepo monorepo.
 
 ### Apps (`apps/`)
 
-| Name | Filter | Description                                                                                      |
-| ---- | ------ | ------------------------------------------------------------------------------------------------ |
-| web  | `web`  | Code Backrooms: Vite 8 + React 19 + React Three Fiber first-person app, deployable to Cloudflare |
+| Name | Filter | Description                                                                                                                |
+| ---- | ------ | -------------------------------------------------------------------------------------------------------------------------- |
+| web  | `web`  | Code Backrooms: Vite 8 + React 19 + React Three Fiber first-person app, and its Cloudflare Worker that fetches GitHub zips |
 
 ### Packages (`packages/`)
 

README.md

@@ -131,7 +131,8 @@ push to `main` and on PRs, plus CodeQL code scanning. `secrets:scan` needs eithe
 - JavaScript and TypeScript only.
 - Up to 400 source files and 8 MB per repository.
 - Calls through objects (`obj.method()`) and callbacks show as closed doors.
-- Loading straight from a GitHub URL is not supported yet.
+- GitHub URLs load public repositories only, and branch names containing `/` are not
+  supported.
 
 ## License
 

apps/web/AGENTS.md

@@ -169,34 +169,76 @@ not be laid out`. The HUD reads `2/7 modules · last 14 ms` on the way;
   local files never writes `?seed=`, since no URL can name those files. The
   filter keeps a superset of the `?code=self` glob (config modules such as
   `vite.config.ts`), so a zip of this repository has a few more modules
-  than Demo. Fetching a GitHub URL is not supported; it would need a
-  Cloudflare Worker proxy (`codeload` sends no CORS headers) feeding
-  `zip-files.ts`.
-- Milestone 21 (current): a moving landing page. Behind the panel the
+  than Demo. A public GitHub repository loads by URL (Milestone 22).
+- Milestone 21: a moving landing page. Behind the panel the
   bundled `demo` example's entrance is drawn with a camera that drifts
   toward its first doorway and back while its gaze sways
   (`landing-backdrop.tsx`, `driftPose` in `backdrop.ts`). It is skipped
   without WebGL2 (it runs under `prefers-reduced-motion` too), swallowed by an error
   boundary if the canvas fails, and dropped as soon as a read starts. The
   world view (`world-app.tsx`) and the backdrop are lazy chunks, so the
   landing page paints before three.js loads.
+- Milestone 22 (current): a public GitHub repository by URL. The landing
+  page takes `github.com/owner/repo`, `owner/repo` or a
+  `/tree/<ref>/<folder>` URL (`github-url.ts`; refs with `/` are not
+  supported). One load is one request to GitHub: the page calls this site's
+  Worker (`api/`, `GET /api/github/zip`), which fetches the repository's
+  zip from codeload.github.com, since codeload sends no CORS headers, and
+  streams it back. The page then reads it with `filesFromZip`, keeping only
+  the folder's files when one is named. The Worker builds the codeload URL
+  from checked names and is not an open proxy. Before it fetches, the
+  visitor must pass a Cloudflare Turnstile check (`turnstile.tsx`, loaded
+  only once a URL is typed). Each IP is rate-limited to 5 loads a minute.
+  Answers are edge-cached for 10 minutes, and zips over 64 MB are refused or
+  cut off. A loaded repository writes `?repo=` so a reload or a shared link
+  fills the field in again. The visitor still presses open, because each
+  Turnstile token is single-use. Without a site key at build time the field
+  is hidden, so production never falls back to the test key.
 
 ---
 
 ## Workspace Commands
 
-| Task      | Command                       |
-| --------- | ----------------------------- |
-| Dev       | `pnpm --filter web dev`       |
-| Build     | `pnpm --filter web build`     |
-| Preview   | `pnpm --filter web preview`   |
-| Typecheck | `pnpm --filter web typecheck` |
-| Test      | `pnpm --filter web test`      |
-| Clean     | `pnpm --filter web clean`     |
-| Deploy    | `pnpm --filter web deploy`    |
+| Task       | Command                        |
+| ---------- | ------------------------------ |
+| Task       | Command                        |
+| ---------- | ------------------------------ |
+| Dev        | `pnpm --filter web dev`        |
+| Dev Worker | `pnpm --filter web dev:worker` |
+| Build      | `pnpm --filter web build`      |
+| Preview    | `pnpm --filter web preview`    |
+| Typecheck  | `pnpm --filter web typecheck`  |
+| Test       | `pnpm --filter web test`       |
+| Clean      | `pnpm --filter web clean`      |
+| Deploy     | `pnpm --filter web deploy`     |
 
 Or run repo-wide via root `pnpm typecheck` / `pnpm test` / `pnpm build`.
 
+### GitHub loading in development
+
+- `dev` alone serves everything except `/api/*`. To load from GitHub, also run
+  `dev:worker` (`wrangler dev` on port 8787), which Vite proxies `/api` to.
+- Dev builds use Turnstile's public always-pass test site key. `wrangler dev`
+  needs the matching test secret in `apps/web/.dev.vars`, which is gitignored:
+  `TURNSTILE_SECRET=1x0000000000000000000000000000000AA`.
+
+### Deploying (Cloudflare Workers Builds, repository connected in the dashboard)
+
+- **Build settings:** set the root directory to the repository root. The build
+  command is `pnpm --filter web build`, and the deploy command is
+  `pnpm --filter web exec wrangler deploy`.
+- **Build variables:** `PNPM_VERSION=12.4.2` and `NODE_VERSION=24.21.0`, since the
+  build image's pnpm default is older. Also set `VITE_TURNSTILE_SITE_KEY` to the
+  widget's site key. Vite inlines it at build time, so it is a _build_ variable.
+- **Runtime secret:** `TURNSTILE_SECRET` under the Worker's Variables and Secrets.
+  `wrangler.jsonc` declares no value for it, so deploys keep it.
+- **Turnstile widget:** create it in the Cloudflare dashboard. Its hostnames must
+  include the production domain, plus the `workers.dev` domain if preview builds
+  are on. A preview without its hostname refuses GitHub loads and everything else
+  still works.
+- **Committed values:** the site key, the rate-limit `namespace_id` and the Worker
+  code are public by design. The secret never goes in the repository.
+
 There are no per-workspace `lint` or `format` scripts: root `pnpm lint` (oxlint) and
 `pnpm format` (Prettier) each cover the whole repo.
 
@@ -210,7 +252,14 @@ src/
   app.tsx             landing page, loading screen, or the world as a lazy chunk
   world-app.tsx       the world: full-screen <Canvas>, seed and current-room
                       state, pointer lock, HUD
-  landing.tsx         the bare URL's page: zip drop and pickers, Demo, examples
+  landing.tsx         the bare URL's page: zip drop and pickers, GitHub URL, Demo, examples
+  github-url.ts       a GitHub URL -> owner, repo, ref, folder; checks shared with the Worker, pure
+  github-files.ts     a GitHub repository -> sources: one call to the Worker, then zip-files
+  github-form.tsx     the landing page's GitHub URL field, check and open button, markup only
+  turnstile.tsx       the Turnstile check before a GitHub load, script loaded on demand
+  api/                the Cloudflare Worker, run only for /api/* (wrangler.jsonc `main`)
+    worker.ts         the entry: routes /api/github/zip, else the static assets
+    github-zip.ts     the route: checks, rate limit, Turnstile, one codeload fetch
   landing-backdrop.tsx  the demo world drifting behind the landing page, lazy
   backdrop.ts         when the backdrop runs and where its camera is, pure
   backdrop-world.ts   the demo world for the backdrop, every area built

apps/web/README.md

@@ -26,6 +26,14 @@ A bare URL opens the landing page. Drop or pick a `.zip` (GitHub's
 tab and never uploaded. Only JavaScript and TypeScript sources and manifests
 are kept, and dependency, build and test directories are skipped.
 
+You can also paste a public GitHub repository URL, such as
+`github.com/owner/repo` or `github.com/owner/repo/tree/main/src` for one
+folder. After a Cloudflare Turnstile check, this site's Worker downloads the
+repository's zip from GitHub in one request and the page reads it like a
+dropped zip. `?repo=owner/repo` fills the field in. Running this locally
+needs `pnpm --filter web dev:worker` beside `dev`. See `AGENTS.md` for that
+and for the deploy settings.
+
 | Limit               | Value  |
 | ------------------- | ------ |
 | Zip size            | 64 MB  |

apps/web/package.json

@@ -5,6 +5,7 @@
   "type": "module",
   "scripts": {
     "dev": "vite --port 3000",
+    "dev:worker": "mkdir -p dist && wrangler dev --port 8787",
     "build": "vite build",
     "preview": "vite preview",
     "deploy": "pnpm build && wrangler deploy",

apps/web/public/_headers

@@ -2,10 +2,12 @@
 # The CSP allows Cloudflare Web Analytics / Browser RUM: the injected beacon script comes from
 # static.cloudflareinsights.com and reports to /cdn-cgi/rum on this origin or cloudflareinsights.com.
 # style-src 'self' is enough: React applies `style` props through the CSSOM, which CSP doesn't block.
+# Cloudflare Turnstile (the check before a GitHub load) runs a script and an iframe from
+# challenges.cloudflare.com. The GitHub zip itself comes from this origin's Worker (/api/*).
 # Loading anything else from another origin (scripts, fonts, APIs) needs an entry here.
 
 /*
-  Content-Security-Policy: default-src 'self'; script-src 'self' https://static.cloudflareinsights.com; connect-src 'self' https://cloudflareinsights.com; img-src 'self' data:; style-src 'self'; font-src 'self'; object-src 'none'; base-uri 'self'; form-action 'self'; frame-ancestors 'none'; upgrade-insecure-requests
+  Content-Security-Policy: default-src 'self'; script-src 'self' https://static.cloudflareinsights.com https://challenges.cloudflare.com; frame-src https://challenges.cloudflare.com; connect-src 'self' https://cloudflareinsights.com; img-src 'self' data:; style-src 'self'; font-src 'self'; object-src 'none'; base-uri 'self'; form-action 'self'; frame-ancestors 'none'; upgrade-insecure-requests
   X-Content-Type-Options: nosniff
   Referrer-Policy: strict-origin-when-cross-origin
   Permissions-Policy: camera=(), microphone=(), geolocation=()

apps/web/src/api/github-zip.test.ts

@@ -0,0 +1,207 @@
+import { describe, expect, it, vi } from "vitest";
+
+import { handleZip as handle } from "./github-zip";
+import type { Fetch, ZipEnv } from "./github-zip";
+
+const SITEVERIFY = "https://challenges.cloudflare.com/turnstile/v0/siteverify";
+
+const env = (overrides: Partial<ZipEnv> = {}): ZipEnv => ({
+  GITHUB_LIMITER: { limit: vi.fn(() => Promise.resolve({ success: true })) },
+  TURNSTILE_SECRET: "secret",
+  ...overrides,
+});
+
+/** Turnstile answers `verified`; codeload answers `zip`. */
+const upstream = (
+  zip: () => Response,
+  verified = true
+): ReturnType<typeof vi.fn<Fetch>> =>
+  vi.fn<Fetch>((url) =>
+    Promise.resolve(
+      url === SITEVERIFY ? Response.json({ success: verified }) : zip()
+    )
+  );
+
+const zipRequest = (
+  query: string,
+  headers: Record<string, string> = { "CF-Turnstile-Token": "token" }
+): Request =>
+  new Request(`https://backrooms.test/api/github/zip?${query}`, {
+    headers: { "CF-Connecting-IP": "203.0.113.7", ...headers },
+  });
+
+const codeloadCalls = (fetcher: ReturnType<typeof vi.fn<Fetch>>): string[] =>
+  fetcher.mock.calls
+    .map(([url]) => url)
+    .filter((url) => url.startsWith("https://codeload.github.com/"));
+
+const errorOf = async (response: Response): Promise<unknown> =>
+  ((await response.json()) as { error: unknown }).error;
+
+describe("the GitHub zip route", () => {
+  it("fetches one zip from codeload, cached, and streams it back", async () => {
+    const fetcher = upstream(() => new Response(new Uint8Array([1, 2, 3])));
+    const response = await handle(
+      zipRequest("owner=pmndrs&repo=zustand&ref=v4.0.0"),
+      env(),
+      fetcher
+    );
+    expect(response.status).toBe(200);
+    expect(response.headers.get("Content-Type")).toBe("application/zip");
+    expect(new Uint8Array(await response.arrayBuffer())).toEqual(
+      new Uint8Array([1, 2, 3])
+    );
+    expect(codeloadCalls(fetcher)).toEqual([
+      "https://codeload.github.com/pmndrs/zustand/zip/v4.0.0",
+    ]);
+    const codeloadInit = fetcher.mock.calls.find(([url]) =>
+      url.startsWith("https://codeload")
+    )?.[1];
+    expect(codeloadInit?.cf).toEqual({ cacheEverything: true, cacheTtl: 600 });
+  });
+
+  it("uses HEAD without a ref", async () => {
+    const fetcher = upstream(() => new Response("zip"));
+    await handle(zipRequest("owner=o&repo=r"), env(), fetcher);
+    expect(codeloadCalls(fetcher)).toEqual([
+      "https://codeload.github.com/o/r/zip/HEAD",
+    ]);
+  });
+
+  it("sends the secret, token and visitor IP to Turnstile", async () => {
+    const fetcher = upstream(() => new Response("zip"));
+    await handle(zipRequest("owner=o&repo=r"), env(), fetcher);
+    const body = fetcher.mock.calls.find(([url]) => url === SITEVERIFY)?.[1]
+      ?.body;
+    expect(body instanceof FormData ? Object.fromEntries(body) : body).toEqual({
+      secret: "secret",
+      response: "token",
+      remoteip: "203.0.113.7",
+    });
+  });
+
+  it.each([
+    "owner=o",
+    "owner=o&repo=r&ref=..",
+    "owner=o&repo=r&ref=a%2Fb",
+    "owner=o%2F..&repo=r",
+    "owner=o&repo=..",
+    "owner=o&repo=r%3Fx%3D1",
+    "owner=evil.com&repo=r",
+  ])("refuses %s before contacting anyone", async (query) => {
+    const fetcher = upstream(() => new Response("zip"));
+    const limiter = env();
+    const response = await handle(zipRequest(query), limiter, fetcher);
+    expect(response.status).toBe(400);
+    expect(fetcher).not.toHaveBeenCalled();
+    expect(limiter.GITHUB_LIMITER.limit).not.toHaveBeenCalled();
+  });
+
+  it("refuses a request without a token, never reaching GitHub", async () => {
+    const fetcher = upstream(() => new Response("zip"));
+    const response = await handle(
+      zipRequest("owner=o&repo=r", {}),
+      env(),
+      fetcher
+    );
+    expect(response.status).toBe(403);
+    expect(fetcher).not.toHaveBeenCalled();
+  });
+
+  it("refuses a token Turnstile rejects, never reaching GitHub", async () => {
+    const fetcher = upstream(() => new Response("zip"), false);
+    const response = await handle(zipRequest("owner=o&repo=r"), env(), fetcher);
+    expect(response.status).toBe(403);
+    expect(codeloadCalls(fetcher)).toEqual([]);
+  });
+
+  it("fails closed when Turnstile cannot be reached", async () => {
+    const fetcher = vi.fn<Fetch>(() => Promise.reject(new Error("offline")));
+    const response = await handle(zipRequest("owner=o&repo=r"), env(), fetcher);
+    expect(response.status).toBe(403);
+    expect(fetcher).toHaveBeenCalledTimes(1);
+  });
+
+  it("refuses when no secret is configured", async () => {
+    const fetcher = upstream(() => new Response("zip"));
+    const response = await handle(
+      zipRequest("owner=o&repo=r"),
+      env({ TURNSTILE_SECRET: undefined }),
+      fetcher
+    );
+    expect(response.status).toBe(503);
+    expect(fetcher).not.toHaveBeenCalled();
+  });
+
+  it("rate-limits by visitor IP before verifying anything", async () => {
+    const fetcher = upstream(() => new Response("zip"));
+    const limited = env({
+      GITHUB_LIMITER: {
+        limit: vi.fn(() => Promise.resolve({ success: false })),
+      },
+    });
+    const response = await handle(
+      zipRequest("owner=o&repo=r"),
+      limited,
+      fetcher
+    );
+    expect(response.status).toBe(429);
+    expect(limited.GITHUB_LIMITER.limit).toHaveBeenCalledWith({
+      key: "203.0.113.7",
+    });
+    expect(fetcher).not.toHaveBeenCalled();
+  });
+
+  it("says a missing repository is not found or private", async () => {
+    const fetcher = upstream(() => new Response("", { status: 404 }));
+    const response = await handle(zipRequest("owner=o&repo=r"), env(), fetcher);
+    expect(response.status).toBe(404);
+    expect(await errorOf(response)).toBe("o/r not found, or it is private");
+  });
+
+  it("passes other GitHub failures on as 502", async () => {
+    const fetcher = upstream(() => new Response("", { status: 503 }));
+    const response = await handle(zipRequest("owner=o&repo=r"), env(), fetcher);
+    expect(response.status).toBe(502);
+  });
+
+  it("refuses a zip whose length is over 64 MB", async () => {
+    const fetcher = upstream(
+      () =>
+        new Response("zip", {
+          headers: { "Content-Length": String(64 * 1024 * 1024 + 1) },
+        })
+    );
+    const response = await handle(zipRequest("owner=o&repo=r"), env(), fetcher);
+    expect(response.status).toBe(413);
+  });
+
+  it("breaks off a zip without a length once it passes 64 MB", async () => {
+    const chunk = new Uint8Array(16 * 1024 * 1024);
+    let sent = 0;
+    const endless = new ReadableStream<Uint8Array>({
+      pull(controller) {
+        sent += 1;
+        controller.enqueue(chunk);
+      },
+    });
+    const fetcher = upstream(() => new Response(endless));
+    const response = await handle(zipRequest("owner=o&repo=r"), env(), fetcher);
+    expect(response.status).toBe(200);
+    await expect(response.arrayBuffer()).rejects.toThrow();
+    expect(sent).toBeLessThanOrEqual(6);
+  });
+
+  it("allows GET only", async () => {
+    const fetcher = upstream(() => new Response("zip"));
+    const response = await handle(
+      new Request("https://backrooms.test/api/github/zip?owner=o&repo=r", {
+        method: "POST",
+      }),
+      env(),
+      fetcher
+    );
+    expect(response.status).toBe(405);
+    expect(fetcher).not.toHaveBeenCalled();
+  });
+});