embed_docsite: replace per-directory GitHub fetch with Trees API + cache

[IZO-Ong]

Short Description

Replaces embed_docsite's per-directory GitHub Contents API walk with a single Trees API call with an on-disk cache keyed by ETag and per-file blob SHA, cutting a full run of ~60 GitHub API requests down to 2.

Fixes #182

Implementation Details

github_utils.py used to fetch docs/ and adaptors/ from OpenFn/docs by recursing into every directory via the Contents API (GET /repos/{owner}/{repo}/contents/{path}) with one request per directory, then downloading each file's download_url separately. On the full corpus that is about one request per directory (~60), which is easy to exhaust GitHub's unauthenticated 60/hr rate limit.

#182 proposed fixing this with git clone --depth 1. This PR takes a different route by using GitHub Trees API (GET /repos/OpenFn/docs/git/trees/main?recursive=1), which returns the entire file tree, path and blob SHA for every file in one request, plus a persistent on-disk cache (services/embed_docsite/docsite_cache/, gitignored) that stores the tree's ETag and each cached file's blob SHA.

• Steady-state cost: a full run (general_docs + adaptor_docs + adaptor_functions) now spends 2 GitHub API requests — one conditional Trees request per markdown docs_type. Measured against the live repo: both come back 304 when nothing changed, and 0 files are re-downloaded. Only files whose blob SHA actually moved get re-fetched, from raw.githubusercontent.com, which is not rate-limited.
• Degradation: if GitHub is unreachable or rate-limited, the cached copy is served and a warning is logged; only a failure with an empty cache is fatal (503 UPSTREAM_ERROR). A cached file missing from disk degrades to an ApolloError instead of an
  unhandled FileNotFoundError.
• Syncs GITHUB_TOKEN (in .env.example) to the docs fetch, raising the limit from 60 to 5000 requests/hour.
• New mode: {"refresh_cache_only": true} warms both caches without needing Pinecone/OpenAI keys, for pre-warming in CI or locally.
• Known limitation: get_repo_tree raises if GitHub reports the tree as truncated (~100,000 files from Github's API). Not a concern at OpenFn/docs' current size (702 files).

Tests: unit tests fully mocked (services/embed_docsite/tests/unit/), plus integration tests over the real cached corpus
(services/embed_docsite/tests/integration/test_docsite_cache_corpus.py, skipped unless the cache is warmed).

AI Usage

Please disclose whether you've used AI in this work (it's cool, we just want to know!):

• Yes, I have used AI
• No, I have not used AI

You can read more details in our
Responsible AI Policy