docs: navigation overhaul, AI-agent access, and design pass

.claude/CLAUDE.md

@@ -24,4 +24,6 @@ uv tool install md-snakeoil && snakeoil --line-length 88 --rules "E,F,B,C4,ISC,P
 
 - @docs/README.md — full command reference, content structure, and links to detail docs
 - @docs/content.md — writing conventions, code blocks, linking, and assets
+- @docs/design-system.md — visual language: tokens, logo, nav, search, diagrams; read before any CSS/nav/visual change
+- @docs/captures.md — UI screenshots and GIFs: where the pixels come from, browser session, scoping rules, helpers
 - @docs/caveats/README.md — known gotchas; add new ones as separate files in this folder

.claude/docs/README.md

@@ -35,4 +35,6 @@ Do not write prose in API reference pages in this repo — edit the docstrings i
 ## More
 
 - @docs/content.md — writing conventions, Python code blocks, linking, assets
+- @docs/design-system.md — visual language: tokens, logo, nav, search, diagrams
+- @docs/captures.md — how UI screenshots and GIFs are taken: cluster, browser session, scoping, crop and GIF helpers
 - @docs/caveats/README.md — known gotchas; add new ones as separate files

.claude/docs/capture_crop.py

@@ -0,0 +1,43 @@
+"""Crop a viewport screenshot to a rectangle measured in the page.
+
+Usage:
+    capture_crop.py <viewport.png> <out.png> '<rect json>' [--margin F] [--top-min Y]
+
+The rect is the JSON an `agent-browser eval` returns from getBoundingClientRect
+plus the viewport width: {"x", "y", "w", "h", "iw"}. The device scale is
+derived from the screenshot width divided by "iw", so any viewport works.
+--margin is a fraction of the rectangle added on every side (0.03 for a card,
+0.18 for a modal). --top-min clamps the crop's top edge in css pixels, used to
+keep a sticky header out of the shot.
+"""
+
+import argparse
+import json
+
+from PIL import Image
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("src")
+    ap.add_argument("out")
+    ap.add_argument("rect")
+    ap.add_argument("--margin", type=float, default=0.0)
+    ap.add_argument("--top-min", type=float, default=None)
+    a = ap.parse_args()
+    rect = json.loads(a.rect)
+    im = Image.open(a.src)
+    dpr = im.width / rect["iw"]
+    x, y, w, h = rect["x"], rect["y"], rect["w"], rect["h"]
+    mx, my = w * a.margin, h * a.margin
+    x0, y0, x1, y1 = x - mx, y - my, x + w + mx, y + h + my
+    if a.top_min is not None:
+        y0 = max(y0, a.top_min)
+    box = tuple(round(v * dpr) for v in (max(0, x0), max(0, y0), x1, y1))
+    box = (box[0], box[1], min(im.width, box[2]), min(im.height, box[3]))
+    im.crop(box).save(a.out)
+    print(a.out, Image.open(a.out).size)
+
+
+if __name__ == "__main__":
+    main()

.claude/docs/capture_gif.py

@@ -0,0 +1,55 @@
+"""Assemble a GIF from viewport screenshots of successive UI states.
+
+Usage:
+    capture_gif.py <frames dir> <out.gif> <margin fraction> [bottom margin fraction]
+
+The directory holds pairs frame_NN.png (a viewport screenshot) and
+rect_NN.json (the card's rectangle in that frame: x, y, w, h, iw, plus a
+"state" label; a frame without a state is skipped). Every frame is cropped
+around the largest rectangle seen, so the card can move between frames,
+halved to 1x and quantised. The first frame holds longer, the last longest.
+"""
+
+import glob
+import json
+import os
+import sys
+
+from PIL import Image
+
+
+def main() -> None:
+    d, out, margin = sys.argv[1], sys.argv[2], float(sys.argv[3])
+    bottom = float(sys.argv[4]) if len(sys.argv) > 4 else margin
+    frames = []
+    for rf in sorted(glob.glob(os.path.join(d, "rect_*.json"))):
+        with open(rf) as fh:
+            r = json.load(fh)
+        if not isinstance(r, dict) or r.get("state") is None:
+            continue
+        p = rf.replace("rect_", "frame_").replace(".json", ".png")
+        if os.path.exists(p):
+            frames.append((p, r))
+    if not frames:
+        raise SystemExit("no frame_NN.png / rect_NN.json pairs with a state")
+    first = Image.open(frames[0][0])
+    dpr = first.width / frames[0][1]["iw"]
+    w = max(r["w"] for _, r in frames)
+    h = max(r["h"] for _, r in frames)
+    mx, my, mb = w * margin, h * margin, h * bottom
+
+    def box(r: dict) -> tuple[int, ...]:
+        return tuple(round(v * dpr) for v in (r["x"] - mx, r["y"] - my, r["x"] + w + mx, r["y"] + h + mb))
+
+    imgs = []
+    for p, r in frames:
+        im = Image.open(p).convert("RGB").crop(box(r))
+        im = im.resize((im.width // 2, im.height // 2), Image.LANCZOS)
+        imgs.append(im.quantize(colors=128, method=Image.Quantize.MEDIANCUT))
+    durations = [1500] + [900] * max(0, len(imgs) - 2) + ([2500] if len(imgs) > 1 else [])
+    imgs[0].save(out, save_all=True, append_images=imgs[1:], duration=durations, loop=0, optimize=True)
+    print(len(imgs), "frames", imgs[0].size, os.path.getsize(out) // 1024, "KB")
+
+
+if __name__ == "__main__":
+    main()

.claude/docs/captures.md

@@ -0,0 +1,83 @@
+# UI Captures
+
+How screenshots and GIFs of the Hopsworks app are taken for the docs.
+Diagrams are a different species, see the Diagrams section of `design-system.md`; this file is only about pixels of the real product.
+Read it before replacing a screenshot, and follow `parity-review.md` for which pages still carry old-UI captures.
+
+## Where the pixels come from
+
+A dev cluster running the target release, never a mock-up and never an older release than the docs version.
+The demo project should look like a real project (a `fraud_detection` style project with feature groups, a deployment, an app, a couple of jobs) so that lists are not empty and names read as real.
+When the state a page describes does not exist, fake it on the cluster or in the browser rather than drawing it: create the object through the SDK or the REST API, stub a response with `agent-browser network route`, or write rows straight into the metadata database.
+Leave harmless fixtures in place for the next agent and write them down in memory; revert anything that changes cluster behaviour (admin variables, auth toggles, Helm values).
+
+## Browser session
+
+Use `agent-browser` with a dedicated session and profile directory, headed, ignoring the self-signed certificate:
+
+```bash
+agent-browser --session hopsdocs --profile "$SCRATCH/chrome-profile" --headed --ignore-https-errors open https://<app>/
+agent-browser --session hopsdocs set viewport 1440 900 2
+```
+
+Never capture from the user's personal Chrome profile: its active tab moves under you.
+The viewport is 1440 css wide at device scale 2, so every capture is 2x; raise the height (1440 by 1500) when a card must fit in one shot.
+A restart of the app (Helm, Payara) logs the session out; ask the user to log in again in the headed window rather than storing credentials.
+
+Read the page with `eval`, click with native `click` on an element you gave an id to in a prior `eval`, and use `find role button click --name "..."` for primary buttons that ignore untrusted clicks (the wizard "Next" buttons do).
+Radix radio groups switch on a click of their `label`, or with arrow keys after focusing a radio.
+Custom dropdowns: click the combobox input, then click the `[role=option]`.
+
+## What a capture shows
+
+Scope the capture to the panel the prose talks about, with some UI around it so the reader sees it is inside the app: never a full app shell, never a bare widget on white.
+Rules that have held on every page so far:
+
+- A form or card: the card from its header to the last relevant block, plus about 3 percent margin on each side; clamp the top to the sticky header's bottom edge so the header never bleeds in.
+- A row range inside a long form (a settings block): cut at the midpoint of the gap to the neighbouring rows so no neighbour is sliced.
+- A modal: the dialog with an 18 percent margin so the blurred page behind it shows it is a modal.
+- A list: the toolbar (primary button and filter) and the table, nothing else.
+- Blur the focused control before shooting (`document.activeElement.blur()`), or the caret and focus ring end up in the docs.
+- Fill placeholders with values that read as real (`transactions`, `fraud_source`, a cursor field, a job name), never `test` or `asdf`.
+
+Measure the rectangle in the page, take a viewport screenshot, and crop:
+
+```bash
+R=$(agent-browser --session hopsdocs eval '(()=>{const e=document.querySelector("#the-card");e.scrollIntoView({block:"start"});window.scrollBy(0,-60);const r=e.getBoundingClientRect();return JSON.stringify({x:r.x,y:r.y,w:r.width,h:r.height,iw:innerWidth})})()' | tail -1)
+R=${R#\"}; R=${R%\"}; R=${R//\\/}
+agent-browser --session hopsdocs screenshot /tmp/vp.png
+python3 .claude/docs/capture_crop.py /tmp/vp.png docs/assets/images/<section>/<name>.png "$R" --margin 0.03 --top-min 72
+```
+
+`capture_crop.py` derives the device scale from the rectangle's `iw`, so it works for any viewport.
+The rectangle is measured after scrolling, because element screenshots lose their target when React re-renders.
+
+## GIFs
+
+A GIF is for a sequence the reader would otherwise have to imagine (a server starting, a job moving through states).
+Record one viewport screenshot per state and, next to it, the rectangle of the card in that frame (the card moves when a sidebar collapses), then assemble:
+
+```bash
+python3 .claude/docs/capture_gif.py "$FRAMES_DIR" docs/assets/images/<section>/<name>.gif 0.05
+```
+
+Frames are cropped around the largest card rectangle, halved to 1x and quantised, so a five-frame GIF stays under a few hundred kilobytes.
+The first frame holds longer, the last one longest.
+
+## Naming and placement
+
+Images live in `docs/assets/images/<section>/` mirroring the page (`guides/fs/feature_group/`, `guides/jobs/`, `admin/oauth2/`).
+Keep the existing file name when replacing an old capture so no page reference changes; delete captures you cannot redo rather than leaving the old UI in, and drop their figure from the page.
+Alt text says what the reader is looking at, one sentence, no "screenshot of".
+
+## Faking states, worked examples
+
+- A feature the cluster does not enable: stub the availability check in the browser, `agent-browser network route "**/isAvailable*" --body '{"enabled":true}'`, then create the objects through the API.
+- Feature monitoring results: register synthetic statistics per commit window through the SDK so the chart has distinct points.
+- A data source that browses tables: the built-in HopsFS and JDBC sources never enable "Next: Select Tables"; create a `SQL` source against the cluster's own MySQL (`mysqld.hopsworks.svc.cluster.local`) with a dedicated read-only user and a small database of realistic tables.
+- Session-capacity badges, alerts, admin panels: the memory file for the dev cluster lists the switches; check it before searching.
+
+## Before you are done
+
+Open the page on the served site and look at the capture in the column: if the text is not readable at the docs width, the scope was too wide, not the resolution too low.
+Tick the page in `parity-review.md` with a note on what was redone.

.claude/docs/caveats/diagram-inside-content-tabs.md

@@ -0,0 +1,5 @@
+# Diagram fragments inside content tabs
+
+A `--8<--` diagram include placed inside a `=== "Tab"` block is re-parsed as Markdown by the tabbed extension, unlike a top-level include which passes through as one raw HTML block. Two things break: a blank line inside the fragment ends the HTML block, so the SVG's inner tags render as paragraph text; and a `$` inside the scene JSON (`$label`, `$ms`) is picked up by the arithmatex math extension, which injects a `<span>` into the script and the scene fails to parse (no toggle, no step bar).
+
+Write tabbed fragments with no blank lines, and spell the scene's dollar keys as JSON unicode escapes, `"\u0024label"` and `"\u0024ms"`, which decode to the same keys. The flywheel figures on the AI Systems page are the reference. Top-level includes need neither.

.claude/docs/caveats/linked-tabs-text-transform.md

@@ -0,0 +1,5 @@
+# Linked tabs and CSS text-transform
+
+Material links content tabs (`content.tabs.link`) by comparing label text with `innerText`. `innerText` carries CSS text transforms on rendered elements but not on hidden ones, so a label uppercased by CSS reads "PYTHON" where it is visible and "Python" in every hidden set, and a switch made on the visible set never reaches the others (only labels that are already uppercase, like "CLI", keep working). The home stepper hit this: switching on step 3 left steps 1 and 2 on the old tab.
+
+Never put `text-transform` on a linked tab label. Write the label text as it should display and style the rest (font, weight, size).

.claude/docs/caveats/tabs-inside-raw-html.md

@@ -0,0 +1,5 @@
+# Content tabs inside raw HTML blocks
+
+A `=== "Tab"` set placed in a `<div markdown>` that sits inside an outer raw HTML block without its own `markdown` attribute renders as literal text: md_in_html only parses nested `markdown` divs when the outermost raw block carries the attribute too. Code fences still render there because superfences is a preprocessor, so the breakage is easy to miss (the home stepper looked fine until tabs went in).
+
+Put `markdown` on the outermost wrapper as well (`<div class="hops-steps" markdown>`); raw children without the attribute, such as the stepper's button rail, pass through untouched.

.claude/docs/caveats/zoom-overlay-clone-scoping.md

@@ -0,0 +1,7 @@
+# Zoom overlay clones escape .md-typeset scoping
+
+The diagram zoom overlay (`diagram-zoom.js`) clones the whole figure into `document.body`, outside `.md-typeset`.
+Any CSS scoped under `.md-typeset` (design tokens especially) stops matching the clone, and a failed `var()` in an SVG `fill` computes to black, so the zoomed diagram renders as black boxes while the inline one looks fine.
+
+Scope diagram tokens and component rules to the figure class alone (`.hops-viz`, `.hops-diagram`), never through `.md-typeset`, and pin theme-dependent values inside `.hops-zoom-stage` because the overlay panel is always dark whatever the page scheme.
+Animated figures get restarted on the clone by `hops-viz.js` listening for the `hops-zoom-open` event; keep that event dispatch when touching `diagram-zoom.js`.

.claude/docs/content.md

@@ -28,3 +28,24 @@ Avoid relative file paths as links (e.g. `../other.md`) — they break after mik
 ## Assets
 
 Images go in `docs/assets/images/<section>/` matching the section of the content that uses them.
+
+## API reference box
+
+A guide ends its code walkthrough (or each walkthrough section, when a page has several) with an `api` admonition, never a bare `### API Reference` heading with loose links.
+Rows are the mkdocstrings symbol badge plus the autoref, exactly what the reader lands on in the API section: entry-point methods first, then each class with the methods the guide called nested under it, then any external doc with a muted `docs` badge.
+No prose per row.
+The last line is the single CTA into the Python API, with the `../` depth matching the page URL.
+
+```markdown
+!!! api "API reference"
+
+    - <code class="doc-symbol doc-symbol-method"></code> [`Project.get_kafka_api`][hopsworks_common.project.Project.get_kafka_api]
+    - <code class="doc-symbol doc-symbol-class"></code> [`KafkaApi`][hopsworks_common.core.kafka_api.KafkaApi]
+        - <code class="doc-symbol doc-symbol-method"></code> [`create_topic`][hopsworks_common.core.kafka_api.KafkaApi.create_topic]
+    - <code class="doc-symbol doc-symbol-docs"></code> [Kafka docs](https://kafka.apache.org/documentation/)
+
+    <a class="hops-api-cta" href="../../../../python-api/hopsworks/">Browse the full Python API :material-arrow-right:</a>
+```
+
+Badges: `class`, `method`, `function`, `attribute`, `docs`.
+The `python-api/` root is not a page; link to `python-api/hopsworks/`.