feat: stateful interactive blocks with formulas and explorable diagrams and state round-trip

[chenyme]

Summary

Interactive components in replies were static: deeix-ui props could only carry data, a value the user changed never reached the model, and a broken node rendered as a blank box. Anything that needed animation, a 3D sketch or a flow field had no home at all, and a suggested follow-up was sent verbatim on click.

1. A block format that can compute and remember (packages/core + apps/web). Blocks gain state, computed and children next to the existing props, plus bind, visible and actions (set, ask, copy). Values are {expression} templates evaluated locally — never eval, a whitelisted interpreter with 30 functions and implicit x / i / row for per-point chart series, where filters and formula/scene primitives (formula renders "expression = substitution = result" and generates its own LaTeX from the parse tree; scene draws shapes from the same values). New packages/core/src/ui-block/ holds the engine: partial-json.ts (any prefix of the JSON is readable, and it reports what was closed and where the text was cut), expression.ts, template.ts, points.ts, primitives.ts, fence.ts, all covered by Node tests including "every streaming prefix of the samples parses and renders". apps/web/shared/lib/safe-expression.ts is replaced by the core interpreter (calculator and function-plot use it through the same numeric interface). Rendering is streaming-tolerant (a node appears when its own closing brace arrives, leaf by leaf, path-keyed so inputs keep focus), and failures are bounded: unknown components and invalid props say so instead of disappearing, a broken value shows "—", and node- and block-level error boundaries keep the rest of the message alive. Dev builds list compile and evaluation problems under the block.

2. State round-trip (front end, backend, prompt). Only the keys the user changed are stored, per (message, block id, state key), debounced for a second and flushed with keepalive when the page is hidden; the current values also travel with the next send, so a change cannot race the debounce. PUT /api/v1/conversations/:id/messages/:message_id/ui-state persists them (new table chat_message_ui_states, one row per message and block, hard delete, message row locked before the block count so the 32-block cap holds under concurrency), and the next request injects the current values for the blocks on the visible branch into the dynamic context as a <ui_state> block — capped at 8 blocks and 4 KB, escaped, and labelled as user-entered data rather than instructions, so it never enters the prefix cache. Fork, delete, export, temporary chats and edits are handled; share pages stay read-only.

3. Inline interactive illustrations (deeix-widget). A second kind of fence whose body is a self-contained HTML fragment the model writes and the host runs in the sandbox that already backs artifacts and custom components (sandbox="allow-scripts", no same origin, SANDBOX_CSP: no network, no forms, all permissions denied). It runs only once its fence is closed (a half-written script would only throw) while the rest of the message keeps streaming, is keyed by its source and titled from <title>/first heading, shares the per-block state store and the next-turn injection, caps at 200 KB (source shown instead) and 900 px, offers "view source" and "open in the side panel", and degrades to the source after 5 s where the host CSP forbids inline scripts (the desktop shell). A block's ask action became a small composer prefilled with the suggestion: the user edits it, sends with the button or Enter, and a reset button brings the suggestion back — nothing is sent from a click the host cannot attribute to the user.

4. Prompt. The component catalog now has a v2 (catalogPromptHeaderLive) that describes state/children/formula/data/actions/primitives, the <widget> section, and a rewritten <when> (use / avoid / choose / followup / check) that prefers the lightest expression (catalog component → panel → widget). v1 clients keep the previous catalog byte for byte: the client declares uiBlockVersion with each send (absent or 1, 2; other values 400), and the routing line is only rewritten for v2. Examples, the function table and the primitive table live in packages/core testdata and are read by the Go tests, so prompt and engine cannot drift.

5. Desktop (macOS): browser-style close shortcuts. The Tauri shell now builds its own menu: ⌘W closes the tab that is showing (the last tab closes the window instead, because closing it would forget its server) and ⇧⌘W closes the window, with File following the tab count as Safari's does. Tauri's default menu bound ⌘W to Close Window, which hid the whole tab container — the behaviour users reported as "the app quit". Two upstream details forced the shape: the predefined Close Window item has a hard-coded accelerator and no setter, and a Cmd+Shift+W accelerator string becomes the unshifted key equivalent w, which AppKit never matches for ⇧⌘W, so the installed item is corrected through AppKit. Other platforms have no menu bar and are unchanged.

Change type

• Bug fix
• Feature
• Documentation
• Refactor
• Configuration / deployment
• Security hardening
• Other

Affected areas

• Frontend / UI
• Backend / API
• Authentication / authorization
• Conversations / streaming
• Files / RAG / extraction
• Model routing / providers
• MCP / tools
• Billing / payments
• Admin console
• Deployment / Docker / configuration
• Documentation

Verification

• Run on the final branch:
  • pnpm --filter @deeix/core test — 119 passed (streaming prefixes of every sample, expressions, templates, points, fences, widget keys).
  • pnpm --filter @deeix/core check, pnpm --filter @deeix/web check (Biome, tsc, architecture rules) — clean.
  • pnpm api:check — Swagger and types.generated.ts are up to date.
  • go test ./internal/application/uicomponent/... ./internal/application/conversation/... — ok (UI state service, repository, handler and injection tests, catalog/example contract tests, prompt tests).
  • cargo fmt --check, cargo clippy --all-targets -- -D warnings, cargo test --lib in apps/desktop/src-tauri — clean, 37 passed.
• Manual in Chrome (Playwright, light and dark): blocks render while streaming and survive every prefix, dragging a slider updates computed values, per-point charts and the formula panel; state survives a reload and comes back in <ui_state> on the next turn; a configuration switch and a timeline drag write state back to the host; the side panel and the inline view stay in sync; copy turns a reply into readable text and tables with the current values; a widget plays its animation and falls back to source when inline scripts are blocked; the .stable illustration no longer shifts its controls when the caption goes from one line to two.
• Desktop: a temporary in-app probe drove the installed native menu items (NSMenu.performActionForItemAtIndex:, the same path a click or key press takes) and confirmed the File items, their key equivalents, the tab-count rewrite, that ⌘W closes the active tab, and that the last tab leaves the window hidden rather than destroyed. AppKit matching was checked separately against NSMenu.performKeyEquivalent.

Screenshots, API examples, or logs

• Screenshots of the loan panel (light and dark), a streaming widget placeholder, the source fallback, and the File menu with one tab versus two.
• Request: PUT /api/v1/conversations/…/messages/…/ui-state with {"blocks": {"loan": {"amount": 300, "years": 20}}}; response returns the full block state of that message.
• Desktop probe output: [probe] 2 tabs: item 0 "Close Tab" key="w" mask=0x100000, item 1 "Close Window" key="W" mask=0x120000, firing item 0 → [tabs] close tab-2, 1 tab again: item 0 "Close Window" key="w", window hidden after the close request.

Configuration, migration, and compatibility notes

• New table chat_message_ui_states, registered with schema.Models() and created by the existing startup migration path; new endpoint plus DTOs, with Swagger and packages/api-contract/src/types.generated.ts regenerated (pnpm api:check passes).
• uiBlockVersion on send requests is optional; absent or 1 keeps the previous catalog and behaviour, so older clients and their prompt caches are unaffected. They see v2 blocks as an unknown fence and fall back to the source.
• Existing messages are untouched: the format is additive, old {"component", "id", "props": …} blocks still parse, and stored state is keyed per block.
• Desktop: macOS-only behaviour change (⌘W / ⇧⌘W); Windows and Linux are unchanged. Widgets and custom components still fall back to source there, because Tauri's script-src 'self' is inherited by the sandbox srcdoc; serving the sandbox document from a custom protocol with its own CSP is the follow-up.
• uiState is added to message list, pagination, export, edit and temporary-chat responses; no other response envelope changes.

Documentation

• Documentation is not needed for this change.
• Documentation was updated.
• Documentation still needs to be updated.

docs/ARCHITECTURE.md records the deliberate exception that ui-block lives in packages/core while only the web consumes it (pure logic, Node tests, contract shared with the backend prompt), and packages/core/README.md documents the module and its testdata. The model-facing format is documented in the catalog prompt itself, which the tests above check. Design notes live in docs/rules/GenUI/Plan.md, which .gitignore keeps out of the repository on purpose.

Security and privacy

• No secrets, tokens, credentials, local config, or personal data are included.

• User data access remains scoped by authenticated user context unless an admin-only path explicitly requires broader access.

• Security-sensitive behavior was reviewed, including authentication, authorization, provider routing, file processing, billing, and admin APIs where relevant.

• Model-authored HTML and script now runs inline in the sandbox that already backs artifacts and custom components: allow-scripts only, no same origin, no network, no forms, every permission denied, a 200 KB cap, and the source is always viewable.

• ui-state only accepts assistant messages of the current conversation owned by the caller, refuses messages that are still generating, and validates block ids, state names, depth and size (message.invalidUiState, message.ui_state_target_invalid, both localised).

• Injected state is user-entered data: XML-escaped, capped at 8 blocks and 4 KB, passed in the dynamic context outside the cached prefix, and explicitly labelled as values rather than instructions; prompt traces record its source.

• ask only prefills the composer (2000 characters, throttled): the host cannot attribute a click inside a sandbox to the user, so the user is the one who sends, and the text they send is the text they can see.

Checklist

• I searched existing issues and pull requests.
• Changes are focused and do not include unrelated refactors.
• Tests or static verification were run where practical.
• User-facing behavior, deployment steps, API contracts, or configuration changes are documented.
• Generated artifacts are included only when this project explicitly requires them.
• Caches, build output, .pyc files, .env files, and local storage data are not committed.