feat: persist request errors and make error IDs searchable in the log center

[chenyme]

Summary

Users are shown an error ID (the request ID) when a request fails, but there was no way to look it up: the underlying cause only reached stdout, and none of the admin log-center tabs indexed it. Several failure paths also recorded no cause at all, so even the stdout log could not explain the failure.

This change makes the failure cause a persisted, searchable artifact and ties every surface to the same ID:

• The error cause is attached to the request context and written to a new system_error_logs table for 5xx responses, recovered panics, and streams that end in a terminal error event after a 2xx header. Plain 4xx stays in stdout only.
• A new Errors tab in the admin log center matches an error ID exactly and shows the raw cause together with the raw upstream response body.
• The conversation-events and model-call tabs also accept an error ID, joined through chat_runs.request_id, so one ID leads to the full request.
• The chat UI shows the error ID on the generation-failure card and appends it to 5xx toasts.

Approach:

• response.InternalError(c, err) now takes the underlying error; every call site was migrated and a guard test rejects InternalError(c, nil). The code and cause are stored on the request context instead of gin.Context.Error, because otelgin would mark 4xx spans as errors.
• middleware.AccessLog gained Route, ErrorCode, and Error (truncated to 4 KB), redacts credentials, and persists server-side failures through an ErrorRecorder.
• Writes are asynchronous and bounded (32 in flight, drop-on-overflow with a throttled warning, 5 s timeout), so a database outage cannot amplify itself through error logging.
• errorDiagnostic lets an error type expose raw diagnostics; llm.UpstreamError returns the upstream response body, which is stored in detail with credentials masked and capped at 16 KB.
• User-initiated stops are explicitly excluded from being recorded as failures.

Change type

• Bug fix
• Feature
• Documentation
• Refactor
• Configuration / deployment
• Security hardening
• Other

Affected areas

• Frontend / UI
• Backend / API
• Authentication / authorization
• Conversations / streaming
• Files / RAG / extraction
• Model routing / providers
• MCP / tools
• Billing / payments
• Admin console
• Deployment / Docker / configuration
• Documentation

Verification

Backend:

• make -C backend lint — passed
• go test ./... — passed
• Trimmed build go build -tags nopostgres,noredis,nos3,noswagger,nomsgpack ./... — passed
• pnpm api:generate and pnpm api:check — regenerated and clean

Added or extended tests:

• error-log repository: exact request-ID match, filters, detail search
• request-ID search on conversation events and usage ledgers
• async write behavior: persists after the request ends, drops beyond the in-flight limit
• AccessLog: persists only 5xx and failed streams, redacts the path
• diagnostic detail extraction with credential masking
• upstream request failed prefix de-duplication
• user-initiated stop is not recorded as a failure
• request-ID trust: only honored from TRUSTED_PROXIES, charset and length enforced

Not run:

• Not run; reason: pnpm --filter @deeix/web check (lint, typecheck, check:arch) was not run locally; apps/web has no unit tests by design. CI covers it.
• Not run; reason: PostgreSQL integration tests. Only SQLite-backed repository tests ran locally.

Screenshots, API examples, or logs

UI screenshots for the new Errors tab and its detail sheet are attached.

GET /api/v1/admin/error-logs?query=e00dcc2b-04e9-4d55-91ba-e8da24cec0ba

{
  "total": 1,
  "results": [
    {
      "requestID": "e00dcc2b-04e9-4d55-91ba-e8da24cec0ba",
      "traceID": "ca05d3c6117a1fbf33d4ae78eef7bc69",
      "method": "POST",
      "route": "/api/v1/conversations/:id/messages/stream",
      "statusCode": 200,
      "errorCode": "upstream.unavailable",
      "cause": "upstream request failed: status=404 message=Not Found",
      "detail": "{ \"error\": { \"message\": \"Not Found\" } }",
      "latencyMS": 2583
    }
  ]
}

Configuration, migration, and compatibility notes

• Schema: system_error_logs is created by AutoMigrate at startup. A backend restart is required, and only failures that happen after the deployment are recorded.
• API contract: MessageResponse gained errorRequestID. Swagger output (backend/docs) and packages/api-contract/src/types.generated.ts were regenerated.
• New endpoint: GET /api/v1/admin/error-logs.
• Log cleanup: accepts a new type errors; retention is then under admin control.
• No config or environment changes. Proxy handling was delivered separately in fix: honor HTTP_PROXY/HTTPS_PROXY when SSRF protection is enabled #849.
• Compatibility: public share pages force errorRequestID to "", so shared conversations never expose the owner's request IDs.

Documentation

• Documentation is not needed for this change.
• Documentation was updated.
• Documentation still needs to be updated.

Note: docs/rules/* is gitignored (.gitignore:169), so the rule updates for this change (backend code style §16.1, layering notes) are local-only and are not part of this diff. No other tracked documentation needed changes.

Security and privacy

• No secrets, tokens, credentials, local config, or personal data are included.
• User data access remains scoped by authenticated user context unless an admin-only path explicitly requires broader access.
• Security-sensitive behavior was reviewed, including authentication, authorization, provider routing, file processing, billing, and admin APIs where relevant.

Reviewer notes:

• X-Request-ID is now honored only when the request comes from a trusted proxy.
• Stored causes and upstream bodies are masked for credentials (bearer tokens, key=value pairs, URL credentials and sensitive query parameters). Masking is scoped to credentials only; everything else is kept verbatim so failures stay diagnosable.
• Error-log rows carry user IP, user agent and upstream response bodies, and are visible to every admin, matching the audit-log tab. If upstream bodies may contain prompt content, consider restricting this tab to super admins.

Checklist

• I searched existing issues and pull requests.
• Changes are focused and do not include unrelated refactors.
• Tests or static verification were run where practical.
• User-facing behavior, deployment steps, API contracts, or configuration changes are documented.
• Generated artifacts are included only when this project explicitly requires them.
• Caches, build output, .pyc files, .env files, and local storage data are not committed.