feat: let admins review and retry vector index tasks across the platform

[chenyme]

Summary

Closes #848. Admins could see vector index counts but could not act on pending or failed files without re-running the whole index. Investigating the report turned up several defects behind the numbers:

• pending counted every none file, including audio, video and images the current config cannot vectorize, so it never reached zero.
• Files left processing by a restart were only recovered when their owner opened the file list. Files left queued by a lost queue message were never recovered.
• Retrying a "no text" file never re-ran extraction, and the extracted text was never saved. The file stayed "no text" for users and was marked empty again on the next rebuild.

Admin task management

• GET /admin/settings/embedding/tasks lists files per bucket (ready, pending, failed, stale, empty, unsupported) across the platform, paginated and searchable. Each item shows owner, MIME type, size, status, failure reason, whether it is stalled, and whether it can be retried, with the reason when it cannot.
• POST /admin/settings/embedding/tasks/retry retries files across users through the existing recoverable queue. Jobs stay owned by the file owner, and the action is audited.
• Status panel:
  • All six buckets are clickable, including those with a zero count.
  • The pending bucket shows how many tasks are stalled.
  • While a rebuild or queued work is running, the panel refreshes every 3 seconds. Refresh pauses while the tab is hidden and stops after a few failed attempts.
• "Rebuild index" is renamed "Index remaining" and states that ready files are never rebuilt. Clicking it while a rebuild is running returns 409 embedding.reindex_in_progress instead of reporting "0 submitted".
• The task dialog has bucket tabs, search, infinite scroll, and single or bulk retry (sent in batches of 100). It uses the shared DialogHeightTransition and useLoadMoreSentinel.

Accurate counts and stall recovery

• Index status now comes from one grouped query instead of seven COUNT queries.
  • pendingCount no longer includes unsupported files (new unsupportedCount) or files whose processing failed (now counted as failed).
  • New fields: stalledCount, activeCount, reindexRunning.
• A background sweep marks files stuck in processing for more than 6 minutes as failed, so they can be retried. This replaces the write that ran per user on the file list read path.
• Every embedding run is now capped at 5 minutes (embedding.JobTimeout). Targeted jobs previously had no cap.
• Tasks queued for more than 30 minutes count as stalled and can be resubmitted.
• When a queue message is delivered for the first time and another worker is already processing the file, it no longer runs a second time. Only a message reclaimed after an expired lease takes over the interrupted run.

Files without text are reprocessed through the pipeline

• Admin retries of files with no usable text (processing failed, no text extracted, or stalled before processing) reset the file and run the full processing pipeline.
  • The extracted text is saved, so users see it, full-context injection can read it, and later rebuilds reuse it instead of paying for OCR again.
  • Extraction covers the whole document, the same as on first upload. The old retry path stopped at the full-context page limit.
• The embedding request is recorded on the file, so the index is built even when auto-embedding on upload is off.
  • Images without OCR are embedded from pixels when the protocol supports image input.
  • If reprocessing fails, the request is released instead of staying queued.
• "Index remaining" sends only files that already have text to the embedding service. Failed and stalled files are reprocessed instead, and so are no-text files when include_empty=true.
• The user-facing "Vectorize" action is unchanged and never triggers re-extraction.

Also included

The icon-only actions in the file header (vectorize / update index, open, download, delete) now use the shared Tooltip, shown above the button like the chat message actions, instead of the native title attribute.

Change type

• Bug fix
• Feature
• Documentation
• Refactor
• Configuration / deployment
• Security hardening
• Other

Affected areas

• Frontend / UI
• Backend / API
• Authentication / authorization
• Conversations / streaming
• Files / RAG / extraction
• Model routing / providers
• MCP / tools
• Billing / payments
• Admin console
• Deployment / Docker / configuration
• Documentation

Verification

• make -C backend lint, go test ./..., go test -race (embedding, processing, settings)
• go build with nopostgres, with noredis,nos3,noswagger, and with all four tags combined
• New repository queries covered by SQLite tests and run against PostgreSQL 16 (pgvector)
• End-to-end pipeline tests using the real repository, extraction and embedding services (only external engines faked):
  • retrying a no-text file saves the text and builds the index with auto-embedding off;
  • a later rebuild does not extract again (one extraction in total);
  • a failed reprocess does not leave the request queued;
  • an image without OCR is embedded from pixels.
• pnpm --filter @deeix/web check, pnpm api:check
• Manual UI check: status panel, task dialog (scroll, switch buckets, single and bulk retry), a retried no-text file showing its text in the user file view, file header tooltips

Screenshots, API examples, or logs

Configuration, migration, and compatibility notes

• No schema migration or new configuration.
• EmbeddingIndexStatusResponse: pendingCount no longer includes unsupported files or processing failures. New fields: stalledCount, activeCount, unsupportedCount, reindexRunning.
• POST /admin/settings/embedding/reindex returns 409 while a rebuild is running. With include_empty=true, no-text files are now re-extracted through the pipeline.
• New error code embedding.reindex_in_progress, added to the zh-CN and en-US errors.json. Swagger and @deeix/api-contract are regenerated.
• When an admin retries a file without text, the configured OCR engine may be called once for that file. Paid engines bill for that call; the result is saved and reused.
• Known limitations:
  • In multi-instance deployments, protection against overlapping rebuilds applies only within each instance (existing behavior).
  • A very large backlog may need "Index remaining" clicked more than once because of the 15-second request limit. Files already submitted are skipped on later clicks.

Documentation

• Documentation is not needed for this change.
• Documentation was updated.
• Documentation still needs to be updated.

Security and privacy

• No secrets, tokens, credentials, local config, or personal data are included.
• User data access remains scoped by authenticated user context unless an admin-only path explicitly requires broader access.
• Security-sensitive behavior was reviewed, including authentication, authorization, provider routing, file processing, billing, and admin APIs where relevant.

Checklist

• I searched existing issues and pull requests.
• Changes are focused and do not include unrelated refactors.
• Tests or static verification were run where practical.
• User-facing behavior, deployment steps, API contracts, or configuration changes are documented.
• Generated artifacts are included only when this project explicitly requires them.
• Caches, build output, .pyc files, .env files, and local storage data are not committed.