feat(indexing): stream local JSON, CSV, XML and Markdown files

[adityamparikh]

Summary

Adds one MCP tool, index-file, for local STDIO deployments. It takes a collection, a file path on the server host and an optional format, and streams that file into Solr so the document bytes never pass through the model's context window.

• Supports JSON, CSV, XML and Markdown. The format is taken from the file extension (case-insensitive) or set explicitly with format.
• JSON, CSV and XML are parsed one record at a time and sent in 1,000-document batches, reusing the existing per-batch fallback and a single final commit. Markdown is indexed as one document.
• Reuses the existing document creators, field-name sanitisation and schema-first guidance. The response reports indexed/total counts, a bounded list of indexed field names, and a warning when any batch failed.
• Paths are absolute or relative to the server's working directory. They are resolved to a real path and must point at a regular UTF-8 file. The server reads with its own OS permissions, so operators scope what is reachable with file permissions or read-only container mounts.
• The tool is registered only when the stdio profile is active and no web application is present. HTTP deployments keep the inline index-*-documents tools unchanged.
• README, tutorial and THREAT_MODEL document the tool and its trust boundary.
• The search optional-argument and error-message fixes that started on this branch now live in fix(search): tolerate blank optional arguments and return safe errors #195.

Why STDIO only

In STDIO mode the server is a child process on the user's own machine, so a path the client passes refers to a file on the same host and is read with the user's own permissions. MCP tool calls carry JSON arguments only; there is no file-transfer primitive, so when a client "attaches" a file the client reads it into the model's context rather than sending it to the server. Over HTTP the server runs on another host, where a client-side path means nothing and a server-side path would let any authenticated client read the server host's filesystem. Remote upload (multipart endpoint or fetch-from-URL) is deferred to a follow-up.

Limitations

• Interrupted or malformed input can leave a partially indexed collection. The tool is annotated idempotent and every error message says to verify counts before retrying.
• A single record, and a whole Markdown document, must fit in memory. Solr's request limits and timeouts still apply.
• XML keeps the existing prefixed field mapping, so plan Solr unique keys accordingly.

Verification

• Java 25: ./gradlew build --console=plain passed in a clean worktree at 993b7cc (492 tests, zero failures, 7 skipped).
• Full working-directory build also passed; Spotless formatting applied.
• Tests cover all four formats above 10 MiB, streaming and batch bounds, format detection and override, malformed input and UTF-8 handling, XML external-entity defences, partial batch failures, reusing a path across collections, end-to-end MCP indexing and reindexing over STDIO, and the HTTP exclusion boundary.
• Native images and additional Solr versions were not tested.

Try locally

Run the server in STDIO mode and call index-file with collection=shows and an absolute path to a UTF-8 dataset, for example /data/shows.json. For Docker, mount only the intended data directory read-only and pass its container path.

🤖 Generated with Claude Code

https://claude.ai/code/session_019zJ9WA8nNyA5Yxb8ueM7vV

[adityamparikh]

Closing this one. The index-file tool only exists in STDIO mode, and there is no equivalent way to get a file into the server over HTTP, so merging it would give the two transports different tool surfaces for the same job. I would rather not ship a capability that half the deployments cannot use.

What HTTP parity would need is a way for a client to hand the server a file: either the MCP file-transfer primitive that is still being specified (SEP-2631), or an out-of-band upload endpoint with its own auth and size limits. Both are bigger than this PR and neither belongs in an indexing change, so I am parking the whole feature rather than carrying a STDIO-only version.

The branch stays as is. Two transport-neutral pieces could be lifted out later if useful: the indexBatch extraction in IndexingService, and the schema-first guidance in the tool descriptions and index-documents prompt. The search-argument fixes that started here already live in #195.