apache · adityamparikh · Sep 11, 2026 · Sep 11, 2026
diff --git a/README.md b/README.md
@@ -97,6 +97,7 @@ Using a different client, or want STDIO/HTTP/Docker options? See the per-client
 |------|-------------|
 | `search` | Full-text search with filtering, faceting, sorting, and pagination |
 | `index-json-documents` | Index documents from a JSON string into a collection |
+| `index-file` | Index a local UTF-8 JSON, CSV, XML or Markdown file (STDIO only; no file-size cap) |
 | `index-csv-documents` | Index documents from a CSV string into a collection |
 | `index-xml-documents` | Index documents from an XML string into a collection |
 | `index-markdown-documents` | Index a markdown document into a collection, extracting front matter, title, headings, and body text |
@@ -110,6 +111,42 @@ Using a different client, or want STDIO/HTTP/Docker options? See the per-client
 
 Every tool advertises MCP behavior hints (`readOnlyHint`, `destructiveHint`, `idempotentHint`) so clients can build sensible approval UX — `search` and the metadata tools are read-only, indexing is destructive but idempotent, schema modification is additive.
 
+**Before indexing:** use `get-schema` and `add-fields` (or the `design-schema`
+prompt) to prepare field types, `docValues`, and `multiValued` settings. Use
+`string` for facet categories and `text_general` for prose; schemaless guesses
+are not a substitute for schema design.
+
+**Index a saved file without repeating its payload:** in local STDIO mode, call
+`index-file` with `{"collection":"shows","path":"/data/shows.json"}`. No extra
+environment variable is required. JSON, CSV, XML and Markdown are detected from
+`.json`, `.csv`, `.xml`, `.md` or `.markdown` (case-insensitive). For an extensionless
+download or to override detection, add `"format":"json"` (or `csv`, `xml`,
+`markdown`/`md`). Reuse the same path for another prepared collection. The result
+reports actual counts and field names, never the file contents.
+
+There is **no application-imposed file-size cap**. JSON/CSV/XML records are parsed
+incrementally and sent in batches of 1,000; Markdown remains one document with its
+front matter, title, headings and body intact. A large individual record or
+Markdown file still needs enough memory, and backend limits/timeouts still apply.
+Indexing is not transactional: a parse error or interrupted call can leave earlier
+batches in Solr. Check counts before retrying and supply stable IDs to avoid duplicates.
+The inline tools remain available with their existing input validation.
+Field mapping matches the inline parsers. In particular, XML uses repeated sibling
+element names to recognize multiple documents; otherwise it flattens the root,
+including its name in field paths. Check the returned field names when preparing
+the schema and assigning stable IDs.
+
+Paths refer to the **MCP server filesystem**, not a remote client's machine.
+Absolute paths are recommended; relative paths use the server's working directory.
+URLs and `~` expansion are not supported. For Docker, mount a data directory
+read-only (for example `--mount type=bind,source=/absolute/data,target=/data,readonly`)
+and use `/data/shows.json`. **The connected local client can ingest any regular file
+readable by the server process, including sensitive files.** OS permissions and
+container isolation provide the boundary; mount only intended data and do not run
+the server with elevated privileges. File ingestion is absent in HTTP/web mode,
+even when both `stdio` and `http` profiles are active. Remote uploads are not part
+of this feature; HTTP clients continue using the inline tools.
+
 ### Resources
 
 | Resource URI | Description |
@@ -154,7 +191,7 @@ Running in **HTTP mode** — OAuth2, CORS, and the `HTTP_SECURITY_ENABLED` toggl
 
 **Using it**
 - [Quick start](#quick-start) · [Client setup](docs/clients/) — Claude Desktop, Claude Code, VS Code, Cursor, JetBrains, MCP Inspector
-- [Tutorial: your first collection](docs/tutorial.md) — index a dataset twice, schemaless then with a designed schema, and see why field types matter
+- [Tutorial: your first collection](docs/tutorial.md) — design a schema first, index a saved dataset, and explore it
 - [Observability](docs/observability.md) — OpenTelemetry traces, metrics, logs
 - Security: [Deployment model (single-tenant)](docs/security/deployment-model.md) · [STDIO model](docs/security/stdio.md) · [HTTP model](docs/security/http.md) · OAuth2 setup: [Auth0](docs/security/auth0.md) · [Keycloak](docs/security/keycloak.md)
 

diff --git a/THREAT_MODEL.md b/THREAT_MODEL.md
@@ -201,8 +201,17 @@ reaching the backend Solr directly, bypassing this server, is out of model (§3)
   Solr always; opens a servlet listener only in HTTP mode; exports OTLP
   telemetry when a collector is configured; and, in HTTP `bootRun`, may start
   `docker compose`-declared services in local dev. It does not spawn child
-  processes for tool execution or read arbitrary files from tool input.
-  *(maintainer — Q-sideeffects.)*
+  processes for tool execution. In local STDIO mode, `index-file` reads regular
+  UTF-8 JSON, CSV, XML and Markdown files without an ingest-root setting or file-size
+  cap. The connected client can ingest any file readable by the process, including
+  sensitive content; OS permissions and container isolation define the boundary.
+  This is not a filesystem sandbox against concurrent local writers. Operators
+  should use least privilege and read-only mounts containing only intended data.
+  Structured records stream in batches; a single record or Markdown document still
+  requires sufficient memory. Interrupted or malformed input may leave partial
+  writes. The tool is absent in HTTP/web mode, including mixed stdio/http profiles.
+  No URL fetching or remote uploads are added.
+  *(documented — `FileIndexingService`; README.)*
 
 ## §5a Configuration variants — the security-relevant knobs
 
@@ -246,6 +255,7 @@ trust table:
 | `search` | `query` (`q`), `filterQueries` (`fq`) | **yes** | passed into `SolrQuery`; Solr query-parser semantics apply — Q-queryinj |
 | `search` | `facetFields`, `sortClauses`, `start`, `rows` | **yes** | forwarded to Solr; `rows` unbounded? — Q-resource |
 | `index-*` | `collection`, `json`/`csv`/`xml` body | **yes** | parsed then written to index; XML parser is XXE-hardened *(documented)* |
+| `index-file` (local STDIO only) | `collection`, `path`, optional `format` | **yes** | any process-readable regular UTF-8 file; JSON/CSV/XML stream in batches, Markdown stays one document; no file-size cap; OS/container boundary, unavailable over HTTP |
 | `create-collection` | `name`, `configSet`, `numShards`, `replicationFactor` | **yes** | issues `CollectionAdminRequest.createCollection` to backend — Q-adminexposure |
 | `add-fields` / `add-field-types` | `collection`, field/type defs | **yes** | additive schema change (existing fields cannot be modified per README) |
 | config (startup only) | `SOLR_URL`, `SOLR_USERNAME`, `SOLR_PASSWORD` | **no — deployer config** | never wire from a tool argument *(documented)* |
@@ -572,8 +582,10 @@ into the body above; the corresponding claims now carry *(maintainer)* tags.
   reason. (§6/§9.)
 - **Q-sideeffects / Q-otel.** Is the outbound side-effect inventory complete? →
   **Confirmed.** SolrJ connection (always), servlet listener (HTTP mode only),
-  OTLP export (when configured). No child processes and no file reads from tool
-  input. Securing OTLP and TLS is operator infrastructure. The `docker compose`
+  OTLP export (when configured). No child processes. Local STDIO `index-file`
+  adds process-readable file ingestion as described in §5/§6; HTTP does not
+  expose file ingestion. Securing OTLP and TLS is operator
+  infrastructure. The `docker compose`
   autostart exists only in the http-profile `bootRun` local-dev path. (§5/§9.)
 
 **Wave 4 — meta / ownership — ANSWERED**

diff --git a/docs/tutorial.md b/docs/tutorial.md
@@ -5,16 +5,17 @@ queryable collection — entirely through natural-language conversation with an
 AI assistant.
 
 The point of this tutorial is not just *how* to index data. It's **why field
-types matter**. You will index a dataset twice: once letting Solr guess, once
-choosing types deliberately, and see exactly what the difference buys you.
+types matter**. Define the schema first, index a saved file, and ask useful
+questions immediately. Schemaless pitfalls are explained below, not required
+as a detour through a broken collection.
 
 **Time:** about 15 minutes.
 
 ```
   1. Start Solr           empty SolrCloud, one container
-  2. Index blind          61 documents, zero schema work        <- it just works
-  3. Inspect the guess    what Solr decided on your behalf      <- and here's the catch
-  4. Design a schema      types chosen for the questions you ask
+  2. Save the dataset     reuse the file without repeating its contents
+  3. Design a schema      types chosen for the questions you ask
+  4. Index and verify     use the tool's actual document count
   5. Search               filters, facets, ranges, sorting
   6. Introspect           stats, health, schema
 ```
@@ -78,94 +79,24 @@ curl -O https://raw.githubusercontent.com/apache/solr-mcp/main/src/test/resource
 }
 ```
 
-If your client cannot read local files, paste the JSON contents directly into the
-conversation instead of referencing the path.
+In local STDIO mode, use `index-file` with the absolute path to `shows.json`.
+No extra environment variable is needed. The tool supports JSON, CSV, XML and
+Markdown, detects the extension (or accepts an explicit `format`), and has no
+file-size cap. It returns counts, not content. JSON/CSV/XML stream in batches;
+Markdown stays one document and must fit in memory.
+
+The path is on the **server**, not necessarily on your client. For Docker, mount
+only the data directory read-only and pass its container path, such as
+`/data/shows.json`. The local client can ingest any file readable by the server
+process, so use OS permissions or container isolation to keep secrets inaccessible.
+Relative paths use the server's working directory, which may differ from your shell.
+HTTP has no file-ingestion tool; use `index-json-documents` with inline JSON there.
+The server does not fetch URLs; download once into a location the local server can
+read. Do not ask the model to reconstruct the entire dataset from memory.
 
 ---
 
-## Step 1 — Index without a schema
-
-Solr's `_default` configset runs in **schemaless** (data-driven) mode: send it
-documents containing fields it has never seen, and it will invent types for them.
-
-> *"Create a Solr collection called shows-auto."*
-
-> *"Index the contents of ./shows.json into the shows-auto collection."*
-
-You should get `61 of 61 documents`. No schema, no field definitions, no
-configuration — and it worked.
-
-This is genuinely useful. Schemaless mode exists so you can get data in and start
-exploring before you know what questions you'll ask. The trouble starts when you
-ask them.
-
----
-
-## Step 2 — Ask a real question
-
-> *"Show me the breakdown of shows-auto by platform."*
-
-This is the most ordinary business question imaginable, and the answer comes back
-empty:
-
-```json
-{ "numFound": 61, "documents": [], "facets": { "platform": {} } }
-```
-
-Read that carefully, because it is worse than an error. Sixty-one documents
-matched. The query succeeded. Solr simply has no breakdown to give you, and it
-says so without complaining. Nothing here tells you that the *data* is fine and
-the *field type* is the problem — which is exactly the failure mode that makes
-schemaless deceptive.
-
-To see the cause, look at what Solr decided on your behalf:
-
-> *"Show me the schema for shows-auto."*
-
-| Field | Solr guessed | What that costs you |
-|-------|-------------|---------------------|
-| `platform` | `text_general` | Tokenized and analyzed, so it is no longer one value. Searching `platform:prime` matches all 20 Amazon Prime Video shows — nonsense for a category — and faceting it yields **no buckets at all**. |
-| `title` | `text_general` | Searchable, but not sortable or exact-matchable. |
-| `imdb_rating` | `pdoubles` | Note the trailing `s` — that plural means **multi-valued**. Every rating is a list, so "highest rated" is not a well-defined question. |
-| `release_year` | `plongs` | Multi-valued too, which makes range filtering awkward. |
-
-Look at any document that comes back and the giveaway is visible — every field is
-wrapped in an array:
-
-```json
-"title": ["Stranger Things"], "imdb_rating": [8.7]
-```
-
-Solr saw one sample of each field and had no reason to assume it would not repeat,
-so it hedged on all of them.
-
-**The lesson:** schemaless is an on-ramp, not a destination. Solr guessed from a
-single document with no knowledge of what you would later want to ask. Faceting,
-range filtering and sorting all depend on types chosen with those questions in
-mind.
-
----
-
-## Step 3 — Reset
-
-Field types **cannot be changed once created**. Worse, at present every collection
-created through `create-collection` shares the same `_default` configset, so the
-guesses from Step 1 are already baked in and a new collection would inherit them
-(see [Known issues](#known-issues)).
-
-So start from a clean slate:
-
-```bash
-docker rm -f solr-tutorial
-docker run -d --name solr-tutorial -p 8983:8983 solr:9-slim solr start -c -f
-```
-
-This takes a few seconds. Wait for the collections endpoint to answer before
-continuing.
-
----
-
-## Step 4 — Design the schema first
+## Step 1 — Design the schema first
 
 Now create the collection and define its fields **before** any documents arrive.
 
@@ -188,18 +119,26 @@ each choice:
 | `string` rather than `text_general` | Exact values. "Amazon Prime Video" stays one facet bucket instead of disappearing into tokens. |
 | `docValues: true` | The column-oriented structure that makes faceting and sorting efficient. |
 | `pint` / `pdouble` | Real numbers, so range filters like `[2020 TO *]` and numeric sorting work. |
-| Single-valued where the data is single-valued | You can sort on it. Sorting by a multi-valued field is not meaningful. |
+| Single-valued where the data is single-valued | A rating is a scalar, not a list; sorting needs no implicit minimum/maximum selection. |
 | `text_general` kept for prose | Analysis and tokenizing is exactly right for `title` and `description`. |
 
 Note that the difference is not "strings are better than text". Both types appear
 in this schema. The difference is matching the type to how the field will be
 *queried* — categories get exact matching, prose gets analysis.
 
-Now index the same data into the new collection:
+## Step 2 — Index and verify
+
+> *"Use index-file with collection shows and the absolute path to shows.json. Report the
+> tool's actual successful and total counts."*
 
-> *"Index the contents of ./shows.json into the shows collection."*
+Expect `Successfully indexed 61 of 61`. Confirm with `search`, `query=*:*`,
+`rows=0`: `numFound` should be 61. You can reuse the same file path to index
+another prepared collection without resending the JSON. Reusing IDs in the same
+collection updates documents, so a second call still leaves 61 documents.
+If a call fails partway through, earlier batches may already be indexed; verify
+the collection before retrying with the same stable document IDs.
 
-And ask the question that failed in Step 2:
+Then ask:
 
 > *"Show me the breakdown of shows by platform."*
 
@@ -208,12 +147,12 @@ Netflix 20, Amazon Prime Video 20, HBO Max 7, Apple TV+ 4,
 Disney+ 4, Hulu 3, Paramount+ 2, Peacock 1
 ```
 
-Same data, same question, same tool. The only thing that changed is that someone
-decided what the fields meant.
+The category field preserves exact platform names, so the breakdown answers the
+question rather than counting analyzed word tokens.
 
 ---
 
-## Step 5 — Search
+## Step 3 — Search
 
 Each of these exercises a different Solr capability. The parameter each one drives
 is noted so you can connect the natural-language request to what actually runs.
@@ -240,7 +179,7 @@ what the server exists to provide.
 
 ---
 
-## Step 6 — Ask about the index itself
+## Step 4 — Ask about the index itself
 
 Search is the headline, but the operational tools are what make this useful in a
 real workflow.
@@ -259,18 +198,33 @@ otherwise mean reading `managed-schema` and knowing what `docValues` implies.
 
 ---
 
+## Why not index schemaless first?
+
+The `_default` configset can guess types for unknown fields, but it does not know
+your intended queries. It may infer `text_general` without docValues for
+`platform`, and multi-valued `pdoubles`/`plongs` for scalar numbers. Faceting an
+analyzed category can return token buckets, no buckets, or an error depending on
+the configuration and Solr version — not reliable exact-category counts.
+
+If you already indexed this way, inspect the relevant fields and copy-field
+rules. An existing string copy such as `platform_str` can provide exact facets;
+do not assume the sibling exists without checking. For a durable fix, use a
+clean configset, define the schema, and reindex from the saved file. These MCP
+tools only **add** fields and cannot change an existing field's type. There is
+no need to deliberately break and reset Solr to complete this tutorial.
+
 ## Known issues
 
-Two rough edges worth knowing about. The first is a tracked defect you will meet
-while following this tutorial; neither is a mistake on your part.
+Two rough edges worth knowing about, especially when reusing an existing Solr.
 
 **Collections share the `_default` configset**
 ([#183](https://github.com/apache/solr-mcp/issues/183)). `create-collection` binds
 each collection to the shared `_default` configset rather than copying it, so
 schemaless field guesses leak into every collection created afterwards. Symptoms
 are `add-fields` failing with `Field 'x' already exists` on a brand-new collection,
 or a "schemaless" collection silently inheriting another collection's explicit
-types. Restarting Solr resets it, which is why Step 3 exists.
+types. Use a fresh, isolated configset for a different schema; simply restarting
+a persistent Solr instance does not reset its managed schema.
 
 **Unknown search parameters are dropped silently.** Arguments the `search` tool does
 not declare are ignored rather than rejected, so a misnamed one looks like a query
@@ -291,7 +245,7 @@ names via your client's tool inspector before assuming the data is wrong.
 
 Things worth trying with what you have running:
 
-- Index your own JSON, CSV or XML and see what the schema guesser makes of it.
+- Design a schema for your own JSON, CSV or XML, then index and verify it.
 - Add a `DenseVectorField` with `add-field-types` and try vector search.
 - Describe a dataset in words and ask the assistant to design a schema for it.
 - Point the server at a Solr you already run — `SOLR_URL` is the only setting.