diff --git a/API.md b/API.md new file mode 100644 index 0000000..53f5fc9 --- /dev/null +++ b/API.md @@ -0,0 +1,210 @@ +# OpenMind Core API + +The OpenMind Core API lets local client applications manage sources, control indexing, search local memory, and ask source-grounded questions without knowing about SQLite, LanceDB, extractors, embeddings, or model-provider internals. + +The first API contract is available at: + +```text +http://127.0.0.1:8765/api/v1 +``` + +Interactive OpenAPI documentation is available at `http://127.0.0.1:8765/docs` while the server is running. + +## Start the server + +Complete `openmind setup` first, then run: + +```bash +openmind serve +``` + +The server always binds to `127.0.0.1`. A different port can be selected without exposing the API to the network: + +```bash +openmind serve --port 9000 +``` + +## Authentication + +OpenMind generates a cryptographically random API token at: + +```text +~/.openmind/api_token +``` + +The token file is restricted to the current operating-system user on platforms that support POSIX permissions. Show the token with: + +```bash +openmind api token +``` + +Send it with every `/api/v1` request: + +```http +Authorization: Bearer +``` + +`GET /health` is the only unauthenticated endpoint. It returns only liveness and the OpenMind version. + +Rotate a token if it may have been exposed: + +```bash +openmind api token --rotate +``` + +Rotation immediately invalidates the previous token. The running API server picks up the new token automatically. + +## First request + +```bash +TOKEN="$(openmind api token)" + +curl http://127.0.0.1:8765/api/v1/status \ + -H "Authorization: Bearer $TOKEN" +``` + +Search local memory: + +```bash +curl http://127.0.0.1:8765/api/v1/search \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"query":"cabin packing list","limit":5}' +``` + +## Endpoints + +| Method | Path | Purpose | +| --- | --- | --- | +| `GET` | `/health` | Public liveness and version check | +| `GET` | `/api/v1/status` | OpenMind, model, memory, and indexing status | +| `GET` | `/api/v1/providers` | Available model providers | +| `GET` | `/api/v1/providers/status` | Current provider connectivity | +| `GET` | `/api/v1/models` | Available chat, embedding, and image models | +| `POST` | `/api/v1/models/load` | Load configured or explicitly selected models | +| `PUT` | `/api/v1/models/selection` | Save validated model choices and optionally load them | +| `GET` | `/api/v1/sources` | List user-approved source folders | +| `POST` | `/api/v1/sources` | Add a source folder | +| `DELETE` | `/api/v1/sources/{source_id}` | Remove source permission from OpenMind | +| `POST` | `/api/v1/index/start` | Start or return the active background indexing job | +| `GET` | `/api/v1/index/status` | Read indexing progress | +| `POST` | `/api/v1/index/pause` | Request indexing pause | +| `POST` | `/api/v1/index/resume` | Resume paused indexing | +| `POST` | `/api/v1/index/stop` | Request indexing stop | +| `POST` | `/api/v1/search` | Search indexed local memory | +| `POST` | `/api/v1/ask` | Return an answer with structured sources | +| `POST` | `/api/v1/ask/stream` | Stream answer text as server-sent events | +| `GET` | `/api/v1/documents/{file_id}` | Inspect an indexed file and its text chunks | +| `POST` | `/api/v1/actions/open` | Open a validated indexed file in its default OS app | + +## Sources + +Add a folder: + +```json +{ + "path": "/Users/example/Documents", + "recursive": true +} +``` + +OpenMind resolves the path and rejects missing files, non-directory paths, and duplicate sources. Removing a source removes OpenMind's permission record; it does not delete the folder or its files. + +## Model selection + +`GET /api/v1/models` separates chat, embedding, and image-capable models. Save selections with: + +```json +{ + "chat_model": "qwen-model-key", + "embedding_model": "nomic-embedding-key", + "image_model": "smolvlm-model-key", + "load": true +} +``` + +The API validates every key against models reported by the configured provider. Set `chat_model` to `null` for search-only mode or `image_model` to `null` to disable image indexing. An embedding model is required. + +## Search and Ask + +Search request: + +```json +{ + "query": "OAuth error screenshot", + "limit": 10 +} +``` + +Search results contain a `file_id`, `source_id`, path, score, snippet, source type, chunk index, and safe metadata. They never contain raw vectors or raw image bytes. + +Ask request: + +```json +{ + "question": "Do I have screenshots related to login errors?", + "limit": 8, + "include_sources": true +} +``` + +The synchronous response contains `answer` and structured `sources`. Use `/api/v1/ask/stream` with the same request body for server-sent events: + +```text +event: delta +data: {"text":"partial answer"} + +event: sources +data: {"sources":[{"file_id":"file_...","path":"/Users/example/Documents/notes.md"}]} + +event: done +data: {} +``` + +## Open indexed files safely + +The open action accepts an indexed `file_id`, not an arbitrary path: + +```json +{ + "file_id": "file_0123456789abcdef" +} +``` + +Before opening anything, OpenMind verifies that the database record is indexed, the file still exists, and its resolved path remains inside an enabled source folder. The API does not expose delete, move, edit, shell-command, or arbitrary-path actions. + +## Browser clients and CORS + +Cross-origin browser access is disabled by default. Allow only the exact development or application origin that needs access: + +```bash +openmind serve --allow-origin http://localhost:3000 +``` + +Repeat `--allow-origin` for multiple origins. Wildcards, credentials embedded in origins, paths, query strings, and fragments are rejected. Native desktop, mobile, editor, and command-line clients do not need CORS configuration. + +## Error responses + +OpenMind uses standard HTTP status codes: + +- `400` for an invalid product-level operation. +- `401` for a missing or invalid bearer token. +- `403` for a recognized but forbidden local action. +- `404` when a source, job, document, or indexed file is unavailable. +- `409` when a source folder is already registered. +- `422` when a request does not match the documented schema. +- `503` when the configured model provider cannot complete a request. + +Error bodies use FastAPI's standard `detail` field. Client applications should not depend on internal exception text. + +## Security boundaries + +- The server binds only to `127.0.0.1`; there is no public-host option. +- Private endpoints require bearer authentication, including on localhost. +- API tokens are generated locally, stored outside the project, omitted from logs, and compared in constant time. +- CORS is disabled unless exact origins are explicitly supplied. +- Request schemas reject unknown fields and bound text and result sizes. +- Only user-approved source folders can be indexed or opened. +- The API does not expose SQLite, LanceDB, embeddings, vectors, extractor calls, worker internals, raw file downloads, or raw image bytes. + +These boundaries are intentional. Client applications consume OpenMind capabilities while storage and provider implementations remain replaceable. diff --git a/CHANGELOG.md b/CHANGELOG.md index 315d9e7..e11a65b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,15 @@ User-facing changes for each OpenMind Core release. ## Unreleased -No unreleased changes. +- No unreleased changes. + +## 0.0.5 - 2026-07-19 + +- Added `openmind --version` and clearer command descriptions. +- Added arrow-key setup menus, checkbox folder selection, and an OpenMind terminal banner. +- Fixed a bug with custom folder selection so standard folders are not selected automatically. +- Added a secure local API so anyone can easily build their own client app on top of OpenMind. + - Apps can manage models, sources, and indexing, search local memory, stream answers, inspect results, and open indexed files. ## 0.0.4 - 2026-07-15 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 832ee0a..455a045 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -118,6 +118,7 @@ Tests should be focused and practical. Add tests when you change: - CLI behavior. +- Local API routes, authentication, validation, or response schemas. - SQLite schema or persistence. - Source scanning. - Extraction or chunking. @@ -127,6 +128,8 @@ Add tests when you change: Mock LM Studio for provider tests. Do not require contributors to run a local model just to pass the default test suite. +API changes must preserve the security boundaries documented in [API.md](API.md). Keep the server loopback-only, require authentication for product routes, validate request bodies, and never expose vectors, raw databases, arbitrary filesystem reads, or shell execution. + ## Privacy and Safety OpenMind must not scan the whole computer by default. diff --git a/FEATURES.md b/FEATURES.md index e4aff2f..2f3b720 100644 --- a/FEATURES.md +++ b/FEATURES.md @@ -25,6 +25,7 @@ Current boundaries: ### Core CLI +- `openmind --version` and `openmind -V` - `openmind init` - `openmind setup` - `openmind status` @@ -34,6 +35,11 @@ Current boundaries: - `openmind source add ` - `openmind source list` - `openmind source remove ` +- Descriptive top-level help text for every command. +- Large OpenMind ASCII banner during first-run setup. +- Arrow-key selection menus for providers and models. +- Checkbox folder selection with arrow keys and the Space key. +- Shared interactive prompt styling across setup and model updates. ### Local Storage @@ -114,7 +120,7 @@ Image files are indexed by generating text descriptions through a local vision m ### LM Studio Provider -- LM Studio is the only user-facing `0.0.4` provider. +- LM Studio is the only user-facing `0.0.5` provider. - Native LM Studio REST model listing: - `GET /api/v1/models` - Native LM Studio model loading: @@ -218,6 +224,28 @@ Image files are indexed by generating text descriptions through a local vision m - LM Studio log mode runs: - `lms log stream` +### Local API + +- `openmind serve` starts the API on `127.0.0.1:8765`. +- Versioned client contract under `/api/v1`. +- Public `GET /health` liveness endpoint. +- Bearer authentication for every private endpoint. +- Random local API token stored with private file permissions. +- `openmind api token` shows the client token. +- `openmind api token --rotate` invalidates the existing token. +- Status and model-provider inspection. +- Model listing, validated selection, and loading. +- Source listing, creation, and removal. +- Start, status, pause, resume, and stop indexing operations. +- Search responses with paths, snippets, scores, metadata, and stable file IDs. +- Source-grounded synchronous Ask responses. +- Server-sent-event streaming for Ask. +- Indexed document and chunk inspection without vectors. +- Safe open-file action restricted to indexed files inside enabled sources. +- Interactive OpenAPI documentation. +- Explicit browser origins through repeatable `--allow-origin`; wildcard CORS is refused. +- No raw SQLite, LanceDB, embedding, vector, extractor, or arbitrary-path endpoints. + ### Test Data - `data/` folder with local indexing fixtures. @@ -239,6 +267,7 @@ Image files are indexed by generating text descriptions through a local vision m - No source enable/disable command yet. - No command to clear or rebuild LanceDB tables yet. - No explicit failed-file retry command yet. +- API access is intentionally local-only; remote binding is not supported. ## Roadmap @@ -254,7 +283,7 @@ Image files are indexed by generating text descriptions through a local vision m - Add faster cancellation checks around embedding batches. - Add clearer model-loaded status. -### 0.0.5 Retrieval Quality +### 0.0.6 Retrieval Quality - Hybrid search: vector plus keyword/BM25. - Better snippets around matched content. @@ -264,7 +293,7 @@ Image files are indexed by generating text descriptions through a local vision m - Better PDF page metadata. - Better CSV/table summaries. -### 0.0.6 Local Memory Quality +### 0.0.7 Local Memory Quality - Persistent conversation sessions. - Session list/resume/delete commands. @@ -273,7 +302,7 @@ Image files are indexed by generating text descriptions through a local vision m - Answer confidence and missing-evidence notices. - Per-source indexing policies. -### 0.0.7 File Coverage +### 0.0.8 File Coverage - OCR for screenshots and image files. - Advanced OCR backend option such as PaddleOCR. @@ -282,12 +311,11 @@ Image files are indexed by generating text descriptions through a local vision m - Email export ingestion. - More document formats. -### 0.0.8 Local Service +### 0.0.9 Local Service Extensions -- FastAPI local API. -- Local web UI or desktop UI can connect to the same engine. - Background worker process management. - File watcher for incremental indexing. +- Optional event stream for indexing progress. ### Future Providers diff --git a/README.md b/README.md index 5e67708..2c47a45 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,7 @@ I am building this because I have a lot of files on my computer, and sometimes I ## What OpenMind Does -OpenMind Core is a local-first CLI for indexing, searching, and asking questions over user-approved folders. +OpenMind Core is a local-first engine, CLI, and authenticated local API for indexing, searching, and asking questions over user-approved folders. - Local app storage under `~/.openmind`. - User-approved folder sources. @@ -41,6 +41,7 @@ OpenMind Core is a local-first CLI for indexing, searching, and asking questions - Interactive ask sessions with temporary conversation memory. - Source-grounded answers. - Developer log inspection. +- Versioned local API for third-party client applications. OpenMind intentionally avoids: @@ -60,7 +61,7 @@ See [FEATURES.md](FEATURES.md) for the complete shipped feature list and roadmap - LM Studio for local chat, embedding, and vision models - macOS, Linux, or another Python-supported environment -OpenMind Core `0.0.4` uses LM Studio as its only user-facing provider. The older Sentence Transformers provider remains only as a development and test fallback. +OpenMind Core `0.0.5` uses LM Studio as its only user-facing provider. The older Sentence Transformers provider remains only as a development and test fallback. ## Install @@ -132,12 +133,14 @@ Setup: 1. Initialize `~/.openmind`. 2. Check that LM Studio is reachable. -3. Let you choose LM Studio as the provider. -4. List available chat, embedding, and image description models. +3. Let you choose a model provider (currently LM Studio). +4. Show arrow-key selectors for available chat, embedding, and image description models. 5. Load the selected models. -6. Ask which folders to index. +6. Show a checkbox selector for folders to index. 7. Start background indexing. +Use the arrow keys to move, `Space` to toggle folders in a checkbox list, and `Enter` to confirm a selection. Setup begins with the OpenMind terminal banner so it is immediately clear which application is running. + Watch indexing progress: ```bash @@ -175,6 +178,7 @@ openmind setup Lower-level initialization: ```bash +openmind --version openmind init openmind status openmind flush @@ -259,6 +263,30 @@ openmind dev logs --log index openmind dev logs --lm-studio ``` +Local API: + +```bash +openmind serve +openmind serve --port 9000 +openmind serve --allow-origin http://localhost:3000 +openmind api token +openmind api token --rotate +``` + +## Local API + +OpenMind exposes the same engine used by the CLI through an authenticated, versioned API for desktop apps, editor extensions, menu-bar tools, and other local clients: + +```text +http://127.0.0.1:8765/api/v1 +``` + +Start it with `openmind serve`. OpenMind creates a private bearer token under `~/.openmind/api_token`; retrieve it with `openmind api token` and send it as `Authorization: Bearer `. The server binds only to `127.0.0.1`, disables browser CORS by default, and never exposes raw database operations, vectors, embeddings, or arbitrary filesystem access. + +The API supports status, providers and models, source management, background indexing controls, search, synchronous and streaming Ask, indexed document details, and safe opening of indexed files. Interactive OpenAPI documentation is available at `http://127.0.0.1:8765/docs` while the server is running. + +See [API.md](API.md) for the complete client contract, security model, endpoint reference, and request examples. + ## LM Studio Integration OpenMind talks to LM Studio at: @@ -365,6 +393,28 @@ flowchart TD Logs --> LogFiles["~/.openmind/logs"] ``` +### Local Client Apps + +Client apps connect to OpenMind through the local API. They use OpenMind's capabilities without needing to know how extraction, storage, embeddings, or model providers work internally. + +```mermaid +flowchart LR + Clients["Local Client Apps
desktop, web, mobile, extensions"] + API["Authenticated Local API
127.0.0.1:8765/api/v1"] + Capabilities["OpenMind Capabilities
models, sources, indexing, search, Ask, documents"] + Engine["OpenMind Engine"] + State["SQLite
state + metadata"] + Memory["LanceDB
searchable memory"] + ModelServer["Local Model Server
(LM Studio, other providers later)"] + + Clients -->|"Bearer token"| API + API --> Capabilities + Capabilities --> Engine + Engine --> State + Engine --> Memory + Engine --> ModelServer +``` + ### SQLite SQLite is used for **project state and metadata**, not the AI memory itself. @@ -412,7 +462,7 @@ Simple way to think about it: OpenMind uses a model provider abstraction for embeddings and answers. -In `0.0.4`, the only implemented user-facing provider is LM Studio. OpenMind talks to LM Studio's local server endpoint; it does not use the LM Studio chat interface. +In `0.0.5`, the only implemented user-facing provider is LM Studio. OpenMind talks to LM Studio's local server endpoint; it does not use the LM Studio chat interface. OpenMind uses the provider endpoint for: @@ -441,6 +491,18 @@ Why they are used: - Rich makes tables, progress views, and errors easier to read - the CLI stays usable before any desktop or web UI exists +### FastAPI + +FastAPI exposes OpenMind's product-level engine capabilities to local client applications. + +Why it is used: + +- typed request and response contracts +- automatic OpenAPI documentation for client developers +- standard bearer authentication +- streaming responses for Ask +- the API remains separate from SQLite, LanceDB, and provider internals + ### uv uv is used for dependency management and development setup. @@ -850,7 +912,6 @@ Near-term work: - Better snippets and citations. - Better OCR and metadata extraction for screenshots, images, and scanned PDFs. - Persistent chat sessions. -- Local API for UI clients. - Additional providers after LM Studio is solid. The full roadmap lives in [FEATURES.md](FEATURES.md). diff --git a/TECHNICAL_SPEC.md b/TECHNICAL_SPEC.md index 31e76ee..b164382 100644 --- a/TECHNICAL_SPEC.md +++ b/TECHNICAL_SPEC.md @@ -1,8 +1,8 @@ -# OpenMind Core 0.0.4 Technical Spec +# OpenMind Core 0.0.5 Technical Spec ## Goal -Build a Python CLI tool named `openmind` that creates a local AI memory over user-approved folders. +Build a Python engine, CLI, and authenticated local API named `openmind` that creates local AI memory over user-approved folders and exposes stable product-level capabilities to client applications. OpenMind Core must: @@ -17,6 +17,8 @@ OpenMind Core must: - Store all app data under `~/.openmind` unless `OPENMIND_HOME` is set. - Use LM Studio as the first user-facing model server for chat, embeddings, and image descriptions. - Provide first-run setup and background indexing progress. +- Expose a versioned API on loopback for local client applications. +- Require bearer authentication for private API operations. ## Folder Structure @@ -25,6 +27,19 @@ openmind-core/ ├── openmind/ │ ├── cli/ │ │ └── main.py +│ ├── api/ +│ │ ├── app.py +│ │ ├── auth.py +│ │ ├── deps.py +│ │ ├── files.py +│ │ ├── schemas.py +│ │ └── routes/ +│ │ ├── system.py +│ │ ├── models.py +│ │ ├── sources.py +│ │ ├── indexing.py +│ │ ├── memory.py +│ │ └── actions.py │ ├── core/ │ │ ├── config.py │ │ ├── engine.py @@ -65,6 +80,7 @@ openmind-core/ │ └── answer.py ├── tests/ ├── pyproject.toml +├── API.md ├── README.md └── TECHNICAL_SPEC.md ``` @@ -75,6 +91,9 @@ Runtime: - `typer` - `rich` +- `questionary` +- `fastapi` +- `uvicorn` - `lancedb` - `sentence-transformers` - `pydantic` @@ -88,6 +107,7 @@ Runtime: Development: +- `httpx2` - `pytest` Dependency management: @@ -257,16 +277,17 @@ openmind uninstall --yes --package 1. Initialize `~/.openmind` if needed. 2. Check LM Studio at `http://localhost:1234`. -3. Present provider selection with LM Studio as the only current option. -4. Fetch `GET /api/v1/models`. -5. Split models by `type`: `llm` for chat and `embedding` for embeddings. -6. Ask the user to choose one chat model when available. -7. Require one embedding model. -8. Load selected models with `POST /api/v1/models/load`. -9. Save config to `~/.openmind/config.toml`. -10. Ask which folders to index. -11. Start background indexing. -12. Tell the user to run `openmind index status`. +3. Display the OpenMind ASCII banner. +4. Present an arrow-key provider selector with LM Studio as the only current option. +5. Fetch `GET /api/v1/models`. +6. Split models by `type`: `llm` for chat and `embedding` for embeddings. +7. Use arrow-key selectors for chat, embedding, and image-description models. +8. Require one embedding model. +9. Load selected models with `POST /api/v1/models/load`. +10. Save config to `~/.openmind/config.toml`. +11. Use a checkbox selector for folders, with a custom-folder option. +12. Start background indexing. +13. Tell the user to run `openmind index status`. ## Config Format @@ -640,6 +661,55 @@ openmind dev logs --lm-studio `--lm-studio` runs `lms log stream`, matching LM Studio's own development guidance for inspecting model input. +## Local API + +`openmind serve` starts a single-process FastAPI application at `127.0.0.1:8765`. The CLI does not expose a host option; remote network binding is outside the `0.0.5` security model. + +The public liveness route is: + +```text +GET /health +``` + +All product routes are versioned under `/api/v1` and require: + +```http +Authorization: Bearer +``` + +The API token is generated with Python's `secrets` module, stored at `~/.openmind/api_token`, restricted to mode `0600` on POSIX platforms, and compared with `secrets.compare_digest`. The server reads the current token for authenticated requests so rotation takes effect without a restart. Token values must not be written to OpenMind or Uvicorn access logs. + +Protected capabilities: + +- system and indexing status +- provider status and model discovery +- validated model selection and loading +- source listing, addition, and removal +- background indexing start, pause, resume, status, and stop +- search with structured source records +- synchronous and server-sent-event Ask, including structured source events +- indexed file and chunk details +- opening an indexed file in its default operating-system application + +The open-file action accepts only a generated file ID. Before launching an OS application, OpenMind must verify that the file record is indexed, the file still exists, and its fully resolved path remains beneath an enabled source directory. + +API schemas reject unknown request fields. Queries, questions, paths, model-key lists, result limits, and file IDs are bounded and validated. Browser CORS is disabled by default. `--allow-origin` accepts exact HTTP or HTTPS origins and refuses wildcard, credential-bearing, path-bearing, query-bearing, or fragment-bearing values. + +The API must not expose: + +- API token values in responses other than the explicit CLI token command +- arbitrary local paths for open or read actions +- raw files or raw image bytes +- embedding vectors +- raw SQLite or LanceDB operations +- manual chunk insertion +- manual embedding or extractor operations +- shell command execution + +FastAPI lifespan initializes one shared `OpenMindEngine` before requests are accepted. Route handlers call engine capabilities rather than reaching around the engine into database implementation details, except read-only indexed document lookup needed to expose sanitized chunk text. + +The client contract and examples are documented in [API.md](API.md). + ## Acceptance Tests The first acceptable build must prove: @@ -658,3 +728,11 @@ The first acceptable build must prove: - Config can save and load selected provider/model settings. - SQLite can create and update indexing job status. - LM Studio ask returns a clear message when the server is unreachable. +- Public health works without a token while all `/api/v1` routes reject missing or invalid tokens. +- The OpenAPI schema declares bearer security for private routes. +- API token files use private permissions and can be rotated. +- The server CLI always binds Uvicorn to `127.0.0.1`. +- Wildcard CORS and malformed request bodies are rejected. +- Search, Ask, streaming Ask, source management, model selection, and indexing controls work through the API. +- Document lookup omits vectors. +- Open-file actions reject files outside enabled source folders. diff --git a/openmind/__init__.py b/openmind/__init__.py index 81f0fde..b1a19e3 100644 --- a/openmind/__init__.py +++ b/openmind/__init__.py @@ -1 +1 @@ -__version__ = "0.0.4" +__version__ = "0.0.5" diff --git a/openmind/api/__init__.py b/openmind/api/__init__.py new file mode 100644 index 0000000..8fa75a3 --- /dev/null +++ b/openmind/api/__init__.py @@ -0,0 +1,5 @@ +"""Local HTTP API for OpenMind clients.""" + +from openmind.api.app import create_app + +__all__ = ["create_app"] diff --git a/openmind/api/app.py b/openmind/api/app.py new file mode 100644 index 0000000..e912f10 --- /dev/null +++ b/openmind/api/app.py @@ -0,0 +1,69 @@ +from __future__ import annotations + +from collections.abc import Callable +from contextlib import asynccontextmanager +from pathlib import Path + +from fastapi import Depends, FastAPI, Request +from fastapi.middleware.cors import CORSMiddleware +from fastapi.responses import JSONResponse + +from openmind import __version__ +from openmind.api.auth import ensure_api_token, require_api_token +from openmind.api.cors import validate_cors_origin +from openmind.api.files import open_local_file +from openmind.api.routes import actions, indexing, memory, models, sources, system +from openmind.core.engine import OpenMindEngine +from openmind.providers.lmstudio.errors import LMStudioError + +API_PREFIX = "/api/v1" + + +def create_app( + engine: OpenMindEngine | None = None, + api_token: str | None = None, + allowed_origins: list[str] | None = None, + file_opener: Callable[[Path], None] | None = None, +) -> FastAPI: + @asynccontextmanager + async def lifespan(app: FastAPI): + current = engine or OpenMindEngine() + current.init() + app.state.engine = current + if api_token is None: + app.state.api_token_loader = lambda: ensure_api_token(current.paths.home) + else: + app.state.api_token_loader = lambda: api_token + app.state.file_opener = file_opener or open_local_file + yield + + app = FastAPI( + title="OpenMind Core API", + version=__version__, + description="Local, authenticated API for OpenMind client applications.", + lifespan=lifespan, + ) + origins = [validate_cors_origin(origin) for origin in (allowed_origins or [])] + if origins: + app.add_middleware( + CORSMiddleware, + allow_origins=origins, + allow_credentials=False, + allow_methods=["GET", "POST", "PUT", "DELETE"], + allow_headers=["Authorization", "Content-Type"], + ) + + protected = [Depends(require_api_token)] + app.include_router(system.public_router) + app.include_router(system.router, prefix=API_PREFIX, dependencies=protected) + app.include_router(models.router, prefix=API_PREFIX, dependencies=protected) + app.include_router(sources.router, prefix=API_PREFIX, dependencies=protected) + app.include_router(indexing.router, prefix=API_PREFIX, dependencies=protected) + app.include_router(memory.router, prefix=API_PREFIX, dependencies=protected) + app.include_router(actions.router, prefix=API_PREFIX, dependencies=protected) + + @app.exception_handler(LMStudioError) + async def provider_error(request: Request, exc: LMStudioError) -> JSONResponse: + return JSONResponse(status_code=503, content={"detail": str(exc)}) + + return app diff --git a/openmind/api/auth.py b/openmind/api/auth.py new file mode 100644 index 0000000..9243b98 --- /dev/null +++ b/openmind/api/auth.py @@ -0,0 +1,81 @@ +from __future__ import annotations + +import os +import secrets +from pathlib import Path +from typing import Annotated + +from fastapi import HTTPException, Request, Security, status +from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer + +TOKEN_FILE_NAME = "api_token" +TOKEN_BYTES = 32 + +bearer_scheme = HTTPBearer(auto_error=False, scheme_name="OpenMind API token") + + +def token_path(home: Path) -> Path: + return home / TOKEN_FILE_NAME + + +def ensure_api_token(home: Path) -> str: + home.mkdir(parents=True, exist_ok=True) + path = token_path(home) + if path.exists(): + token = path.read_text(encoding="utf-8").strip() + if len(token) < 32: + raise RuntimeError(f"OpenMind API token is invalid: {path}") + _secure_permissions(path) + return token + return _write_new_token(path) + + +def rotate_api_token(home: Path) -> str: + home.mkdir(parents=True, exist_ok=True) + path = token_path(home) + token = secrets.token_urlsafe(TOKEN_BYTES) + temporary = path.with_name(f".{path.name}.{secrets.token_hex(6)}.tmp") + _write_private_file(temporary, token) + temporary.replace(path) + _secure_permissions(path) + return token + + +def require_api_token( + request: Request, + credentials: Annotated[ + HTTPAuthorizationCredentials | None, + Security(bearer_scheme), + ], +) -> None: + expected = request.app.state.api_token_loader() + supplied = credentials.credentials if credentials is not None else "" + if not supplied or not secrets.compare_digest(supplied, expected): + raise HTTPException( + status_code=status.HTTP_401_UNAUTHORIZED, + detail="A valid OpenMind API bearer token is required.", + headers={"WWW-Authenticate": "Bearer"}, + ) + + +def _write_new_token(path: Path) -> str: + token = secrets.token_urlsafe(TOKEN_BYTES) + try: + descriptor = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + except FileExistsError: + return ensure_api_token(path.parent) + with os.fdopen(descriptor, "w", encoding="utf-8") as handle: + handle.write(token + "\n") + _secure_permissions(path) + return token + + +def _write_private_file(path: Path, token: str) -> None: + descriptor = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + with os.fdopen(descriptor, "w", encoding="utf-8") as handle: + handle.write(token + "\n") + + +def _secure_permissions(path: Path) -> None: + if os.name != "nt": + path.chmod(0o600) diff --git a/openmind/api/cors.py b/openmind/api/cors.py new file mode 100644 index 0000000..558fac8 --- /dev/null +++ b/openmind/api/cors.py @@ -0,0 +1,27 @@ +from __future__ import annotations + +from urllib.parse import urlsplit + + +def validate_cors_origin(origin: str) -> str: + value = origin.strip().rstrip("/") + parsed = urlsplit(value) + try: + parsed.port + except ValueError as exc: + raise ValueError("CORS origins must use a valid port.") from exc + if ( + value == "*" + or parsed.scheme not in {"http", "https"} + or not parsed.hostname + or "*" in parsed.hostname + or parsed.username + or parsed.password + or parsed.path + or parsed.query + or parsed.fragment + ): + raise ValueError( + "CORS origins must be exact http(s) origins such as http://localhost:3000." + ) + return value diff --git a/openmind/api/deps.py b/openmind/api/deps.py new file mode 100644 index 0000000..bdd7e11 --- /dev/null +++ b/openmind/api/deps.py @@ -0,0 +1,14 @@ +from __future__ import annotations + +from typing import Annotated + +from fastapi import Depends, Request + +from openmind.core.engine import OpenMindEngine + + +def get_engine(request: Request) -> OpenMindEngine: + return request.app.state.engine + + +EngineDependency = Annotated[OpenMindEngine, Depends(get_engine)] diff --git a/openmind/api/files.py b/openmind/api/files.py new file mode 100644 index 0000000..87c7e8e --- /dev/null +++ b/openmind/api/files.py @@ -0,0 +1,31 @@ +from __future__ import annotations + +import os +import subprocess +import sys +from pathlib import Path + + +def open_local_file(path: Path) -> None: + if sys.platform == "darwin": + command = ["open", str(path)] + elif os.name == "nt": + os.startfile(str(path)) # type: ignore[attr-defined] + return + else: + command = ["xdg-open", str(path)] + subprocess.Popen( + command, + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + start_new_session=True, + ) + + +def is_path_inside(path: Path, directory: Path) -> bool: + try: + path.relative_to(directory) + return True + except ValueError: + return False diff --git a/openmind/api/routes/__init__.py b/openmind/api/routes/__init__.py new file mode 100644 index 0000000..d6751ab --- /dev/null +++ b/openmind/api/routes/__init__.py @@ -0,0 +1 @@ +"""HTTP route modules for the OpenMind API.""" diff --git a/openmind/api/routes/actions.py b/openmind/api/routes/actions.py new file mode 100644 index 0000000..cfab932 --- /dev/null +++ b/openmind/api/routes/actions.py @@ -0,0 +1,39 @@ +from __future__ import annotations + +from pathlib import Path + +from fastapi import APIRouter, HTTPException, Request, status + +from openmind.api.deps import EngineDependency +from openmind.api.files import is_path_inside +from openmind.api.schemas import OpenFileRequest, OpenFileResponse + +router = APIRouter(prefix="/actions", tags=["actions"]) + + +@router.post("/open", response_model=OpenFileResponse) +def open_file( + payload: OpenFileRequest, + request: Request, + engine: EngineDependency, +) -> OpenFileResponse: + record = engine.sqlite.file_by_id(payload.file_id) + if record is None or record.status != "indexed": + raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Indexed file not found.") + + resolved = Path(record.path).expanduser().resolve() + if not resolved.is_file(): + raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="File no longer exists.") + allowed = any( + is_path_inside(resolved, Path(source.path).expanduser().resolve()) + for source in engine.list_sources() + if source.enabled + ) + if not allowed: + raise HTTPException( + status_code=status.HTTP_403_FORBIDDEN, + detail="The file is outside enabled OpenMind sources.", + ) + + request.app.state.file_opener(resolved) + return OpenFileResponse(opened=True, file_id=record.id, path=str(resolved)) diff --git a/openmind/api/routes/indexing.py b/openmind/api/routes/indexing.py new file mode 100644 index 0000000..74b8099 --- /dev/null +++ b/openmind/api/routes/indexing.py @@ -0,0 +1,82 @@ +from __future__ import annotations + +from fastapi import APIRouter, HTTPException, status + +from openmind.api.deps import EngineDependency +from openmind.api.schemas import IndexJobResponse +from openmind.core.models import IndexJob + +router = APIRouter(prefix="/index", tags=["indexing"]) + + +@router.post("/start", response_model=IndexJobResponse, status_code=status.HTTP_202_ACCEPTED) +def start_index(engine: EngineDependency) -> IndexJobResponse: + return _job_response(engine.start_index_job()) + + +@router.get("/status", response_model=IndexJobResponse) +def index_status(engine: EngineDependency) -> IndexJobResponse: + job = engine.index_job_status() + return _job_response(job) + + +@router.post("/pause", response_model=IndexJobResponse) +def pause_index(engine: EngineDependency) -> IndexJobResponse: + return _required_job(engine.pause_index_job()) + + +@router.post("/resume", response_model=IndexJobResponse) +def resume_index(engine: EngineDependency) -> IndexJobResponse: + return _required_job(engine.resume_index_job()) + + +@router.post("/stop", response_model=IndexJobResponse) +def stop_index(engine: EngineDependency) -> IndexJobResponse: + return _required_job(engine.stop_index_job()) + + +def _required_job(job: IndexJob | None) -> IndexJobResponse: + if job is None: + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail="No indexing job has been started.", + ) + return _job_response(job) + + +def _job_response(job: IndexJob | None) -> IndexJobResponse: + if job is None: + return IndexJobResponse( + job_id=None, + state="idle", + total_files=0, + processed_files=0, + indexed_files=0, + skipped_files=0, + already_indexed_files=0, + failed_files=0, + chunks_created=0, + current_file=None, + error=None, + progress=0.0, + started_at=None, + completed_at=None, + updated_at=None, + ) + return IndexJobResponse( + job_id=job.id, + state=job.status, + total_files=job.total_files, + processed_files=job.processed_files, + indexed_files=job.indexed_files, + skipped_files=job.skipped_files, + already_indexed_files=job.already_indexed_files, + failed_files=job.failed_files, + chunks_created=job.total_chunks, + current_file=job.current_file, + error=job.error, + progress=job.progress_percent, + started_at=job.started_at, + completed_at=job.completed_at, + updated_at=job.updated_at, + ) diff --git a/openmind/api/routes/memory.py b/openmind/api/routes/memory.py new file mode 100644 index 0000000..9282be9 --- /dev/null +++ b/openmind/api/routes/memory.py @@ -0,0 +1,120 @@ +from __future__ import annotations + +import json +from collections.abc import Iterator +from typing import Annotated + +from fastapi import APIRouter, HTTPException, Path, status +from fastapi.responses import StreamingResponse + +from openmind.api.deps import EngineDependency +from openmind.api.schemas import ( + AskRequest, + AskResponse, + DocumentChunkResponse, + DocumentResponse, + SearchRequest, + SearchResponse, + SearchResultResponse, +) +from openmind.core.models import SearchResult + +router = APIRouter(tags=["memory"]) + + +@router.post("/search", response_model=SearchResponse) +def search(request: SearchRequest, engine: EngineDependency) -> SearchResponse: + results = engine.search(request.query, limit=request.limit) + return SearchResponse( + query=request.query, + results=[_search_result(result) for result in results], + ) + + +@router.post("/ask", response_model=AskResponse) +def ask(request: AskRequest, engine: EngineDependency) -> AskResponse: + answer, results = engine.ask_with_sources(request.question, limit=request.limit) + sources = [_search_result(result) for result in results] if request.include_sources else [] + return AskResponse(answer=answer, sources=sources) + + +@router.post("/ask/stream") +def ask_stream(request: AskRequest, engine: EngineDependency) -> StreamingResponse: + stream, results = engine.ask_stream_with_sources(request.question, limit=request.limit) + + def events() -> Iterator[str]: + try: + for chunk in stream: + yield _sse("delta", {"text": chunk}) + if request.include_sources: + yield _sse( + "sources", + { + "sources": [ + _search_result(result).model_dump(mode="json") for result in results + ] + }, + ) + yield _sse("done", {}) + except Exception: + yield _sse("error", {"message": "OpenMind could not complete the answer."}) + + return StreamingResponse( + events(), + media_type="text/event-stream", + headers={ + "Cache-Control": "no-cache", + "X-Accel-Buffering": "no", + }, + ) + + +@router.get("/documents/{file_id}", response_model=DocumentResponse) +def document( + file_id: Annotated[str, Path(pattern=r"^file_[0-9a-f]{16}$")], + engine: EngineDependency, +) -> DocumentResponse: + record = engine.sqlite.file_by_id(file_id) + if record is None or record.status != "indexed": + raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Document not found.") + chunks = engine.lance.chunks_for_file(file_id) + return DocumentResponse( + id=record.id, + source_id=record.source_id, + path=record.path, + file_name=record.name, + extension=record.extension, + status=record.status, + size=record.size, + modified_at=record.modified_at, + indexed_at=record.indexed_at, + error=record.error, + chunks=[DocumentChunkResponse(**chunk) for chunk in chunks], + ) + + +def _search_result(result: SearchResult) -> SearchResultResponse: + return SearchResultResponse( + id=result.id, + file_id=result.file_id, + source_id=result.source_id, + score=result.score, + source_type=result.extension.lstrip(".") or _source_type(result), + path=result.path, + file_name=result.file_name, + title=result.title, + snippet=result.snippet, + chunk_index=result.chunk_index, + metadata=result.metadata, + ) + + +def _source_type(result: SearchResult) -> str: + suffix = result.metadata.get("extension") or "" + if not suffix: + suffix = "." + result.file_name.rsplit(".", 1)[-1] if "." in result.file_name else "file" + return str(suffix).lstrip(".") or "file" + + +def _sse(event: str, data: dict) -> str: + return f"event: {event}\ndata: {json.dumps(data, ensure_ascii=True)}\n\n" diff --git a/openmind/api/routes/models.py b/openmind/api/routes/models.py new file mode 100644 index 0000000..71725f3 --- /dev/null +++ b/openmind/api/routes/models.py @@ -0,0 +1,125 @@ +from __future__ import annotations + +from fastapi import APIRouter, HTTPException, status + +from openmind.api.deps import EngineDependency +from openmind.api.schemas import ( + ModelInfo, + ModelLoadRequest, + ModelLoadResponse, + ModelLoadResult, + ModelSelectionRequest, + ModelSelectionResponse, + ModelsResponse, +) +from openmind.providers.lmstudio.models import LMStudioModel, split_models, vision_models + +router = APIRouter(prefix="/models", tags=["models"]) + + +@router.get("", response_model=ModelsResponse) +def list_models(engine: EngineDependency) -> ModelsResponse: + models = engine.list_lmstudio_models() + chat_models, embedding_models = split_models(models) + image_models = vision_models(models) + return ModelsResponse( + provider="lmstudio", + chat_models=[_model_info(model) for model in chat_models], + embedding_models=[_model_info(model) for model in embedding_models], + image_models=[_model_info(model) for model in image_models], + ) + + +@router.post("/load", response_model=ModelLoadResponse) +def load_models(request: ModelLoadRequest, engine: EngineDependency) -> ModelLoadResponse: + if request.model_keys is None: + results = engine.load_configured_models() + else: + available = {model.key for model in engine.list_lmstudio_models()} + unknown = [key for key in request.model_keys if key not in available] + if unknown: + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail=f"Unknown model key(s): {', '.join(unknown)}", + ) + client = engine.lmstudio_client() + results = [client.load_model_if_needed(key) for key in request.model_keys] + return ModelLoadResponse(results=[_load_result(result) for result in results]) + + +@router.put("/selection", response_model=ModelSelectionResponse) +def select_models( + request: ModelSelectionRequest, + engine: EngineDependency, +) -> ModelSelectionResponse: + models = engine.list_lmstudio_models() + by_key = {model.key: model for model in models} + _require_model_type(by_key, request.embedding_model, "embedding") + if request.chat_model: + _require_model_type(by_key, request.chat_model, "llm") + if request.image_model: + model = _require_model_type(by_key, request.image_model, "llm") + if not model.supports_images: + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail=f"Model does not advertise image support: {request.image_model}", + ) + + config = engine.config.model_copy(deep=True) + config.provider.name = "lmstudio" + config.models.chat_model = request.chat_model or "" + config.models.embedding_model = request.embedding_model + config.extraction.images.enabled = request.image_model is not None + if request.image_model: + config.extraction.images.model = request.image_model + engine.save_config(config) + + load_results: list[dict] = [] + if request.load: + load_results = engine.load_configured_models() + return ModelSelectionResponse( + provider="lmstudio", + chat_model=request.chat_model, + embedding_model=request.embedding_model, + image_model=request.image_model, + load_results=[_load_result(result) for result in load_results], + ) + + +def _model_info(model: LMStudioModel) -> ModelInfo: + return ModelInfo( + key=model.key, + name=model.display_name, + type=model.type, + loaded=model.is_loaded, + supports_images=model.supports_images, + max_context_length=model.max_context_length, + quantization=model.quantization, + ) + + +def _load_result(result: dict) -> ModelLoadResult: + return ModelLoadResult( + model=str(result.get("model", "")), + status=str(result.get("status", "loaded")), + skipped=bool(result.get("skipped", False)), + ) + + +def _require_model_type( + models: dict[str, LMStudioModel], + model_key: str, + expected_type: str, +) -> LMStudioModel: + model = models.get(model_key) + if model is None: + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail=f"Unknown model key: {model_key}", + ) + if model.type != expected_type: + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail=f"Model {model_key} is not a {expected_type} model.", + ) + return model diff --git a/openmind/api/routes/sources.py b/openmind/api/routes/sources.py new file mode 100644 index 0000000..ec7db9a --- /dev/null +++ b/openmind/api/routes/sources.py @@ -0,0 +1,39 @@ +from __future__ import annotations + +import sqlite3 + +from fastapi import APIRouter, HTTPException, status + +from openmind.api.deps import EngineDependency +from openmind.api.schemas import SourceCreateRequest, SourceListResponse, SourceResponse + +router = APIRouter(prefix="/sources", tags=["sources"]) + + +@router.get("", response_model=SourceListResponse) +def list_sources(engine: EngineDependency) -> SourceListResponse: + return SourceListResponse( + sources=[SourceResponse(**source.model_dump()) for source in engine.list_sources()] + ) + + +@router.post("", response_model=SourceResponse, status_code=status.HTTP_201_CREATED) +def add_source(request: SourceCreateRequest, engine: EngineDependency) -> SourceResponse: + try: + source = engine.add_source(request.path, recursive=request.recursive) + except sqlite3.IntegrityError as exc: + raise HTTPException( + status_code=status.HTTP_409_CONFLICT, + detail="This source folder is already registered.", + ) from exc + except (FileNotFoundError, NotADirectoryError) as exc: + raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=str(exc)) from exc + return SourceResponse(**source.model_dump()) + + +@router.delete("/{source_id}", status_code=status.HTTP_204_NO_CONTENT) +def remove_source(source_id: str, engine: EngineDependency) -> None: + if not source_id.startswith("src_") or len(source_id) > 64: + raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Source not found.") + if not engine.remove_source(source_id): + raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Source not found.") diff --git a/openmind/api/routes/system.py b/openmind/api/routes/system.py new file mode 100644 index 0000000..f089053 --- /dev/null +++ b/openmind/api/routes/system.py @@ -0,0 +1,68 @@ +from __future__ import annotations + +from fastapi import APIRouter + +from openmind import __version__ +from openmind.api.deps import EngineDependency +from openmind.api.schemas import ( + HealthResponse, + ProviderInfo, + ProvidersResponse, + ProviderStatusResponse, + StatusResponse, +) + +public_router = APIRouter(tags=["system"]) +router = APIRouter(tags=["system"]) + + +@public_router.get("/health", response_model=HealthResponse) +def health() -> HealthResponse: + return HealthResponse(status="ok", version=__version__) + + +@router.get("/status", response_model=StatusResponse) +def status(engine: EngineDependency) -> StatusResponse: + summary = engine.status() + job = engine.index_job_status() + active_states = {"pending", "discovering", "running", "pause_requested", "paused"} + indexing_state = job.status if job and job.status in active_states else "idle" + image_model = ( + engine.config.extraction.images.model if engine.config.extraction.images.enabled else None + ) + return StatusResponse( + status="ready", + version=__version__, + provider=engine.config.provider.name, + chat_model=engine.config.models.chat_model or None, + embedding_model=engine.config.models.embedding_model or None, + image_model=image_model, + sources=summary.sources, + indexed_files=summary.indexed_files, + indexed_chunks=engine.lance.count_chunks(), + indexing_state=indexing_state, + last_index_job_status=job.status if job else None, + ) + + +@router.get("/providers/status", response_model=ProviderStatusResponse) +def provider_status(engine: EngineDependency) -> ProviderStatusResponse: + reachable, message = engine.provider_status() + return ProviderStatusResponse( + provider=engine.config.provider.name, + reachable=reachable, + message=message, + ) + + +@router.get("/providers", response_model=ProvidersResponse) +def providers(engine: EngineDependency) -> ProvidersResponse: + return ProvidersResponse( + providers=[ + ProviderInfo( + name="lmstudio", + display_name="LM Studio", + configured=engine.config.provider.name == "lmstudio", + ) + ] + ) diff --git a/openmind/api/schemas.py b/openmind/api/schemas.py new file mode 100644 index 0000000..5877e97 --- /dev/null +++ b/openmind/api/schemas.py @@ -0,0 +1,198 @@ +from __future__ import annotations + +from typing import Annotated, Any + +from pydantic import BaseModel, ConfigDict, Field, StringConstraints + +QueryText = Annotated[str, StringConstraints(strip_whitespace=True, min_length=1, max_length=4000)] +ModelKey = Annotated[str, StringConstraints(strip_whitespace=True, min_length=1, max_length=300)] + + +class ApiSchema(BaseModel): + model_config = ConfigDict(extra="forbid") + + +class HealthResponse(ApiSchema): + status: str + version: str + + +class StatusResponse(ApiSchema): + status: str + version: str + provider: str + chat_model: str | None + embedding_model: str | None + image_model: str | None + sources: int + indexed_files: int + indexed_chunks: int + indexing_state: str + last_index_job_status: str | None + + +class ProviderStatusResponse(ApiSchema): + provider: str + reachable: bool + message: str + + +class ProviderInfo(ApiSchema): + name: str + display_name: str + configured: bool + + +class ProvidersResponse(ApiSchema): + providers: list[ProviderInfo] + + +class ModelInfo(ApiSchema): + key: str + name: str + type: str + loaded: bool + supports_images: bool + max_context_length: int | None = None + quantization: dict[str, Any] | None = None + + +class ModelsResponse(ApiSchema): + provider: str + chat_models: list[ModelInfo] + embedding_models: list[ModelInfo] + image_models: list[ModelInfo] + + +class ModelLoadRequest(ApiSchema): + model_keys: list[ModelKey] | None = Field(default=None, min_length=1, max_length=3) + + +class ModelLoadResult(ApiSchema): + model: str + status: str + skipped: bool + + +class ModelLoadResponse(ApiSchema): + results: list[ModelLoadResult] + + +class ModelSelectionRequest(ApiSchema): + chat_model: ModelKey | None = None + embedding_model: ModelKey + image_model: ModelKey | None = None + load: bool = True + + +class ModelSelectionResponse(ApiSchema): + provider: str + chat_model: str | None + embedding_model: str + image_model: str | None + load_results: list[ModelLoadResult] + + +class SourceCreateRequest(ApiSchema): + path: Annotated[ + str, + StringConstraints(strip_whitespace=True, min_length=1, max_length=4096), + ] + recursive: bool = True + + +class SourceResponse(ApiSchema): + id: str + path: str + recursive: bool + enabled: bool + created_at: str + + +class SourceListResponse(ApiSchema): + sources: list[SourceResponse] + + +class IndexJobResponse(ApiSchema): + job_id: str | None + state: str + total_files: int + processed_files: int + indexed_files: int + skipped_files: int + already_indexed_files: int + failed_files: int + chunks_created: int + current_file: str | None + error: str | None + progress: float + started_at: str | None + completed_at: str | None + updated_at: str | None + + +class SearchRequest(ApiSchema): + query: QueryText + limit: int = Field(default=5, ge=1, le=50) + + +class SearchResultResponse(ApiSchema): + id: str + file_id: str + source_id: str + score: float + source_type: str + path: str + file_name: str + title: str + snippet: str + chunk_index: int + metadata: dict[str, Any] + + +class SearchResponse(ApiSchema): + query: str + results: list[SearchResultResponse] + + +class AskRequest(ApiSchema): + question: QueryText + limit: int = Field(default=5, ge=1, le=20) + include_sources: bool = True + + +class AskResponse(ApiSchema): + answer: str + sources: list[SearchResultResponse] + + +class DocumentChunkResponse(ApiSchema): + id: str + text: str + chunk_index: int + title: str + metadata: dict[str, Any] + + +class DocumentResponse(ApiSchema): + id: str + source_id: str + path: str + file_name: str + extension: str + status: str + size: int + modified_at: float + indexed_at: str | None + error: str | None + chunks: list[DocumentChunkResponse] + + +class OpenFileRequest(ApiSchema): + file_id: str = Field(pattern=r"^file_[0-9a-f]{16}$") + + +class OpenFileResponse(ApiSchema): + opened: bool + file_id: str + path: str diff --git a/openmind/cli/main.py b/openmind/cli/main.py index 1004987..b672218 100644 --- a/openmind/cli/main.py +++ b/openmind/cli/main.py @@ -7,14 +7,18 @@ import sys import time from pathlib import Path +from typing import Any +import questionary import typer +from questionary import Choice, Style from rich.console import Console from rich.live import Live from rich.progress import Progress from rich.prompt import Prompt from rich.table import Table +from openmind import __version__ from openmind.core.config import ( DEFAULT_IMAGE_DESCRIPTION_MODEL, DEFAULT_LMSTUDIO_BASE_URL, @@ -34,19 +38,65 @@ models_app = typer.Typer(help="Manage LM Studio models.") provider_app = typer.Typer(help="Inspect provider status.") dev_app = typer.Typer(help="Developer tools.") +api_app = typer.Typer(help="Manage local API access.") app.add_typer(source_app, name="source") app.add_typer(index_app, name="index") app.add_typer(models_app, name="models") app.add_typer(provider_app, name="provider") app.add_typer(dev_app, name="dev") +app.add_typer(api_app, name="api") console = Console() +OPENMIND_BANNER = r""" + ___ __ __ _ _ + / _ \ _ __ ___ _ __ | \/ (_)_ __ __| | +| | | | '_ \ / _ \ '_ \| |\/| | | '_ \ / _` | +| |_| | |_) | __/ | | | | | | | | | | (_| | + \___/| .__/ \___|_| |_|_| |_|_|_| |_|\__,_| + |_| +""".strip("\n") + +PROMPT_STYLE = Style( + [ + ("qmark", "fg:#00d7af bold"), + ("question", "bold"), + ("answer", "fg:#00d7af bold"), + ("pointer", "fg:#00d7af bold"), + ("highlighted", "fg:#00d7af bold"), + ("selected", "fg:#00d7af"), + ("instruction", "fg:#808080"), + ] +) + +NO_MODEL = "__openmind_no_model__" +CUSTOM_FOLDER = "__openmind_custom_folder__" + + +def _version_callback(value: bool) -> None: + if value: + console.print(f"openmind {__version__}") + raise typer.Exit() + + +@app.callback() +def main( + version: bool = typer.Option( + False, + "--version", + "-V", + callback=_version_callback, + is_eager=True, + help="Show the installed OpenMind version and exit.", + ), +) -> None: + pass + def engine() -> OpenMindEngine: return OpenMindEngine() -@app.command("init") +@app.command("init", help="Initialize OpenMind's local app data.") def init_command() -> None: paths = engine().init() console.print("[green]OpenMind initialized[/green]") @@ -55,10 +105,11 @@ def init_command() -> None: console.print(f"LanceDB: {paths.lancedb_path}") -@app.command("setup") +@app.command("setup", help="Configure models, sources, and background indexing.") def setup_command() -> None: current = engine() paths = current.init() + console.print(f"[bold cyan]{OPENMIND_BANNER}[/bold cyan]") console.print("[bold]Welcome to OpenMind.[/bold]") console.print("OpenMind creates a private AI memory over your local files.") console.print("Checking local environment...") @@ -66,13 +117,15 @@ def setup_command() -> None: console.print("[green]✓[/green] SQLite ready") console.print("[green]✓[/green] LanceDB ready") - console.print("Choose AI provider:") - console.print("1. LM Studio") - provider_choice = typer.prompt("Selected", default="1") - if provider_choice.strip() != "1": + provider_choice = _select_prompt( + "Choose AI provider", + choices=[Choice("LM Studio", value="lmstudio")], + default="lmstudio", + ) + if provider_choice != "lmstudio": raise typer.BadParameter("Only LM Studio is supported right now.") - base_url = typer.prompt("LM Studio base URL", default=DEFAULT_LMSTUDIO_BASE_URL) + base_url = _text_prompt("LM Studio base URL", default=DEFAULT_LMSTUDIO_BASE_URL) config = OpenMindConfig( provider=ProviderSettings(name="lmstudio", base_url=base_url, api_token_env="LM_API_TOKEN"), models=ModelSettings(chat_model="", embedding_model=""), @@ -331,10 +384,12 @@ def models_update( else "" ) - console.print("Choose AI provider:") - console.print("1. LM Studio") - provider_choice = typer.prompt("Selected", default="1") - if provider_choice.strip() != "1": + provider_choice = _select_prompt( + "Choose AI provider", + choices=[Choice("LM Studio", value="lmstudio")], + default="lmstudio", + ) + if provider_choice != "lmstudio": raise typer.BadParameter("Only LM Studio is supported right now.") default_base_url = ( @@ -342,7 +397,7 @@ def models_update( if config.provider.name == "lmstudio" else DEFAULT_LMSTUDIO_BASE_URL ) - base_url = typer.prompt("LM Studio base URL", default=default_base_url) + base_url = _text_prompt("LM Studio base URL", default=default_base_url) config.provider = ProviderSettings( name="lmstudio", base_url=base_url, @@ -419,6 +474,70 @@ def provider_status() -> None: console.print(f"[{color}]{message}[/{color}]") +@app.command("serve", help="Start the authenticated local OpenMind API.") +def serve_command( + port: int = typer.Option(8765, min=1, max=65535, help="Local API port."), + allow_origin: list[str] | None = typer.Option( + None, + "--allow-origin", + help="Allow an exact browser origin. Repeat for multiple origins; wildcards are refused.", + ), +) -> None: + from openmind.api.app import create_app + from openmind.api.auth import ensure_api_token, token_path + + current = engine() + current.init() + ensure_api_token(current.paths.home) + origins = [_validate_api_origin(origin) for origin in (allow_origin or [])] + + console.print("[bold]OpenMind local API[/bold]") + console.print(f"Server: http://127.0.0.1:{port}") + console.print(f"Docs: http://127.0.0.1:{port}/docs") + console.print(f"API token: {token_path(current.paths.home)}") + console.print("The server accepts local connections only. Press Ctrl+C to stop.") + + import uvicorn + + uvicorn.run( + create_app( + engine=current, + allowed_origins=origins, + ), + host="127.0.0.1", + port=port, + log_level="info", + access_log=False, + ) + + +@api_app.command("token", help="Show or rotate the local API bearer token.") +def api_token_command( + rotate: bool = typer.Option( + False, + "--rotate", + help="Replace the API token and invalidate existing client credentials.", + ), + yes: bool = typer.Option(False, "--yes", "-y", help="Skip rotation confirmation."), +) -> None: + from openmind.api.auth import ensure_api_token, rotate_api_token + + current = engine() + current.init() + if rotate: + if not yes and not typer.confirm( + "Rotate the API token and disconnect clients using the current token?", + default=False, + ): + console.print("[yellow]Token rotation cancelled.[/yellow]") + return + token = rotate_api_token(current.paths.home) + console.print("[green]API token rotated.[/green]") + else: + token = ensure_api_token(current.paths.home) + console.print(token, markup=False, highlight=False) + + @dev_app.command("logs") def dev_logs( follow: bool = typer.Option(True, "--follow/--no-follow", help="Keep watching for new log lines."), @@ -447,7 +566,7 @@ def dev_logs( _tail_log_files(log_files, lines=lines, follow=follow) -@app.command("search") +@app.command("search", help="Search indexed local memory.") def search_command(query: str, limit: int = typer.Option(5, min=1, max=50)) -> None: try: results = engine().search(query, limit=limit) @@ -464,7 +583,7 @@ def search_command(query: str, limit: int = typer.Option(5, min=1, max=50)) -> N console.print(f" Snippet: {result.snippet}") -@app.command("ask") +@app.command("ask", help="Ask grounded questions or start an interactive session.") def ask_command( question: str | None = typer.Argument(None), limit: int = typer.Option(5, min=1, max=20), @@ -496,7 +615,7 @@ def ask_command( raise typer.Exit(1) from exc -@app.command("status") +@app.command("status", help="Show OpenMind storage and indexing information.") def status_command() -> None: status = engine().status() table = Table(title="OpenMind Status") @@ -510,7 +629,7 @@ def status_command() -> None: console.print(table) -@app.command("flush") +@app.command("flush", help="Clear indexed memory without deleting user files.") def flush_command( yes: bool = typer.Option( False, @@ -552,6 +671,7 @@ def flush_command( console.print() console.print("Will keep:") console.print(f"- Config: {current.paths.config_path}") + console.print(f"- Local API token: {home / 'api_token'}") if not include_sources: console.print("- Saved source folder records") console.print("- User source folders and files") @@ -608,7 +728,7 @@ def flush_command( console.print("Run `openmind index start` to build memory again.") -@app.command("uninstall") +@app.command("uninstall", help="Remove OpenMind local data and optionally the package.") def uninstall_command( yes: bool = typer.Option( False, @@ -635,6 +755,7 @@ def uninstall_command( console.print("This removes OpenMind-owned local data:") console.print(f"- App home: {home}") console.print(f"- Config: {current.paths.config_path}") + console.print(f"- Local API token: {home / 'api_token'}") console.print(f"- SQLite state: {current.paths.sqlite_path}") console.print(f"- LanceDB memory: {current.paths.lancedb_path}") console.print(f"- Logs: {current.paths.logs_path}") @@ -730,6 +851,7 @@ def _choose_model( title: str, models: list[LMStudioModel], allow_empty: bool, + empty_label: str = "Search-only mode (no chat model)", ) -> LMStudioModel | None: if not models: if allow_empty: @@ -740,16 +862,13 @@ def _choose_model( console.print("Download one manually in LM Studio, then run setup again.") raise typer.Exit(1) - console.print(title + ":") - for index, model in enumerate(models, start=1): - loaded = " loaded" if model.is_loaded else "" - console.print(f"{index}. {model.display_name} ({model.key}){loaded}") - choice = typer.prompt("Choose model", default="1") - try: - selected_index = int(choice) - 1 - return models[selected_index] - except (ValueError, IndexError) as exc: - raise typer.BadParameter("Invalid model selection.") from exc + choices = [_model_choice(model) for model in models] + if allow_empty: + choices.append(Choice(empty_label, value=NO_MODEL)) + selected = _select_prompt(title, choices=choices, default=models[0].key) + if selected == NO_MODEL: + return None + return next(model for model in models if model.key == selected) def _choose_image_model( @@ -757,7 +876,12 @@ def _choose_image_model( chat_models: list[LMStudioModel], ) -> LMStudioModel | None: if image_models: - return _choose_model("Available image description models", image_models, allow_empty=True) + return _choose_model( + "Choose an image description model", + image_models, + allow_empty=True, + empty_label="Disable image indexing", + ) console.print("[yellow]No vision model was detected in LM Studio.[/yellow]") console.print( @@ -768,14 +892,23 @@ def _choose_image_model( console.print("Image indexing will be disabled for now.") return None - use_chat_list = typer.confirm( - "Choose from available chat models anyway", - default=False, + action = _select_prompt( + "Image indexing", + choices=[ + Choice("Disable image indexing for now", value="disable"), + Choice("Choose from available chat models", value="choose"), + ], + default="disable", ) - if not use_chat_list: + if action == "disable": console.print("Image indexing will be disabled for now.") return None - return _choose_model("Available chat models", chat_models, allow_empty=True) + return _choose_model( + "Choose an image description model", + chat_models, + allow_empty=True, + empty_label="Disable image indexing", + ) def _choose_model_key( @@ -799,26 +932,75 @@ def _choose_model_key( console.print("Download one manually in LM Studio, then run this command again.") raise typer.Exit(1) - console.print(title + ":") - if current_key: - console.print(f"Current: {current_key}") + choices: list[Choice] = [] + model_keys = {model.key for model in models} + if current_key and current_key not in model_keys: + choices.append(Choice(f"Keep current model ({current_key})", value=current_key)) + choices.extend(_model_choice(model) for model in models) if allow_empty: - console.print(f"0. {empty_label}") - for index, model in enumerate(models, start=1): - loaded = " loaded" if model.is_loaded else "" - console.print(f"{index}. {model.display_name} ({model.key}){loaded}") - - default = "keep" if current_key else "1" - choice = typer.prompt("Choose model", default=default).strip() - if current_key and choice.lower() in {"keep", "k"}: - return current_key - if allow_empty and choice.lower() in {"0", "none", "skip", "search-only"}: + choices.append(Choice(empty_label, value=NO_MODEL)) + + default = current_key if current_key in model_keys else models[0].key + selected = _select_prompt(title, choices=choices, default=default) + if selected == NO_MODEL: return "" + return str(selected) + + +def _model_choice(model: LMStudioModel) -> Choice: + loaded = " [loaded]" if model.is_loaded else "" + return Choice(f"{model.display_name} ({model.key}){loaded}", value=model.key) + + +def _select_prompt( + message: str, + choices: list[Choice], + default: Any = None, +) -> Any: + answer = questionary.select( + message, + choices=choices, + default=default, + style=PROMPT_STYLE, + instruction="(Use arrow keys and press Enter)", + use_indicator=True, + ).ask() + if answer is None: + raise typer.Abort() + return answer + + +def _checkbox_prompt(message: str, choices: list[Choice]) -> list[Any]: + answer = questionary.checkbox( + message, + choices=choices, + style=PROMPT_STYLE, + instruction="(Use arrow keys, Space to select, Enter to continue)", + validate=lambda selected: bool(selected) or "Select at least one option.", + ).ask() + if answer is None: + raise typer.Abort() + return answer + + +def _text_prompt(message: str, default: str = "") -> str: + answer = questionary.text( + message, + default=default, + style=PROMPT_STYLE, + ).ask() + if answer is None: + raise typer.Abort() + return answer.strip() + + +def _validate_api_origin(origin: str) -> str: try: - selected_index = int(choice) - 1 - return models[selected_index].key - except (ValueError, IndexError) as exc: - raise typer.BadParameter("Invalid model selection.") from exc + from openmind.api.cors import validate_cors_origin + + return validate_cors_origin(origin) + except ValueError as exc: + raise typer.BadParameter(str(exc)) from exc def _load_lmstudio_model(client, model_key: str, label: str) -> dict: @@ -941,15 +1123,7 @@ def _choose_sources(current: OpenMindEngine) -> None: Path("~/Desktop").expanduser(), ] available = [path for path in candidates if path.exists() and path.is_dir()] - console.print("Choose folders to index:") - for index, path in enumerate(available, start=1): - console.print(f"{index}. {path}") - console.print(f"{len(available) + 1}. Add custom folder") - raw = typer.prompt("Selected numbers or paths, comma separated", default="1") - selected_paths = _resolve_source_selection(raw, available) - custom_option = len(available) + 1 - if any(part.strip() == str(custom_option) for part in raw.split(",")): - selected_paths.append(Path(typer.prompt("Custom folder")).expanduser()) + selected_paths = _choose_source_paths(available) seen_paths: set[Path] = set() for path in selected_paths: @@ -965,30 +1139,23 @@ def _choose_sources(current: OpenMindEngine) -> None: _print_existing_source_status(current, str(normalized_path)) -def _resolve_source_selection(raw: str, available: list[Path]) -> list[Path]: - selected_paths: list[Path] = [] - custom_option = len(available) + 1 - for part in [value.strip() for value in raw.split(",") if value.strip()]: - if part == str(custom_option): - continue - try: - index = int(part) - except ValueError: - path = Path(part).expanduser() - if not path.exists() or not path.is_dir(): - raise typer.BadParameter( - f"Invalid folder selection: {part}. " - "Enter a listed number or an existing folder path." - ) - selected_paths.append(path) - continue - if 1 <= index <= len(available): - selected_paths.append(available[index - 1]) - continue - raise typer.BadParameter( - f"Invalid folder selection: {part}. " - "Enter a listed number or an existing folder path." +def _choose_source_paths(available: list[Path]) -> list[Path]: + choices = [Choice(str(path), value=str(path)) for path in available] + choices.append( + Choice( + "Add a custom folder...", + value=CUSTOM_FOLDER, ) + ) + selected = _checkbox_prompt("Choose folders to index", choices) + selected_paths = [Path(value) for value in selected if value != CUSTOM_FOLDER] + + if CUSTOM_FOLDER in selected: + raw_path = _text_prompt("Custom folder path") + custom_path = Path(raw_path).expanduser() + if not custom_path.exists() or not custom_path.is_dir(): + raise typer.BadParameter(f"Folder does not exist or is not a directory: {raw_path}") + selected_paths.append(custom_path) return selected_paths diff --git a/openmind/core/engine.py b/openmind/core/engine.py index b24a7c9..e419d3b 100644 --- a/openmind/core/engine.py +++ b/openmind/core/engine.py @@ -73,9 +73,9 @@ def save_config(self, config: OpenMindConfig) -> None: self.answer_provider = self._build_answer_provider() self.extractors = self._build_extractor_registry() - def add_source(self, path: str) -> Source: + def add_source(self, path: str, recursive: bool = True) -> Source: self.init() - return self.sources.add(path) + return self.sources.add(path, recursive=recursive) def list_sources(self) -> list[Source]: self.init() @@ -296,6 +296,21 @@ def ask( show_thinking: bool = False, history: list[dict[str, str]] | None = None, ) -> str: + answer, _ = self.ask_with_sources( + question, + limit=limit, + show_thinking=show_thinking, + history=history, + ) + return answer + + def ask_with_sources( + self, + question: str, + limit: int = 5, + show_thinking: bool = False, + history: list[dict[str, str]] | None = None, + ) -> tuple[str, list[SearchResult]]: self._log("ask.start", "Answering question", question=question, limit=limit) results = self.search(self._conversation_search_query(question, history), limit=limit) answer = self.answer_provider.answer( @@ -305,7 +320,7 @@ def ask( history=history, ) self._log("ask.finish", "Answer finished", question=question, sources=len(results)) - return answer + return answer, results def ask_stream( self, @@ -314,18 +329,42 @@ def ask_stream( show_thinking: bool = False, history: list[dict[str, str]] | None = None, ) -> Iterator[str]: - self._log("ask.start", "Streaming answer", question=question, limit=limit) - results = self.search(self._conversation_search_query(question, history), limit=limit) - if results: - yield _retrieval_preamble(results) - for chunk in self.answer_provider.stream_answer( + stream, _ = self.ask_stream_with_sources( question, - results, + limit=limit, show_thinking=show_thinking, history=history, - ): - yield chunk - self._log("ask.finish", "Streaming answer finished", question=question, sources=len(results)) + ) + yield from stream + + def ask_stream_with_sources( + self, + question: str, + limit: int = 5, + show_thinking: bool = False, + history: list[dict[str, str]] | None = None, + ) -> tuple[Iterator[str], list[SearchResult]]: + self._log("ask.start", "Streaming answer", question=question, limit=limit) + results = self.search(self._conversation_search_query(question, history), limit=limit) + + def stream() -> Iterator[str]: + if results: + yield _retrieval_preamble(results) + for chunk in self.answer_provider.stream_answer( + question, + results, + show_thinking=show_thinking, + history=history, + ): + yield chunk + self._log( + "ask.finish", + "Streaming answer finished", + question=question, + sources=len(results), + ) + + return stream(), results def status(self) -> StatusSummary: self.init() diff --git a/openmind/core/models.py b/openmind/core/models.py index bf64b0c..a8b9c71 100644 --- a/openmind/core/models.py +++ b/openmind/core/models.py @@ -69,6 +69,9 @@ class SearchResult(BaseModel): score: float chunk_index: int metadata: dict[str, Any] = Field(default_factory=dict) + file_id: str = "" + source_id: str = "" + extension: str = "" class IndexSummary(BaseModel): diff --git a/openmind/storage/lance_store.py b/openmind/storage/lance_store.py index 298c079..97a10f3 100644 --- a/openmind/storage/lance_store.py +++ b/openmind/storage/lance_store.py @@ -1,6 +1,7 @@ from __future__ import annotations import json +import re from datetime import UTC, datetime from pathlib import Path from typing import Any @@ -70,8 +71,35 @@ def search(self, vector: list[float], limit: int = 5) -> list[SearchResult]: rows = table.search(vector).limit(limit).to_list() return [self._result_from_row(row) for row in rows] + def count_chunks(self) -> int: + table = self._table_or_none() + return table.count_rows() if table is not None else 0 + + def chunks_for_file(self, file_id: str, limit: int = 1000) -> list[dict[str, Any]]: + if not re.fullmatch(r"file_[0-9a-f]{16}", file_id): + raise ValueError("Invalid file id.") + table = self._table_or_none() + if table is None: + return [] + rows = ( + table.search() + .where(f"file_id = '{file_id}'") + .limit(limit) + .to_list() + ) + return [ + { + "id": str(row["id"]), + "text": str(row.get("text", "")), + "chunk_index": int(row.get("chunk_index", 0)), + "title": str(row.get("title", "")), + "metadata": self._parse_metadata(row.get("metadata")), + } + for row in rows + ] + def _table_or_none(self): - if self.table_name not in self.db.table_names(): + if self.table_name not in self.db.list_tables().tables: return None return self.db.open_table(self.table_name) @@ -80,11 +108,7 @@ def _result_from_row(self, row: dict[str, Any]) -> SearchResult: score = 1.0 / (1.0 + distance) text = str(row.get("text", "")) snippet = text[:280].replace("\n", " ").strip() - metadata_raw = row.get("metadata") or "{}" - try: - metadata = json.loads(metadata_raw) - except json.JSONDecodeError: - metadata = {} + metadata = self._parse_metadata(row.get("metadata")) return SearchResult( id=str(row["id"]), path=str(row["path"]), @@ -95,4 +119,16 @@ def _result_from_row(self, row: dict[str, Any]) -> SearchResult: score=score, chunk_index=int(row["chunk_index"]), metadata=metadata, + file_id=str(row.get("file_id", "")), + source_id=str(row.get("source_id", "")), + extension=str(row.get("extension", "")), ) + + def _parse_metadata(self, metadata_raw: Any) -> dict[str, Any]: + metadata_raw = metadata_raw or "{}" + if isinstance(metadata_raw, dict): + return metadata_raw + try: + return json.loads(str(metadata_raw)) + except (json.JSONDecodeError, TypeError): + return {} diff --git a/openmind/storage/sqlite_store.py b/openmind/storage/sqlite_store.py index 91878df..62dfb05 100644 --- a/openmind/storage/sqlite_store.py +++ b/openmind/storage/sqlite_store.py @@ -151,6 +151,13 @@ def file_by_path(self, path: str) -> FileRecord | None: return None return self._file_from_row(row) + def file_by_id(self, file_id: str) -> FileRecord | None: + with self.connect() as conn: + row = conn.execute("SELECT * FROM files WHERE id = ?", (file_id,)).fetchone() + if row is None: + return None + return self._file_from_row(row) + def upsert_file(self, record: FileRecord) -> None: with self.connect() as conn: conn.execute( diff --git a/pyproject.toml b/pyproject.toml index ab81ba7..9546c58 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "openmind-core" -version = "0.0.4" +version = "0.0.5" description = "Local-first AI memory engine for user-approved folders." readme = "README.md" requires-python = ">=3.11" @@ -27,6 +27,9 @@ classifiers = [ dependencies = [ "typer>=0.12.0", "rich>=13.7.0", + "questionary>=2.0.1,<3", + "fastapi>=0.115,<1", + "uvicorn>=0.30,<1", "lancedb>=0.13.0", "sentence-transformers>=3.0.0", "pydantic>=2.7.0", @@ -41,11 +44,13 @@ dependencies = [ [project.optional-dependencies] dev = [ + "httpx2>=2,<3", "pytest>=8.2.0", ] [dependency-groups] dev = [ + "httpx2>=2,<3", "pytest>=8.2.0", ] diff --git a/tests/test_api.py b/tests/test_api.py new file mode 100644 index 0000000..1c5a291 --- /dev/null +++ b/tests/test_api.py @@ -0,0 +1,408 @@ +from __future__ import annotations + +import os +from pathlib import Path +from types import SimpleNamespace + +import pytest +from fastapi.testclient import TestClient +from typer.testing import CliRunner + +from openmind import __version__ +from openmind.api.app import create_app +from openmind.api.auth import ensure_api_token, rotate_api_token, token_path +from openmind.cli.main import app as cli_app +from openmind.core.config import AppPaths, ModelSettings, OpenMindConfig, ProviderSettings +from openmind.core.models import FileRecord, IndexJob, SearchResult, Source, StatusSummary +from openmind.providers.lmstudio.models import LMStudioModel + +TOKEN = "test-token-abcdefghijklmnopqrstuvwxyz-123456" +FILE_ID = "file_0123456789abcdef" +SOURCE_ID = "src_0123456789ab" +API = "/api/v1" + + +class FakeLanceStore: + def count_chunks(self): + return 3 + + def chunks_for_file(self, file_id): + assert file_id == FILE_ID + return [ + { + "id": "chunk_1", + "text": "Cabin packing notes", + "chunk_index": 0, + "title": "Holiday notes", + "metadata": {"extension": ".md"}, + } + ] + + +class FakeSQLiteStore: + def __init__(self, record): + self.record = record + + def file_by_id(self, file_id): + return self.record if file_id == self.record.id else None + + +class FakeEngine: + def __init__(self, tmp_path: Path): + source_path = tmp_path / "documents" + source_path.mkdir() + file_path = source_path / "holiday.md" + file_path.write_text("Cabin packing notes", encoding="utf-8") + self.paths = AppPaths( + home=tmp_path / ".openmind", + config_path=tmp_path / ".openmind" / "config.toml", + sqlite_path=tmp_path / ".openmind" / "openmind.sqlite", + lancedb_path=tmp_path / ".openmind" / "lancedb", + logs_path=tmp_path / ".openmind" / "logs", + ) + self.config = OpenMindConfig( + provider=ProviderSettings(name="lmstudio"), + models=ModelSettings(chat_model="qwen", embedding_model="nomic"), + ) + self.source = Source( + id=SOURCE_ID, + path=str(source_path), + recursive=True, + enabled=True, + created_at="2026-07-19T10:00:00+00:00", + ) + self.record = FileRecord( + id=FILE_ID, + source_id=SOURCE_ID, + path=str(file_path), + name=file_path.name, + extension=".md", + size=file_path.stat().st_size, + modified_at=file_path.stat().st_mtime, + content_hash="hash", + status="indexed", + indexed_at="2026-07-19T10:05:00+00:00", + ) + self.sqlite = FakeSQLiteStore(self.record) + self.lance = FakeLanceStore() + self.job = None + self.loaded = [] + + def init(self): + self.paths.ensure() + return self.paths + + def status(self): + return StatusSummary( + sources=1, + enabled_sources=1, + files=1, + indexed_files=1, + app_home=str(self.paths.home), + ) + + def index_job_status(self): + return self.job + + def provider_status(self): + return True, "LM Studio is reachable." + + def list_lmstudio_models(self): + return [ + LMStudioModel(type="llm", key="qwen", display_name="Qwen"), + LMStudioModel(type="embedding", key="nomic", display_name="Nomic"), + LMStudioModel( + type="llm", + key="smolvlm", + display_name="SmolVLM", + capabilities={"vision": True}, + ), + ] + + def lmstudio_client(self): + return SimpleNamespace(load_model_if_needed=self._load_model) + + def _load_model(self, key): + self.loaded.append(key) + return {"model": key, "status": "loaded", "skipped": False} + + def load_configured_models(self): + keys = [self.config.models.embedding_model] + if self.config.models.chat_model: + keys.insert(0, self.config.models.chat_model) + if self.config.extraction.images.enabled: + keys.append(self.config.extraction.images.model) + return [self._load_model(key) for key in keys] + + def save_config(self, config): + self.config = config + + def list_sources(self): + return [self.source] + + def add_source(self, path, recursive=True): + return Source( + id="src_abcdef012345", + path=str(Path(path).resolve()), + recursive=recursive, + enabled=True, + created_at="2026-07-19T11:00:00+00:00", + ) + + def remove_source(self, source_id): + return source_id == self.source.id + + def start_index_job(self): + self.job = IndexJob(id="job_123", status="pending") + return self.job + + def pause_index_job(self): + self.job = IndexJob(id="job_123", status="pause_requested") if self.job else None + return self.job + + def resume_index_job(self): + self.job = IndexJob(id="job_123", status="running") if self.job else None + return self.job + + def stop_index_job(self): + self.job = IndexJob(id="job_123", status="stop_requested") if self.job else None + return self.job + + def search(self, query, limit=5): + return [ + SearchResult( + id="chunk_1", + file_id=FILE_ID, + source_id=SOURCE_ID, + path=self.record.path, + file_name=self.record.name, + extension=".md", + title="Holiday notes", + text="Cabin packing notes", + snippet="Cabin packing notes", + score=0.91, + chunk_index=0, + metadata={"extension": ".md"}, + ) + ][:limit] + + def ask_with_sources(self, question, limit=5): + return "Bring a jacket.", self.search(question, limit=limit) + + def ask_stream(self, question, limit=5): + yield "Bring " + yield "a jacket." + + def ask_stream_with_sources(self, question, limit=5): + return self.ask_stream(question, limit=limit), self.search(question, limit=limit) + + +def auth_headers(token=TOKEN): + return {"Authorization": f"Bearer {token}"} + + +def test_health_is_public_but_private_routes_require_token(tmp_path): + app = create_app(engine=FakeEngine(tmp_path), api_token=TOKEN) + with TestClient(app) as client: + health = client.get("/health") + missing = client.get(f"{API}/status") + invalid = client.get(f"{API}/status", headers=auth_headers("wrong-token")) + valid = client.get(f"{API}/status", headers=auth_headers()) + + assert health.status_code == 200 + assert health.json() == {"status": "ok", "version": __version__} + assert missing.status_code == 401 + assert invalid.status_code == 401 + assert invalid.headers["www-authenticate"] == "Bearer" + assert valid.status_code == 200 + assert valid.json()["indexed_chunks"] == 3 + + +def test_openapi_marks_private_routes_as_bearer_authenticated(tmp_path): + app = create_app(engine=FakeEngine(tmp_path), api_token=TOKEN) + with TestClient(app) as client: + schema = client.get("/openapi.json").json() + + assert "security" not in schema["paths"]["/health"]["get"] + assert schema["paths"][f"{API}/status"]["get"]["security"] + assert schema["paths"][f"{API}/search"]["post"]["security"] + assert "OpenMind API token" in schema["components"]["securitySchemes"] + + +def test_models_sources_and_index_controls(tmp_path): + engine = FakeEngine(tmp_path) + app = create_app(engine=engine, api_token=TOKEN) + with TestClient(app) as client: + models = client.get(f"{API}/models", headers=auth_headers()) + selection = client.put( + f"{API}/models/selection", + headers=auth_headers(), + json={ + "chat_model": "qwen", + "embedding_model": "nomic", + "image_model": "smolvlm", + "load": False, + }, + ) + source = client.post( + f"{API}/sources", + headers=auth_headers(), + json={"path": str(tmp_path), "recursive": False}, + ) + started = client.post(f"{API}/index/start", headers=auth_headers()) + paused = client.post(f"{API}/index/pause", headers=auth_headers()) + resumed = client.post(f"{API}/index/resume", headers=auth_headers()) + stopped = client.post(f"{API}/index/stop", headers=auth_headers()) + + assert models.status_code == 200 + assert models.json()["image_models"][0]["key"] == "smolvlm" + assert selection.status_code == 200 + assert source.status_code == 201 + assert source.json()["recursive"] is False + assert started.status_code == 202 + assert paused.json()["state"] == "pause_requested" + assert resumed.json()["state"] == "running" + assert stopped.json()["state"] == "stop_requested" + + +def test_search_ask_stream_document_and_safe_open(tmp_path): + engine = FakeEngine(tmp_path) + opened = [] + app = create_app( + engine=engine, + api_token=TOKEN, + file_opener=lambda path: opened.append(path), + ) + with TestClient(app) as client: + search = client.post( + f"{API}/search", + headers=auth_headers(), + json={"query": "cabin", "limit": 5}, + ) + ask = client.post( + f"{API}/ask", + headers=auth_headers(), + json={"question": "What should I pack?"}, + ) + stream = client.post( + f"{API}/ask/stream", + headers=auth_headers(), + json={"question": "What should I pack?"}, + ) + document = client.get(f"{API}/documents/{FILE_ID}", headers=auth_headers()) + opened_response = client.post( + f"{API}/actions/open", + headers=auth_headers(), + json={"file_id": FILE_ID}, + ) + + assert search.status_code == 200 + assert search.json()["results"][0]["file_id"] == FILE_ID + assert search.json()["results"][0]["source_type"] == "md" + assert ask.json()["answer"] == "Bring a jacket." + assert ask.json()["sources"][0]["path"] == engine.record.path + assert "event: delta" in stream.text + assert "event: sources" in stream.text + assert FILE_ID in stream.text + assert "event: done" in stream.text + assert document.json()["chunks"][0]["text"] == "Cabin packing notes" + assert opened_response.status_code == 200 + assert opened == [Path(engine.record.path).resolve()] + + +def test_open_action_rejects_file_outside_enabled_sources(tmp_path): + engine = FakeEngine(tmp_path) + outside = tmp_path / "outside.md" + outside.write_text("private", encoding="utf-8") + engine.record.path = str(outside) + app = create_app(engine=engine, api_token=TOKEN, file_opener=lambda path: None) + + with TestClient(app) as client: + response = client.post( + f"{API}/actions/open", + headers=auth_headers(), + json={"file_id": FILE_ID}, + ) + + assert response.status_code == 403 + + +def test_api_rejects_empty_queries_extra_fields_and_invalid_file_ids(tmp_path): + app = create_app(engine=FakeEngine(tmp_path), api_token=TOKEN) + with TestClient(app) as client: + empty = client.post(f"{API}/search", headers=auth_headers(), json={"query": " "}) + extra = client.post( + f"{API}/search", + headers=auth_headers(), + json={"query": "cabin", "raw_vectors": True}, + ) + invalid_file = client.post( + f"{API}/actions/open", + headers=auth_headers(), + json={"file_id": "../../etc/passwd"}, + ) + + assert empty.status_code == 422 + assert extra.status_code == 422 + assert invalid_file.status_code == 422 + + +def test_api_token_is_private_and_rotatable(tmp_path): + home = tmp_path / ".openmind" + + first = ensure_api_token(home) + second = ensure_api_token(home) + rotated = rotate_api_token(home) + + assert first == second + assert rotated != first + assert token_path(home).read_text(encoding="utf-8").strip() == rotated + if os.name != "nt": + assert token_path(home).stat().st_mode & 0o777 == 0o600 + + +def test_running_api_picks_up_rotated_token(tmp_path): + engine = FakeEngine(tmp_path) + original = ensure_api_token(engine.paths.home) + app = create_app(engine=engine) + + with TestClient(app) as client: + before = client.get(f"{API}/status", headers=auth_headers(original)) + rotated = rotate_api_token(engine.paths.home) + stale = client.get(f"{API}/status", headers=auth_headers(original)) + current = client.get(f"{API}/status", headers=auth_headers(rotated)) + + assert before.status_code == 200 + assert stale.status_code == 401 + assert current.status_code == 200 + + +def test_app_rejects_wildcard_cors_even_outside_cli(tmp_path): + with pytest.raises(ValueError, match="exact http\\(s\\) origins"): + create_app( + engine=FakeEngine(tmp_path), + api_token=TOKEN, + allowed_origins=["*"], + ) + + +def test_serve_command_binds_to_loopback_and_rejects_wildcard_cors(monkeypatch, tmp_path): + engine = FakeEngine(tmp_path) + uvicorn_calls = [] + monkeypatch.setattr("openmind.cli.main.engine", lambda: engine) + monkeypatch.setattr("uvicorn.run", lambda api, **kwargs: uvicorn_calls.append(kwargs)) + + served = CliRunner().invoke(cli_app, ["serve", "--port", "9876"]) + wildcard = CliRunner().invoke(cli_app, ["serve", "--allow-origin", "*"]) + + assert served.exit_code == 0 + assert uvicorn_calls == [ + { + "host": "127.0.0.1", + "port": 9876, + "log_level": "info", + "access_log": False, + } + ] + assert wildcard.exit_code != 0 + assert "exact http(s) origins" in wildcard.output diff --git a/tests/test_cli_help.py b/tests/test_cli_help.py new file mode 100644 index 0000000..1d801fc --- /dev/null +++ b/tests/test_cli_help.py @@ -0,0 +1,40 @@ +from typer.testing import CliRunner + +from openmind import __version__ +from openmind.cli.main import OPENMIND_BANNER, app + + +def test_version_option_reports_installed_version(): + runner = CliRunner() + + long_result = runner.invoke(app, ["--version"]) + short_result = runner.invoke(app, ["-V"]) + + assert long_result.exit_code == 0 + assert long_result.output.strip() == f"openmind {__version__}" + assert short_result.exit_code == 0 + assert short_result.output.strip() == f"openmind {__version__}" + + +def test_top_level_help_describes_each_command(): + result = CliRunner().invoke(app, ["--help"]) + + assert result.exit_code == 0 + for description in ( + "Initialize OpenMind's local app data.", + "Configure models, sources, and background indexing.", + "Search indexed local memory.", + "Ask grounded questions or start an interactive session.", + "Show OpenMind storage and indexing information.", + "Clear indexed memory without deleting user files.", + "Remove OpenMind local data and optionally the package.", + ): + assert description in result.output + + +def test_setup_banner_is_large_ascii_art(): + lines = OPENMIND_BANNER.splitlines() + + assert len(lines) == 6 + assert max(len(line) for line in lines) >= 45 + assert OPENMIND_BANNER.isascii() diff --git a/tests/test_cli_models.py b/tests/test_cli_models.py index 35e9992..ab69bb3 100644 --- a/tests/test_cli_models.py +++ b/tests/test_cli_models.py @@ -20,8 +20,21 @@ def read(self): return json.dumps(self.payload).encode("utf-8") +def mock_prompt_answers(monkeypatch, *answers): + selected = iter(answers) + monkeypatch.setattr( + "openmind.cli.main._select_prompt", + lambda *args, **kwargs: next(selected), + ) + monkeypatch.setattr( + "openmind.cli.main._text_prompt", + lambda message, default="": default, + ) + + def test_models_update_saves_selected_lmstudio_models(monkeypatch, tmp_path): monkeypatch.setenv("OPENMIND_HOME", str(tmp_path)) + mock_prompt_answers(monkeypatch, "lmstudio", "gemma", "nomic") loaded_models = [] def fake_urlopen(request, timeout): @@ -58,7 +71,7 @@ def fake_urlopen(request, timeout): monkeypatch.setattr("urllib.request.urlopen", fake_urlopen) - result = CliRunner().invoke(app, ["models", "update"], input="1\n\n2\n1\n") + result = CliRunner().invoke(app, ["models", "update"]) assert result.exit_code == 0 config = OpenMindConfig.load(tmp_path / "config.toml") @@ -71,6 +84,7 @@ def fake_urlopen(request, timeout): def test_models_update_saves_selected_image_description_model(monkeypatch, tmp_path): monkeypatch.setenv("OPENMIND_HOME", str(tmp_path)) + mock_prompt_answers(monkeypatch, "lmstudio", "qwen", "nomic", "smolvlm") loaded_models = [] def fake_urlopen(request, timeout): @@ -108,7 +122,7 @@ def fake_urlopen(request, timeout): monkeypatch.setattr("urllib.request.urlopen", fake_urlopen) - result = CliRunner().invoke(app, ["models", "update"], input="1\n\n1\n1\n1\n") + result = CliRunner().invoke(app, ["models", "update"]) assert result.exit_code == 0 config = OpenMindConfig.load(tmp_path / "config.toml") @@ -121,6 +135,7 @@ def fake_urlopen(request, timeout): def test_models_update_can_keep_existing_models_without_loading(monkeypatch, tmp_path): monkeypatch.setenv("OPENMIND_HOME", str(tmp_path)) + mock_prompt_answers(monkeypatch, "lmstudio", "qwen", "nomic") OpenMindConfig( provider=ProviderSettings(name="lmstudio", base_url="http://localhost:1234"), models=ModelSettings(chat_model="qwen", embedding_model="nomic"), @@ -153,7 +168,6 @@ def fake_urlopen(request, timeout): result = CliRunner().invoke( app, ["models", "update", "--no-load"], - input="1\n\n\n\n", ) assert result.exit_code == 0 @@ -164,6 +178,7 @@ def fake_urlopen(request, timeout): def test_models_update_skips_models_that_are_already_loaded(monkeypatch, tmp_path): monkeypatch.setenv("OPENMIND_HOME", str(tmp_path)) + mock_prompt_answers(monkeypatch, "lmstudio", "qwen", "nomic") def fake_urlopen(request, timeout): if request.full_url.endswith("/api/v1/models"): @@ -191,7 +206,7 @@ def fake_urlopen(request, timeout): monkeypatch.setattr("urllib.request.urlopen", fake_urlopen) - result = CliRunner().invoke(app, ["models", "update"], input="1\n\n1\n1\n") + result = CliRunner().invoke(app, ["models", "update"]) assert result.exit_code == 0 assert "already loaded" in result.output diff --git a/tests/test_cli_sources.py b/tests/test_cli_sources.py index e46b55e..869dad6 100644 --- a/tests/test_cli_sources.py +++ b/tests/test_cli_sources.py @@ -1,9 +1,6 @@ from typer.testing import CliRunner -import pytest -import typer - -from openmind.cli.main import _resolve_source_selection, app +from openmind.cli.main import CUSTOM_FOLDER, _choose_source_paths, app from openmind.core.models import FileRecord from openmind.storage.sqlite_store import SQLiteStore @@ -42,22 +39,57 @@ def test_source_add_reports_existing_indexed_source(monkeypatch, tmp_path): assert "already accessible" in second.output -def test_setup_source_selection_accepts_pasted_folder_path(tmp_path): +def test_setup_source_selection_supports_multiple_folders(monkeypatch, tmp_path): docs = tmp_path / "docs" data = tmp_path / "data" docs.mkdir() data.mkdir() + monkeypatch.setattr( + "openmind.cli.main._checkbox_prompt", + lambda message, choices: [str(docs), CUSTOM_FOLDER], + ) + monkeypatch.setattr( + "openmind.cli.main._text_prompt", + lambda message, default="": str(data), + ) - selected = _resolve_source_selection(f"1,{data}", [docs]) + selected = _choose_source_paths([docs]) assert selected == [docs, data] -def test_setup_source_selection_rejects_unknown_text(tmp_path): +def test_setup_custom_source_does_not_preselect_first_folder(monkeypatch, tmp_path): docs = tmp_path / "docs" + custom = tmp_path / "custom" docs.mkdir() + custom.mkdir() + custom_prompted = [] + + def choose_custom(message, choices): + assert all(not choice.checked for choice in choices) + return [CUSTOM_FOLDER] + + def enter_custom_path(message, default=""): + custom_prompted.append(message) + return str(custom) + + monkeypatch.setattr("openmind.cli.main._checkbox_prompt", choose_custom) + monkeypatch.setattr("openmind.cli.main._text_prompt", enter_custom_path) + + selected = _choose_source_paths([docs]) + + assert selected == [custom] + assert custom_prompted == ["Custom folder path"] + + +def test_setup_source_selection_uses_checked_folders(monkeypatch, tmp_path): + docs = tmp_path / "docs" + docs.mkdir() + monkeypatch.setattr( + "openmind.cli.main._checkbox_prompt", + lambda message, choices: [str(docs)], + ) - with pytest.raises(typer.BadParameter) as exc: - _resolve_source_selection("not-a-folder", [docs]) + selected = _choose_source_paths([docs]) - assert "Enter a listed number or an existing folder path" in str(exc.value) + assert selected == [docs] diff --git a/tests/test_lance_store.py b/tests/test_lance_store.py new file mode 100644 index 0000000..5e1dc0b --- /dev/null +++ b/tests/test_lance_store.py @@ -0,0 +1,36 @@ +from openmind.core.models import Chunk +from openmind.storage.lance_store import LanceStore + + +def test_lance_store_counts_and_returns_sanitized_file_chunks(tmp_path): + store = LanceStore(tmp_path / "lancedb") + chunk = Chunk( + id="chunk_1", + document_id="doc_1", + source_id="src_0123456789ab", + file_id="file_0123456789abcdef", + path="/docs/holiday.md", + file_name="holiday.md", + extension=".md", + title="Holiday", + text="Cabin packing notes", + chunk_index=0, + content_hash="hash", + modified_at=1.0, + metadata={"extension": ".md"}, + ) + + store.add_chunks([chunk], [[0.1, 0.2, 0.3]]) + chunks = store.chunks_for_file(chunk.file_id) + + assert store.count_chunks() == 1 + assert chunks == [ + { + "id": "chunk_1", + "text": "Cabin packing notes", + "chunk_index": 0, + "title": "Holiday", + "metadata": {"extension": ".md"}, + } + ] + assert "vector" not in chunks[0] diff --git a/uv.lock b/uv.lock index fdcad71..70f7247 100644 --- a/uv.lock +++ b/uv.lock @@ -167,6 +167,22 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/02/c3/253a89ee03fc9b9682f1541728eb66db7db22148cd94f89ab22528cd1e1b/deprecation-2.1.0-py2.py3-none-any.whl", hash = "sha256:a10811591210e1fb0e768a8c25517cabeabcba6f0bf96564f8ff45189f90b14a", size = 11178, upload-time = "2020-04-20T14:23:36.581Z" }, ] +[[package]] +name = "fastapi" +version = "0.139.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "annotated-doc" }, + { name = "pydantic" }, + { name = "starlette" }, + { name = "typing-extensions" }, + { name = "typing-inspection" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/cd/95/d3f0ae10836324a2eab98a52b61210ac609f08200bf4bb0dc8132d32f78a/fastapi-0.139.2.tar.gz", hash = "sha256:333145a6891e9b5b3cfceb69baf817e8240cde4d4588ae5a10bf56ffacb6255e", size = 423428, upload-time = "2026-07-16T15:06:17.912Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5f/c7/cb03251d9dfb177246a9809a76f189d21df32dbd4a845951881d11323b7f/fastapi-0.139.2-py3-none-any.whl", hash = "sha256:b9ad015a835173d59865e2f5d8296fbc2b317bf56a2ba1a5bfbdd03de2fd4b1c", size = 130234, upload-time = "2026-07-16T15:06:19.557Z" }, +] + [[package]] name = "filelock" version = "3.29.5" @@ -247,6 +263,19 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/7e/f5/f66802a942d491edb555dd61e3a9961140fd64c90bce1eafd741609d334d/httpcore-1.0.9-py3-none-any.whl", hash = "sha256:2d400746a40668fc9dec9810239072b40b4484b640a8c38fd654a024c7a1bf55", size = 78784, upload-time = "2025-04-24T22:06:20.566Z" }, ] +[[package]] +name = "httpcore2" +version = "2.7.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "h11" }, + { name = "truststore" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/d5/fe/6a3f9f1a8bb8733326140737446aaf72fddb8b54b8f202302f5c84960613/httpcore2-2.7.0.tar.gz", hash = "sha256:6dc0fedf329a52a990930a5579edfebaea81118ea700ea0dd7de2b5e5be49efc", size = 65593, upload-time = "2026-07-14T20:40:01.111Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/6f/6c/62e2e279e63fc4f7a5ee841ef13175a8bbc613f258e9dcc186e9de803a42/httpcore2-2.7.0-py3-none-any.whl", hash = "sha256:1452f589fe23f55b44546cd884294c41a29330af902bc0b71a761fd52d18f92b", size = 81506, upload-time = "2026-07-14T20:39:58.053Z" }, +] + [[package]] name = "httpx" version = "0.28.1" @@ -262,6 +291,22 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/2a/39/e50c7c3a983047577ee07d2a9e53faf5a69493943ec3f6a384bdc792deb2/httpx-0.28.1-py3-none-any.whl", hash = "sha256:d909fcccc110f8c7faf814ca82a9a4d816bc5a6dbfea25d6591d6985b8ba59ad", size = 73517, upload-time = "2024-12-06T15:37:21.509Z" }, ] +[[package]] +name = "httpx2" +version = "2.7.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, + { name = "httpcore2" }, + { name = "idna" }, + { name = "truststore" }, + { name = "typing-extensions", marker = "python_full_version < '3.13'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a3/4a/129b2e21b90ac2985d3928d96792bccc39bc6dfe796c5eee2d8ec06d4105/httpx2-2.7.0.tar.gz", hash = "sha256:8b30709aed5c8465b0dd3b95c09ce301c8f79e7e7a2d00ab0af551e0d0375b07", size = 94487, upload-time = "2026-07-14T20:40:02.318Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1d/b8/c341bba6411bdfda786020343c47a75ef472f6085caf82391b142b1a3ad9/httpx2-2.7.0-py3-none-any.whl", hash = "sha256:ed2a2719c696789e09493bd8e2bec3d8bd925cc6e26b68389ec25ade132f7bf4", size = 90234, upload-time = "2026-07-14T20:39:59.531Z" }, +] + [[package]] name = "huggingface-hub" version = "1.22.0" @@ -949,10 +994,11 @@ wheels = [ [[package]] name = "openmind-core" -version = "0.0.4" +version = "0.0.5" source = { editable = "." } dependencies = [ { name = "beautifulsoup4" }, + { name = "fastapi" }, { name = "lancedb" }, { name = "pandas" }, { name = "pillow" }, @@ -960,25 +1006,31 @@ dependencies = [ { name = "pypdf" }, { name = "pypdfium2" }, { name = "python-docx" }, + { name = "questionary" }, { name = "rapidocr-onnxruntime" }, { name = "rich" }, { name = "sentence-transformers" }, { name = "typer" }, + { name = "uvicorn" }, ] [package.optional-dependencies] dev = [ + { name = "httpx2" }, { name = "pytest" }, ] [package.dev-dependencies] dev = [ + { name = "httpx2" }, { name = "pytest" }, ] [package.metadata] requires-dist = [ { name = "beautifulsoup4", specifier = ">=4.12.0" }, + { name = "fastapi", specifier = ">=0.115,<1" }, + { name = "httpx2", marker = "extra == 'dev'", specifier = ">=2,<3" }, { name = "lancedb", specifier = ">=0.13.0" }, { name = "pandas", specifier = ">=2.2.0" }, { name = "pillow", specifier = ">=12.3.0" }, @@ -987,15 +1039,20 @@ requires-dist = [ { name = "pypdfium2", specifier = ">=5.11.0" }, { name = "pytest", marker = "extra == 'dev'", specifier = ">=8.2.0" }, { name = "python-docx", specifier = ">=1.1.0" }, + { name = "questionary", specifier = ">=2.0.1,<3" }, { name = "rapidocr-onnxruntime", specifier = ">=1.4.4" }, { name = "rich", specifier = ">=13.7.0" }, { name = "sentence-transformers", specifier = ">=3.0.0" }, { name = "typer", specifier = ">=0.12.0" }, + { name = "uvicorn", specifier = ">=0.30,<1" }, ] provides-extras = ["dev"] [package.metadata.requires-dev] -dev = [{ name = "pytest", specifier = ">=8.2.0" }] +dev = [ + { name = "httpx2", specifier = ">=2,<3" }, + { name = "pytest", specifier = ">=8.2.0" }, +] [[package]] name = "overrides" @@ -1170,6 +1227,18 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538, upload-time = "2025-05-15T12:30:06.134Z" }, ] +[[package]] +name = "prompt-toolkit" +version = "3.0.52" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "wcwidth" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a1/96/06e01a7b38dce6fe1db213e061a4602dd6032a8a97ef6c1a862537732421/prompt_toolkit-3.0.52.tar.gz", hash = "sha256:28cde192929c8e7321de85de1ddbe736f1375148b02f2e17edd840042b1be855", size = 434198, upload-time = "2025-08-27T15:24:02.057Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/84/03/0d3ce49e2505ae70cf43bc5bb3033955d2fc9f932163e84dc0779cc47f48/prompt_toolkit-3.0.52-py3-none-any.whl", hash = "sha256:9aac639a3bbd33284347de5ad8d68ecc044b91a762dc39b7c21095fcd6a19955", size = 391431, upload-time = "2025-08-27T15:23:59.498Z" }, +] + [[package]] name = "protobuf" version = "7.35.1" @@ -1532,6 +1601,18 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/f1/12/de94a39c2ef588c7e6455cfbe7343d3b2dc9d6b6b2f40c4c6565744c873d/pyyaml-6.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:ebc55a14a21cb14062aa4162f906cd962b28e2e9ea38f9b4391244cd8de4ae0b", size = 149341, upload-time = "2025-09-25T21:32:56.828Z" }, ] +[[package]] +name = "questionary" +version = "2.1.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "prompt-toolkit" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/f6/45/eafb0bba0f9988f6a2520f9ca2df2c82ddfa8d67c95d6625452e97b204a5/questionary-2.1.1.tar.gz", hash = "sha256:3d7e980292bb0107abaa79c68dd3eee3c561b83a0f89ae482860b181c8bd412d", size = 25845, upload-time = "2025-08-28T19:00:20.851Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3c/26/1062c7ec1b053db9e499b4d2d5bc231743201b74051c973dadeac80a8f43/questionary-2.1.1-py3-none-any.whl", hash = "sha256:a51af13f345f1cdea62347589fbb6df3b290306ab8930713bfae4d475a7d4a59", size = 36753, upload-time = "2025-08-28T19:00:19.56Z" }, +] + [[package]] name = "rapidocr-onnxruntime" version = "1.4.4" @@ -1992,6 +2073,19 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/5e/f5/0c41cb68dcae6b7de4fac4188a3a9589e21fb31df21ea3a2e888db95e6c9/soupsieve-2.8.4-py3-none-any.whl", hash = "sha256:e7e6b0769c8f51ed59acab6e994b00621096cfb1c640a7509295987388fbaf65", size = 37304, upload-time = "2026-05-24T13:55:55.406Z" }, ] +[[package]] +name = "starlette" +version = "1.3.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, + { name = "typing-extensions", marker = "python_full_version < '3.13'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/eb/e3/7c1dc7381d9f8ab7d854328ebfa884e62cb3f3d8549ddfd37c7814f42afa/starlette-1.3.1.tar.gz", hash = "sha256:05d0213193f2fbaae60e2ecb593b4add4262ad4e46536b54abe36f11a71724e0", size = 2703240, upload-time = "2026-06-12T09:23:11.602Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ec/bb/2799cc2ede3ed41131f8975621e7213dfc7ef4acbbaadfa440f32500c370/starlette-1.3.1-py3-none-any.whl", hash = "sha256:c7372aae11c3c3f26a42df7bd626cec2f47d03483d261d369516a615a53714c6", size = 73632, upload-time = "2026-06-12T09:23:10.017Z" }, +] + [[package]] name = "sympy" version = "1.14.0" @@ -2133,6 +2227,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/f0/ac/229b7d4589d2e5937310e72c6d46e89599d16a4a12b479ffa1499fee8eb8/triton-3.7.1-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:10ba85fa2cca4a2fbdeb36bf1cb082f2c252bda55bf9fccd74f65ec5bc647e68", size = 197824404, upload-time = "2026-06-17T19:53:42.772Z" }, ] +[[package]] +name = "truststore" +version = "0.10.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/53/a3/1585216310e344e8102c22482f6060c7a6ea0322b63e026372e6dcefcfd6/truststore-0.10.4.tar.gz", hash = "sha256:9d91bd436463ad5e4ee4aba766628dd6cd7010cf3e2461756b3303710eebc301", size = 26169, upload-time = "2025-08-12T18:49:02.73Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/19/97/56608b2249fe206a67cd573bc93cd9896e1efb9e98bce9c163bcdc704b88/truststore-0.10.4-py3-none-any.whl", hash = "sha256:adaeaecf1cbb5f4de3b1959b42d41f6fab57b2b1666adb59e89cb0b53361d981", size = 18660, upload-time = "2025-08-12T18:49:01.46Z" }, +] + [[package]] name = "typer" version = "0.26.8" @@ -2186,3 +2289,25 @@ sdist = { url = "https://files.pythonhosted.org/packages/53/0c/06f8b233b8fd13b9e wheels = [ { url = "https://files.pythonhosted.org/packages/7f/3e/5db95bcf282c52709639744ca2a8b149baccf648e39c8cc87553df9eae0c/urllib3-2.7.0-py3-none-any.whl", hash = "sha256:9fb4c81ebbb1ce9531cce37674bbc6f1360472bc18ca9a553ede278ef7276897", size = 131087, upload-time = "2026-05-07T16:13:17.151Z" }, ] + +[[package]] +name = "uvicorn" +version = "0.51.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "click" }, + { name = "h11" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a2/65/b7c6c443ccc58678c91e1e973bbe2a878591538655d6e1d47f24ba1c51f3/uvicorn-0.51.0.tar.gz", hash = "sha256:f6f4b69b657c312f516dd2d268ab9ae6f254b11e4bac504f37b2ab58b24dd0b0", size = 94412, upload-time = "2026-07-08T10:59:05.962Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/45/ec/dbb7e5a6b91f86bfb9eb7d2988a2730907b6a729875b949c7f022e8b88fa/uvicorn-0.51.0-py3-none-any.whl", hash = "sha256:5d38af6cd620f2ae3849fb44fd4879e0890aa1febe8d47eb355fb45d93fe6a5b", size = 73219, upload-time = "2026-07-08T10:59:04.44Z" }, +] + +[[package]] +name = "wcwidth" +version = "0.8.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/34/74/c6428f875774288bec1396f5bfcbc2d925700a4dad61727fd5f2b12f249d/wcwidth-0.8.2.tar.gz", hash = "sha256:91fbef97204b96a3d4d421609b80340b760cf33e26da123ff243d76b1fda8dda", size = 1466253, upload-time = "2026-06-29T18:11:11.601Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/96/42/3e5985a0a7e57de470b320c6d6a1a67c844f6737a587f3d44dd13d1819e7/wcwidth-0.8.2-py3-none-any.whl", hash = "sha256:d63947694a0539a1d51e01eda7caf800c291020e6cdd7e28ad7b14dd33ad4f85", size = 323166, upload-time = "2026-06-29T18:11:09.888Z" }, +]