diff --git a/docs/architecture.mdx b/docs/architecture.mdx index 16ad140b..dcb7f768 100644 --- a/docs/architecture.mdx +++ b/docs/architecture.mdx @@ -21,18 +21,22 @@ flowchart TB EngineA1["Engine
Ollama"] EngineA2["Engine
LM Studio"] EngineB1["Engine
Ollama"] + EngineB2["Engine
llama.cpp"] ModelA1["Models
present on this engine"] ModelA2["Models"] ModelB1["Models"] + ModelB2["Models"] Cluster --- NodeA Cluster --- NodeB NodeA --- EngineA1 NodeA --- EngineA2 NodeB --- EngineB1 + NodeB --- EngineB2 EngineA1 --- ModelA1 EngineA2 --- ModelA2 EngineB1 --- ModelB1 + EngineB2 --- ModelB2 ``` - A **cluster** is a set of nodes that have paired with each other. There is no @@ -48,7 +52,7 @@ flowchart TB the two nodes interchangeable for a request. PAIR evaluates eligibility at the bottom of that hierarchy. A request names a -model, so it can only be served by a node whose running engine holds it. Cluster +model, so only a node whose running engine holds that model can serve it. Cluster membership makes a node reachable. It does not make it capable. ## Process Architecture @@ -83,10 +87,10 @@ attempts bounded restart with backoff. Workers log to stderr, and stdout is reserved for newline-delimited JSON-RPC when stdio is used. The terminal interface is a separate broker client for headless use, and is what -the `nvpair` command on PATH resolves to. Do not run it and the desktop -application on the same machine at the same time, because each starts its own -broker and worker tree and the two would then contend for the same local ports, -engines, and persisted settings. +the `nvpair` command on `PATH` resolves to. Do not run it and the desktop +application on the same machine at the same time. Each starts its own broker and +worker tree, and the two would then contend for the same local ports, engines, +and persisted settings. ### Health and Failure Detection @@ -94,26 +98,26 @@ There is no `/health` endpoint on any PAIR service, and the broker does not heartbeat its workers. What exists is uneven, so it is worth being precise about the layers that are checked: -| Layer | How it is checked | +| Layer | How it is Checked | | --- | --- | | Engines | A readiness probe on start, then a periodic health probe. Status carries a `healthy` flag, and a failed probe raises a service error. | | Workers | Process liveness only. An exit is detected and restarted. A hang is not detected. | | The broker | A `ping` request clients can call, which the terminal interface's **Service** tab uses. | | Peer nodes | Reachability, not health: missed announcements, then recent inference activity, recent telemetry, or a probe of the node-info or engine-manager port before eviction. The inference proxies report activity but are never probed. | -Worker supervision is restart-based. An unexpected exit is retried with -exponential backoff from roughly one second to 16, with a budget of five attempts -for each unhealthy streak and a 60-second window after which a worker that -stayed up is considered recovered and its budget resets. A crash raises a sticky +Worker supervision is restart-based. The broker retries an unexpected exit with +exponential backoff from roughly one second to 16 seconds, with a budget of five +attempts for each unhealthy streak. A worker that stays up for a 60-second window +is considered recovered and its budget resets. A crash raises a sticky error entry so the interface can show that a component is down. Recovery clears it, and a worker that exhausts its budget is left down with the entry standing. `nvpair-errors` is the exception that cannot report its own death, so its crashes go to stderr instead. The gap this leaves is a worker that is running but wedged. It is still holding its -stdio pipe, but no longer answering. Nothing detects that today. It is detected when -requests or by a UI state that quietly stops updating. To recover, restart the service from -**Settings > Service**. +stdio pipe but no longer answering. Nothing detects that today. It shows up instead +as jobs that never complete or UI state that quietly stops updating. To recover, +restart the service from **Settings > Service**. ## Control Flow @@ -155,11 +159,11 @@ Most peer traffic runs over *mutual Transport Layer Security (TLS)*. Each node holds a stable universally unique identifier (UUID) and a self-signed leaf certificate, and pairing pins each side's certificate against the other's UUID. A cluster-gated endpoint verifies that its caller presents a certificate pinned for -a current member and refuses anything else, so a machine that can reach the port -but has not been paired cannot read model inventory, read workloads, or drive a -remote engine. +a current member and refuses anything else. A machine that can reach the port but +has not been paired therefore cannot read model inventory, read workloads, or +drive a remote engine. -Certificates and pinning are owned by `nvpair-cluster-manager`. Refer to +`nvpair-cluster-manager` owns certificates and pinning. Refer to [SECURITY.md](../SECURITY.md) for the trust model and its limits. Not every surface is gated, and the exceptions are deliberate: @@ -198,11 +202,11 @@ distinguished by the connection's first byte. A TLS handshake record starts with This is why an endpoint is `http://127.0.0.1:11434` for an application on the machine itself while the same port serves authenticated TLS to the network. Local clients need no certificates. Routed inference between paired nodes uses mutual -TLS; explicitly added unpaired engines may use plaintext HTTP. +TLS. Explicitly added unpaired engines can use plaintext HTTP. The loopback restriction is enforced, not merely conventional. The listener binds -all interfaces so the TLS personality can accept peers, but a plaintext request -from any non-loopback address is refused with `403`. Without that check the port +all interfaces so the TLS personality can accept peers, but the proxy refuses a +plaintext request from any non-loopback address with `403`. Without that check the port would be an open relay for anything on the network. The cluster manager uses the same protocol split, but its plaintext endpoint @@ -220,8 +224,9 @@ reach PAIR: cannot chain a request onward through a third node. The second point is why the routing decision belongs to the machine the request -originates on. To give a workstation the whole cluster, make it a node. After it is a node, its local -proxy then routes on its behalf, whether or not it runs an engine itself. +originates on. To give a workstation the whole cluster, make it a node. After it +is a node, its local proxy routes on its behalf, whether or not it runs an engine +itself. ### What Peers Exchange @@ -237,9 +242,9 @@ shows: Each of these listens on its own port. Refer to the [port map](#port-map) below. Electron polls every discovered node's `/v1/node-info` directly because that -detail is richer than the broker's control surface exposes. Healthy nodes are -sampled every two seconds; consecutive failures back off to 4, 8, 16, then 30 -seconds and reset on success. This is the one place Electron makes service-data +detail is richer than the broker's control surface exposes. Electron samples +healthy nodes every two seconds. Consecutive failures back off to 4, 8, 16, then +30 seconds and reset on success. This is the one place Electron makes service-data HTTP calls of its own. Separately, the backend scanner samples a compact maximum-GPU utilization value for scheduling with the same healthy cadence and capped failure backoff, which the broker relays straight to the job scheduler. @@ -284,7 +289,7 @@ The proxy owns the per-request decision, and it does not produce a single winner It produces an **ordered failover list** of nodes eligible for that request, then walks it. -For model-bearing inference, eligibility is a hard gate: the proxy takes a +For model-bearing inference, eligibility is a hard gate. The proxy takes a request-local copy of the latest discovery snapshot and keeps only nodes whose per-engine inventory advertises the requested model. Within that owner set, or across all candidates when the request has no parsed model, the order is: @@ -350,7 +355,7 @@ two different mechanisms, and they compose in one direction: the requested model, then applies the scheduler's ordering to the remaining owners. -So load balancing happens **among the nodes that can serve the request**, not +Load balancing therefore happens **among the nodes that can serve the request**, not across the cluster as a whole. #### Capability Gate @@ -358,18 +363,19 @@ across the cluster as a whole. When an inference request contains a non-empty model, only nodes whose current inventory for that engine advertises the model enter the failover list. An empty inventory and an inventory that lists other models are both ineligible. Ollama's -implicit `:latest` tag is normalized; LM Studio model IDs match exactly. +implicit `:latest` tag is normalized. LM Studio and llama.cpp model IDs match +exactly. If no advertised owner is routable, the proxy returns an actionable local `502` without sending the request to an engine. It does not broaden the candidate list -or refresh inventory synchronously; a later discovery update makes a newly +or refresh inventory synchronously. A later discovery update makes a newly advertised owner eligible. A request whose model cannot be parsed keeps the ordinary non-model ordering. -Model listings are not routed at all. A `GET` of `/v1/models`, Ollama's -`/api/tags`, or llama.cpp's `/models` is fanned out to every candidate for that -engine concurrently and the replies are merged, which is why the answer is the +Model listings are not routed at all. The proxy fans a `GET` of `/v1/models`, +Ollama's `/api/tags`, or llama.cpp's `/models` out to every candidate for that +engine concurrently and merges the replies. That is why the answer is the cluster's inventory for that engine rather than one node's. #### Failover and Inventory Freshness @@ -422,9 +428,9 @@ running one enormous request looks more idle than a node running two trivial one **Model load state is not considered.** Eligibility asks whether a node *has* the model, not whether it is already loaded in memory. PAIR knows which models are -loaded, and the interface shows it, but routing does not use it, so a request can -be sent to a node that must cold-load the model while a node holding it warm sits -one place lower in the order. +loaded, and the interface shows it, but routing does not use it. A request can +therefore go to a node that must cold-load the model while a node holding it warm +sits one place lower in the order. **Only work PAIR routed contributes to pending counts.** Inference sent straight to an engine's own port is absent from workload events. GPU-heavy external work @@ -439,7 +445,7 @@ occupy different devices. schedule. Two nodes dispatching at the same moment can briefly steer work to the same idle peer, because each sees an eventually consistent picture and a peer's counts trail by relay latency. Per-proxy reservations cover the local burst case, -but they are local: one node cannot see what another just dispatched. The system +but they are local. One node cannot see what another just dispatched. The system self-corrects through workload relay and periodic reconciliation rather than preventing the collision. @@ -447,7 +453,7 @@ preventing the collision. ranked yet is ordered by node ID, which has nothing to do with load. That happens at cold start, or for a manual proxy target the scheduler's discovery never saw. The fallback degrades ranking quality without causing a misroute. Missing -telemetry for a ranked node is different: that node takes the neutral pressure +telemetry for a ranked node is different. That node takes the neutral pressure value and stays in the normal ranking. Improving this, and likely offering a choice of policies, is planned work. Which @@ -490,9 +496,9 @@ own application and let PAIR start it on a port nothing else is serving. An application that already works with Ollama is configured for `11434`. If PAIR listened somewhere else, every tool would need reconfiguring to gain anything, so -PAIR inverts it: the **proxy** takes the port the engine would normally use, and -the engine moves behind it — Ollama to `11435` and upwards, LM Studio to `1235` -and upwards. The managed llama.cpp router stays on `8081` while its facade takes +PAIR inverts it. The **proxy** takes the port the engine would normally use, and +the engine moves behind it. Ollama moves to `11435` and upwards, and LM Studio +moves to `1235` and upwards. The managed llama.cpp router stays on `8081` while its facade takes `8080`. Existing clients keep working untouched and transparently gain the cluster. This is also what makes the engine unreachable from outside. Engines PAIR starts @@ -521,7 +527,8 @@ The rearrangement is conditional, and PAIR yields rather than fights: The practical consequence is the one in [Troubleshooting](troubleshooting.mdx#requests-work-but-pair-shows-no-jobs). Start the Ollama desktop application and it takes `11434` for itself, so PAIR cannot, -and requests reach that local Ollama without ever being routed. +and requests reach that local Ollama without ever being routed. A `llama-server` +you start yourself on `8080` does the same to the llama.cpp facade. ### Port Map @@ -563,8 +570,8 @@ each host, and that record carries two kinds of content: - The ports its sibling services registered: node-info, engine proxies, errors, workloads, cluster manager, and engine manager - The node's identity: `uuid=`, `cluster-uuid=` after clustering, and where to - reach it — `ip=` for the address the node ranks first, and `ips=` for the whole - ranked list + reach it, with `ip=` for the address the node ranks first and `ips=` for the + whole ranked list Publishing the list rather than one address is what lets a peer keep trying. A machine can be reachable on one interface and not another, so a peer works @@ -605,7 +612,7 @@ egress on multi-homed Windows hosts. Those sockets set `SO_REUSEADDR` so they coexist with Bonjour, Avahi, and other PAIR processes. A Darwin sender also sets `SO_REUSEPORT`, matching the BSD multicast sharing behavior needed to coexist with the system mDNS responder. -Linux deliberately omits `SO_REUSEPORT`: there it can load-balance incoming +Linux deliberately omits `SO_REUSEPORT`. On Linux it can load-balance incoming unicast datagrams into a short-lived send socket and steal replies from the long-lived receiver. @@ -627,9 +634,9 @@ distinguish a paired node from a stranger it can merely see. ### Node Eviction -The scanner scans every five seconds. Once a node has missed three scans in a -row, about fifteen seconds, the scanner checks on it at every scan, and any answer -resets the count. It drops the node only after twelve consecutive misses, about a +The scanner scans every five seconds. After a node has missed three scans in a +row, about 15 seconds, the scanner checks on it at every scan, and any answer +resets the count. It drops the node only after 12 consecutive misses, about a minute, with every check along the way failing. These are the checks: - Inference the node served in the last minute, which the proxies report @@ -701,11 +708,12 @@ The important boundaries are: Optional socket or named-pipe mode relies on operating-system endpoint permissions. JSON-RPC has no independent per-message token. 3. **Loopback HTTP.** Local clients can submit sensitive inference content. - Listener addresses, browser access, CORS, and host account security matter. - The proxies add no permissive CORS default: they pass the upstream engine's - headers and its refusals through, so a web page reaches a model through PAIR - only where the engine would have allowed it directly, and a cluster-wide - request needs every engine that answered to allow the origin. Refer to + Listener addresses, browser access, cross-origin resource sharing (CORS), and + host account security matter. The proxies add no permissive CORS default. They + pass the upstream engine's headers and its refusals through, so a web page + reaches a model through PAIR only where the engine would have allowed it + directly. A cluster-wide request needs every engine that answered to allow the + origin. Refer to [Engine settings](engine-settings.mdx) for allowing an origin. 4. **LAN discovery and metadata.** The network can reveal service presence and selected host information. Some enrichment endpoints use plain HTTP. @@ -736,7 +744,7 @@ stale or hand-placed binary is caught rather than silently supervised. `services/build.sh` and `services/build.bat` stage every executable together, stamped from `versions.json`. Building one component by hand without restaging is -the one thing to avoid: the broker keeps running whatever is in `build/bin/`. +the one thing to avoid. The broker keeps running whatever is in `build/bin/`. Both paths produce binaries you run on the machine that built them. This tree provides build materials only. There is no packaging, installer, or signing step diff --git a/docs/building.mdx b/docs/building.mdx index 9d84e182..463ab229 100644 --- a/docs/building.mdx +++ b/docs/building.mdx @@ -211,7 +211,7 @@ Set-Location services\build\bin '{"jsonrpc":"2.0","id":1,"method":"ping"}' | .\nvpair-ui-broker.exe ``` -You may see an `app:ready` notification alongside the response, and discovery may +You can see an `app:ready` notification alongside the response, and discovery can be empty while LAN browsing starts. Set `NVPAIR_LOG_LEVEL=debug` or pass `--log-level debug` for launch diagnostics. The accepted levels are `debug`, `info`, `warn`, and `error`. @@ -239,7 +239,7 @@ opening a pull request are in ## Cleaning Up a Build from Source -A source build has nothing to uninstall — deleting the checkout removes the +A source build has nothing to uninstall. Deleting the checkout removes the application. What it leaves behind is the same per-user data any install creates, so reset that the same way. Alongside the in-app **Settings > Service > Reset app data**, a script does the same job without the @@ -254,7 +254,7 @@ On Windows use `scripts\wipe-app-data.cmd` with the same flags. Neither script needs Node, and both exclude engine model libraries such as `~/.ollama`, `~/.lmstudio/models`, and `~/.llamacpp` by design, so a reset does not cost you re-downloading models. -Start with `--dry-run`; it prints the exact paths and deletes nothing. +Start with `--dry-run`. It prints the exact paths and deletes nothing. If the machine belongs to a cluster, leave the cluster as well, or the other nodes keep listing it as a member. Refer to diff --git a/docs/developing.mdx b/docs/developing.mdx index 2ce807c3..4d612cab 100644 --- a/docs/developing.mdx +++ b/docs/developing.mdx @@ -34,10 +34,10 @@ cryptography in TypeScript, the change belongs in a Go service instead. ### `desktop/` -| Path | What it owns | +| Path | What it Owns | | --- | --- | | `src/ui/` | The React renderer: components, stores, and the renderer-side API | -| `src/ui/stores/` | One store per domain — nodes, engines, models, workloads, errors, and metrics | +| `src/ui/stores/` | One store for each domain: nodes, engines, models, workloads, errors, and metrics | | `src/ui/api/pair-api.ts` | The renderer's view of the service surface | | `src/preload/` | The typed bridge exposing `window.pairApi` and `window.windowApi` | | `src/electron/` | Main process: broker supervision, service bridge, and interprocess communication (IPC) | @@ -54,10 +54,10 @@ The files you touch most often: | `src/shared/types/ws-channels.ts` | Every logical service channel: `WsInvokeChannelMap` for requests, `WsPushChannelMap` for pushes | | `src/shared/types/ipc-channels.ts` | `IpcChannelMap`, the contract for Electron-native IPC | | `src/electron/service-bridge/empty-handlers.ts` | `handleServiceBridgeInvoke`, the single dispatch every service call goes through | -| `src/electron/service-bridge/modular-state.ts` | `ModularBridgeState` — backend notifications become renderer push events here | +| `src/electron/service-bridge/modular-state.ts` | `ModularBridgeState`, where backend notifications become renderer push events | | `src/electron/service-bridge/modular-supervisor.ts` | The broker process and its JSON-RPC link | | `src/shared/constants/modular-binaries.ts` | The canonical inventory of shipped binaries and who launches them | -| `src/shared/constants/modular-runtime.ts` | Backend-coupled constants — the only place a port or path literal belongs | +| `src/shared/constants/modular-runtime.ts` | Backend-coupled constants and the only place a port or path literal belongs | ### `services/` diff --git a/docs/engine-lifecycle.mdx b/docs/engine-lifecycle.mdx index 1a2c705c..a188620a 100644 --- a/docs/engine-lifecycle.mdx +++ b/docs/engine-lifecycle.mdx @@ -9,7 +9,8 @@ An **engine** is the local inference runtime Personal AI Router (PAIR) uses to run models. Today that means Ollama, LM Studio, or llama.cpp on a given machine. PAIR can install and run those engines for you, or work with a supported existing Ollama or LM Studio installation. This page explains what you can -expect when you install, start, stop, update, or remove an engine. +expect when you install, start, stop, or remove an engine, and how to move one +to a newer release. For first-time setup, refer to [Getting started](getting-started.mdx). To change an engine's ports or launch command, refer to @@ -83,18 +84,18 @@ When you install an engine, consider the following: because upstream builds it without Metal. The engine download, roughly 0.3–0.7 GiB for the CUDA archives and about 11 MiB on macOS, happens on demand and is not part of the PAIR installer. Package choice is not - driver-aware; llama.cpp retains CPU fallback when GPU acceleration is + driver-aware. llama.cpp retains CPU fallback when GPU acceleration is unavailable. PAIR-managed llama.cpp models sleep after five minutes without inference work. This releases model and KV-cache memory, and the next request wakes the model -automatically. The llama.cpp worker remains running and may retain some GPU +automatically. The llama.cpp worker remains running and can retain some GPU memory. To download models: -1. Use **Add model**. -2. Use **Load** when the engine requires an explicit load. +1. Select **Add model**. +2. Select **Load** when the engine requires an explicit load. ![The Install engines dialog with Ollama downloading, reporting progress as it installs.](assets/onboarding/engine-lifecycle/01-engine-installing.png) @@ -127,23 +128,23 @@ an engine PAIR installed. Use it when you no longer want PAIR to own that engine on the machine. Uninstalling an engine does **not** remove its models. Downloaded model files stay -on disk in the engine's own storage — `~/.ollama`, `~/.llamacpp`, or -`~/.lmstudio/models` — so uninstalling and reinstalling does not cost you +on disk in the engine's own storage (`~/.ollama`, `~/.llamacpp`, or +`~/.lmstudio/models`), so uninstalling and reinstalling does not cost you re-downloading them. The same is true of removing PAIR itself, or of resetting its data. Both remove the engines PAIR installed as far as they can, and never -your models: a reset stops and tells you if an engine cannot be removed, while +your models. A reset stops and tells you if an engine cannot be removed, while uninstalling PAIR leaves that engine in place and finishes. To reclaim that space, use **Delete** on individual models, or remove the engine's storage directory yourself. PAIR keeps the model folder at the engine's default location. If you moved LM Studio's models folder somewhere else inside `~/.lmstudio`, removing an LM -Studio that PAIR installed — by uninstalling it, or by removing PAIR's data — +Studio that PAIR installed (by uninstalling it, or by removing PAIR's data) deletes that folder with the rest of `~/.lmstudio`. A models folder outside `~/.lmstudio`, or one linked back to `~/.lmstudio/models`, is kept. PAIR also leaves alone any engine you installed yourself. **Uninstall** acts only -on a copy PAIR installed; remove the others with the engine's own uninstaller. +on a copy PAIR installed. Remove the others with the engine's own uninstaller. Port changes for an NVPAIR-installed engine also live in **Engine settings**. Prefer the controls in PAIR over editing the engine's own config when PAIR is @@ -151,7 +152,7 @@ managing it. If PAIR **adopted** an engine that was already running, meaning it did not start the process, what it can do depends on how that engine is controlled: -- **Ollama** is managed as a process, so PAIR will not move an adopted one. It +- **Ollama** is managed as a process, so PAIR does not move an adopted one. It refuses the port change rather than killing something it did not start. Stop the engine in its own application first if you want PAIR to manage it fully. - **LM Studio** publishes an official stop command, so PAIR can stop an adopted diff --git a/docs/engine-settings.mdx b/docs/engine-settings.mdx index 62a62611..c2d0fb9e 100644 --- a/docs/engine-settings.mdx +++ b/docs/engine-settings.mdx @@ -6,33 +6,33 @@ SPDX-License-Identifier: Apache-2.0 # Engine Settings in NVIDIA Personal AI Router Open a device's Ollama, LM Studio, or llama.cpp row and expand **Settings**. -Edit the server port, proxy port and engine arguments, then select the green -**Apply** button. A paired device that supports settings can be edited from -another device in the cluster. +Edit the server port, proxy port, and engine arguments, then select **Apply**. +A paired device that supports settings can be edited from another device in the +cluster. -PAIR fixes the executable, lifecycle commands and loopback address. Unsupported +PAIR fixes the executable, lifecycle commands, and loopback address. Unsupported vendor options can fail at startup. -For installing, starting and updating engines, refer to +For installing, starting, and uninstalling engines, refer to [Managing engines](engine-lifecycle.mdx). -## Argument notation +## Argument Notation The text contains engine arguments and optional leading environment assignments. PAIR supplies the executable and startup subcommand. Leave the text empty to use managed defaults. Tokens are passed directly to the engine, using the same notation on every operating system. -- Spaces, tabs and line breaks separate unquoted tokens. +- Spaces, tabs, and line breaks separate unquoted tokens. - Single quotes enclose literal text. Double quotes enclose literal text with - two escapes, `\"` and `\\`; any other backslash stays literal, so a Windows + two escapes, `\"` and `\\`. Any other backslash stays literal, so a Windows path such as `C:\new\tools` needs no doubling. - A quoted and an unquoted part can form one token, as in `--name="two words"`. An empty quoted string is an empty argument. -- Nothing is expanded: `$HOME`, `%USERPROFILE%`, `{port}` and `~` are literal, - and there are no comments, pipelines, redirects or command substitution. - Unquoted `|`, `&`, `;`, `<`, `>`, a backtick or `$(` is rejected — quote them - when the engine really wants that character. +- Nothing is expanded: `$HOME`, `%USERPROFILE%`, `{port}`, and `~` are literal, + and there are no comments, pipelines, redirects, or command substitution. + Unquoted `|`, `&`, `;`, `<`, `>`, a backtick, or `$(` is rejected. Quote them + when the engine needs that character. - At most 16 KiB and 256 tokens, valid UTF-8. Environment assignments go first, followed by engine arguments: @@ -42,7 +42,7 @@ OLLAMA_ORIGINS="https://example.com" --port 1235 --bind 127.0.0.1 ``` -## Environment variables +## Environment Variables Environment assignments are saved and passed literally to the engine on every launch. PAIR does not maintain a list of supported variables or validate their @@ -50,7 +50,7 @@ engine-specific values. Editing or removing one takes effect at the next launch. Managed CORS settings, including LM Studio's `--cors` switch, can only be changed on the device running the engine. A paired device can preserve an existing CORS -policy while changing other settings, but cannot enable, disable or replace it. +policy while changing other settings, but cannot enable, disable, or replace it. Other environment assignments can also be edited from a paired device. Origin lists need a scheme and a specific host, so bare `localhost` and `*` are @@ -58,7 +58,7 @@ rejected, including wildcards hidden inside extra quotes. Wildcard subdomains such as `https://*.example.com` and ports such as `http://localhost:*` are supported. Managed llama.cpp starts with an empty origin list, so cross-origin browser access is disabled until specific origins are added locally. List only the -origins you want to allow. This is how you give a web page access to your models: +origins you want to allow. This is how you give a web page access to your models. PAIR's local endpoints add no permissive default of their own, so a browser reaches a model through PAIR only where the engine would have allowed it directly. For a request that spans the cluster, every engine that answers has to allow the @@ -68,50 +68,52 @@ An engine does not need a CORS launch option to work with PAIR. The proxy follow the engine's HTTP responses regardless of how its CORS policy is configured. If the engine grants no CORS permission, PAIR does not add permission for it. -Startup failures include a bounded excerpt of the engine’s own output. PAIR -redacts supplied values by exact match, but an engine may transform or encode +Startup failures include a bounded excerpt of the engine's own output. PAIR +redacts supplied values by exact match, but an engine can transform or encode them. Do not put secrets in launch values. -## Ports and validation +## Ports and Validation PAIR understands only the networking controls needed to keep engine and proxy configuration consistent: -| Engine | Server port | Loopback bind | Browser access | +| Engine | Server Port | Loopback Bind | Browser Access | | --- | --- | --- | --- | | Ollama | `OLLAMA_HOST=127.0.0.1:` | The host in `OLLAMA_HOST` | `OLLAMA_ORIGINS` | | LM Studio | `--port`, `-p` | `--bind`, `LMS_SERVER_HOST` | `--cors` | | llama.cpp | `--port` | `--host` | `--cors-origins` | -For LM Studio, `--port 1235`, `--port=1235`, `-p 1235`, `-p1235` and `-p=1235` +For LM Studio, `--port 1235`, `--port=1235`, `-p 1235`, `-p1235`, and `-p=1235` all synchronize the same server-port field. Conflicting repetitions and invalid -ports are rejected. Write managed short options separately; bundles containing -them are ambiguous and rejected. CORS switches do not take `=true` or `=false`: -add or remove the switch locally. Origin lists are normalized before validation; -conflicting repeated lists are rejected. Other vendor options remain opaque. -llama.cpp must remain in router mode; model-selection arguments such as `-m` +ports are rejected. Write managed short options separately. Bundles containing +them are ambiguous and rejected. CORS switches do not take `=true` or `=false`. +Add or remove the switch locally. Origin lists are normalized before validation, +and conflicting repeated lists are rejected. Other vendor options remain opaque. +llama.cpp must remain in router mode. Model-selection arguments such as `-m` or `-hf` switch it to an incompatible single-model server and fail readiness. Changing the server-port field updates the port in the command. Editing the command's port updates the numeric field. If both are edited before validation finishes, the numeric field takes precedence. The server and proxy must use -different ports; a collision opens an error dialog explaining which port +different ports. A collision opens an error dialog explaining which port collides. Invalid settings and failed saves also open an error dialog. -Apply validates the complete change. Confirmation is requested only when a running engine -needs to restart. Progress appears as a spinner inside Apply. A running engine -restarts for launch/server changes; changing only the proxy port leaves the -engine running. Stopped engines remain stopped. Saved settings survive restart. +**Apply** validates the complete change. PAIR asks for confirmation only when a +running engine needs to restart. Progress appears as a spinner inside **Apply**. +A running engine restarts for launch and server changes. Changing only the proxy +port leaves the engine running. Stopped engines remain stopped. Saved settings +survive restart. -## Concurrent edits +## Concurrent Edits Changes from another device update a clean editor. An unfinished draft stays -intact; applying a stale draft opens an error dialog. **Reload saved settings** deliberately -discards that draft. Failed starts retain the saved settings for correction and -show one error dialog. Validation on leaving a field quietly synchronizes ports; -errors appear only when Apply is selected, without duplicate text below the command. - -Offline, unpaired or older devices cannot be edited. A process started outside -PAIR may require stopping it in its own application before changing its launch. +intact, and applying a stale draft opens an error dialog. **Reload saved +settings** deliberately discards that draft. Failed starts retain the saved +settings for correction and show one error dialog. Validation on leaving a field +quietly synchronizes ports. Errors appear only when you select **Apply**, without +duplicate text below the command. + +Offline, unpaired, or older devices cannot be edited. A process started outside +PAIR can require stopping it in its own application before changing its launch. The displayed arguments describe a future PAIR launch, not an external process's original arguments. diff --git a/docs/getting-started.mdx b/docs/getting-started.mdx index e13ce788..437920a3 100644 --- a/docs/getting-started.mdx +++ b/docs/getting-started.mdx @@ -5,7 +5,7 @@ SPDX-License-Identifier: Apache-2.0 # Getting Started with NVIDIA Personal AI Router -NVIDIA Personal AI Router (PAIR), shown as **NVIDIA PAIR** once installed, +NVIDIA Personal AI Router (PAIR), shown as **NVIDIA PAIR** after installation, connects compatible systems on the same local network and routes independent inference requests to a node that can serve them. Install PAIR on each participating system, pair the systems, enable an inference @@ -14,12 +14,12 @@ engine, and then point compatible applications at PAIR's local endpoint. > PAIR routes each request to one node. It does not pool GPU memory, combine > GPUs into a larger logical GPU, or split one model or request across systems. -If a step does not go as described stop and check -[Troubleshooting](troubleshooting.mdx). For example, nothing starts, no nodes appear, pairing -stalls, or an application cannot connect. It is organized by symptom, so you can go -straight to the one you are seeing. +If a step does not go as described, stop and refer to +[Troubleshooting](troubleshooting.mdx). Common examples are that nothing starts, +no nodes appear, pairing stalls, or an application cannot connect. Troubleshooting +is organized by symptom, so you can go straight to the one you are seeing. -> **Setting PAIR up on a system with no desktop?** The desktop application needs +> **On a system with no desktop, use the terminal interface.** The desktop application needs > a graphical session, so a headless system is driven from PAIR's terminal > interface instead. Follow > [Using the PAIR terminal interface](terminal-interface.mdx), which covers @@ -29,7 +29,7 @@ straight to the one you are seeing. ## Before You Begin All you need is a machine to install PAIR on. One machine is enough to run local -inference; two or more on the same local network let you try pairing and routing. +inference. Two or more on the same local network let you try pairing and routing. Nothing else has to be in place first: @@ -67,7 +67,7 @@ Choose from: - **macOS disk image:** Open the `.dmg` and drag **NVIDIA PAIR** to **Applications**, then launch it from there. - When updating from `PAIR.app` to `NVIDIA PAIR.app`, **Restart & install** may + When updating from `PAIR.app` to `NVIDIA PAIR.app`, **Restart & install** might finish installing the update without reopening the application. If PAIR does not reopen, open **NVIDIA PAIR** from your Applications folder. This applies only to the update that renames the app. @@ -77,7 +77,7 @@ terminal interface for each platform. It contains no desktop application, so it is the headless option rather than another way to install this one. Refer to [Using the PAIR terminal interface](terminal-interface.mdx). -### Quick Start From Source +### Quick Start from Source If you prefer to compile PAIR, install the prerequisites listed under [Prerequisites](building.mdx#prerequisites), then: @@ -93,7 +93,7 @@ npm start desktop assets, and launches the desktop application. No separate services build is required. -Install or start PAIR on every system that will participate. Refer to +Install or start PAIR on every system that participates. Refer to [Building and running PAIR from source](building.mdx) for standalone service builds and for running the services directly. @@ -168,17 +168,17 @@ A node is eligible for a request only when it is online, a compatible engine is running, and the requested model is available there. To test routing across multiple nodes, prepare the same model on each of those nodes. -llama.cpp downloads use exact `owner/repository:quantization` IDs. Its Add model +llama.cpp downloads use exact `owner/repository:quantization` IDs. Its **Add model** list is populated from popular GGUF repositories published by `ggml-org`, -`bartowski`, and `unsloth`. Typing filters those results locally; pressing -Enter or the search button searches public Hugging Face repositories. PAIR -offers one verified `Q4_K_M` option per result. Load, Eject, and Delete use that -same exact model id; Delete removes the selected entry from llama.cpp's -persistent cache. +`bartowski`, and `unsloth`. Typing filters those results locally. Pressing +**Enter** or selecting the search button searches public Hugging Face +repositories. PAIR offers one verified `Q4_K_M` option for each result. +**Load**, **Eject**, and **Delete** use that same exact model ID. **Delete** +removes the selected entry from llama.cpp's persistent cache. Refer to [Managing engines](engine-lifecycle.mdx) for details about: -- Installing, starting, stopping, updating, and uninstalling engines. +- Installing, starting, stopping, and uninstalling engines. - Managing engines in a cluster. - Quitting or relaunching PAIR. @@ -189,7 +189,7 @@ inference engine or the node that serves the request. PAIR calls that local address an endpoint. Select **Endpoints** in the top toolbar to open the **API endpoints** window. It -lists one copyable `http://127.0.0.1:` URL per engine. Copy the one for the +lists one copyable `http://127.0.0.1:` URL for each engine. Copy the one for the engine you prepared. If it says **No engines are running**, return to step 4 and start an engine. @@ -213,7 +213,7 @@ Tools already pointed at those ports keep working without reconfiguration, and the engine moves to the next free port. Your application connects to PAIR's proxy, not directly to the engine. A port -from a standalone engine installation may now lead somewhere else. Copy the URL +from a standalone engine installation can now lead somewhere else. Copy the URL from **Endpoints** instead of typing a port from memory. [Step 7](#7-connecting-your-agents-and-port-information) lists the defaults and how to change them. @@ -245,7 +245,7 @@ only when something you already have is written against Ollama's `/api/...` API. The choice does not affect routing. Both styles are routed across your cluster the same way, and neither one changes which node serves the request. -### OpenAI-Style Request (Works With Every Engine) +### OpenAI-Style Request (Works with Every Engine) ```bash curl /v1/chat/completions \ @@ -283,7 +283,7 @@ curl -N /api/chat \ Then confirm the request was routed: open **Overview**, use the **Jobs** filter in the left column, and read **Running on** or **Ran on** on the job card to identify -the serving node. Once a job has been sent to a node, its card shows **Sent to** +the serving node. After a job has been sent to a node, its card shows **Sent to** until the job starts, and keeps it if the job ends without starting. ![A job card under the Jobs filter naming the node that served the request.](assets/onboarding/getting-started/09-job-served-by-node.png) @@ -293,7 +293,7 @@ always runs on one selected node. PAIR does not split an in-flight request between systems. **If you would rather not assemble requests yourself,** PAIR can send them for -you: **Test** on **Settings → Service** runs a burst of inference through the +you: **Test** on **Settings > Service** runs a burst of inference through the same path and shows the resulting jobs. It is the quickest way to see routing work across several nodes at once. Refer to [Generate Traffic Without an Application](#generate-traffic-without-an-application). @@ -305,7 +305,7 @@ proxy takes the port the engine would normally use, so existing clients keep working without reconfiguration, and PAIR moves the engine itself to the next free port. -| What an application connects to | Default port | +| What an Application Connects To | Default Port | | --- | --- | | Ollama-compatible proxy | `11434` | | LM Studio / OpenAI-compatible proxy | `1234` | @@ -373,7 +373,7 @@ current URL. ### Using an Engine's Own Command Line An engine PAIR installed for you is a normal installation, and you can drive it -with its own command-line interface (CLI). Two things make that less obvious: +with its own CLI. Two things make that less obvious: - The binaries are not on your `PATH` yet. - The engine is not on the port its CLI expects by default. @@ -438,15 +438,15 @@ add an alias yourself. ## Use PAIR with Existing Applications -Any client that lets you set a base URL and a model name can use PAIR — Hermes, -for example. Point the base URL at the endpoint you copied in step 5, and use a +Any client that lets you set a base URL and a model name can use PAIR, for +example Hermes. Point the base URL at the endpoint you copied in step 5, and use a model you prepared in step 4. What that base URL serves depends on the engine behind it: -| Endpoint | Default base URL | Paths it serves | +| Endpoint | Default Base URL | Paths it Serves | | --- | --- | --- | -| Ollama | `http://127.0.0.1:11434` | Ollama's own API — `/api/chat`, `/api/generate`, `/api/embed`, `/api/tags` — plus the OpenAI-compatible and Anthropic APIs listed below| +| Ollama | `http://127.0.0.1:11434` | Ollama's own API (`/api/chat`, `/api/generate`, `/api/embed`, and `/api/tags`), plus the OpenAI-compatible and Anthropic APIs listed below | | LM Studio | `http://127.0.0.1:1234` | The OpenAI-compatible and Anthropic APIs listed below | | llama.cpp | `http://127.0.0.1:8080` | The OpenAI-compatible API listed below, plus `/models` for model listings | @@ -459,7 +459,7 @@ All three endpoints expose these OpenAI-compatible paths: - `GET /v1/models` — [available models](https://developers.openai.com/api/reference/resources/models/methods/list) Ollama and LM Studio support the following Anthropic APIs: -- `POST /v1/messages` - [Messages](https://platform.claude.com/docs/en/api/messages/create) +- `POST /v1/messages` — [Messages](https://platform.claude.com/docs/en/api/messages/create) The distinction matters when you fill in an application's settings. A client written against the OpenAI API usually wants the `/v1` included, as in @@ -497,7 +497,7 @@ request and does not forward it onward. **If you need a network-reachable inference endpoint,** that is outside what PAIR does. You would configure an engine to listen on your network yourself -and take on the exposure that implies. PAIR does not offer it by default and there +and take on the exposure that implies. PAIR does not offer it by default, and there is no plan to add it as an option, because it would turn any node into an open relay for anything on the network. @@ -508,9 +508,9 @@ thing that fails tells you where the problem is. ### 1. Check Service Health -Open **Settings → Service** and confirm that **Status** reads **Running**. That +Open **Settings > Service** and confirm that **Status** reads **Running**. That checks the background services without involving inference at all. If it reads -anything else, nothing below will work. Refer to +anything else, nothing below works. Refer to [PAIR does not become ready](troubleshooting.mdx#pair-does-not-become-ready). ![Settings > Service with Status reading Running, beside the Test, Stop, and Restart controls and the log options.](assets/onboarding/getting-started/11-service-health.png) @@ -541,7 +541,7 @@ A reply alone does not prove routing. To identify the serving node: 1. Open **Overview**. 2. Use the **Jobs** filter in the left column. -3. Read **Running on** or **Ran on** on the job card. Once a job has been sent +3. Read **Running on** or **Ran on** on the job card. After a job has been sent to a node, its card shows **Sent to** until the job starts. ### 5. Verify Cluster Routing @@ -560,7 +560,7 @@ to observe how PAIR routes them across eligible nodes. ### Generate Traffic Without an Application If you would rather watch routing work than assemble requests yourself, select -**Test** on **Settings → Service**. It runs a sixty-second burst of synthetic +**Test** on **Settings > Service**. It runs a 60-second burst of synthetic inference and takes you to **Overview**, where it appears as ordinary job activity you can filter and inspect like any other. A toast tracks the run and is where you stop it early. @@ -578,7 +578,7 @@ want to observe. ### Read a Failure -| Symptom | Usual meaning | +| Symptom | Usual Meaning | | --- | --- | | Connection refused | Nothing is listening on that port. Check the port and service status. | | `502` with `no active node` | The proxy is running, but no routable engine node is available. | @@ -588,18 +588,18 @@ want to observe. | `400` or `422` | The request is malformed. PAIR does not retry it because it would fail on every node. | When a status is not specific enough, the services' own reporting is behind -**Open log file** on **Settings → Service**. +**Open log file** on **Settings > Service**. ## Keeping PAIR Up to Date An installed PAIR looks for a newer release shortly after it starts, and every -six hours while it keeps running. Checking is all it does on its own: PAIR never +six hours while it keeps running. Checking is all it does on its own. PAIR never downloads or installs an update without you asking it to. What it offers is a shorter path than fetching an installer yourself, not an automatic upgrade. When a release is available, **Overview** shows an **Update available** message, -and **Settings → Service → Application updates** offers **Download update** -beside the version you are running. Once the download finishes, that card offers +and **Settings > Service > Application updates** offers **Download update** +beside the version you are running. After the download finishes, that card offers **Restart & install**, which stops PAIR's background services before handing over to the installer. **Check for updates** on the same card checks whenever you want, rather than waiting for the next scheduled check. @@ -611,8 +611,9 @@ downloading again. Updating keeps your settings, logs, cluster identity, and cluster membership, so a node stays paired across an update. Model weights are untouched as well, because they belong to the engine rather than to PAIR. Updating PAIR does not -update its inference engines — engines are updated separately from **Engine -settings**, described in [Managing engines](engine-lifecycle.mdx). +update its inference engines. To move an +engine to a newer release, refer to +[Updating and Uninstalling an Engine](engine-lifecycle.mdx#updating-and-uninstalling-an-engine). If you are updating from a version before 1.0.0, a few setups have a small extra step, such as tidying up the old app on macOS. The diff --git a/docs/inference-dispatcher.mdx b/docs/inference-dispatcher.mdx index cd377576..ff8a2086 100644 --- a/docs/inference-dispatcher.mdx +++ b/docs/inference-dispatcher.mdx @@ -28,15 +28,15 @@ The executable uses only the Go standard library. .\scripts\inference-dispatcher.ps1 --backend lmstudio --count 5 --mode parallel ``` -Both wrappers require `go` on `PATH` — the same toolchain `services/build.sh` +Both wrappers require `go` on `PATH`, the same toolchain `services/build.sh` already needs. Node is not required. They compile the module to a temporary directory and run it from the caller's working directory, so a relative `--result-log` path means what you expect. A prebuilt copy ships beside the other binaries for the Inference Demo, which -both the desktop app and the terminal interface spawn once per request; see -[architecture.mdx](architecture.mdx). The wrappers exist so the same tool can be -driven by hand. +both the desktop app and the terminal interface spawn once for each request. +Refer to [Architecture](architecture.mdx). The wrappers exist so the same tool +can be driven by hand. ## Model Selection @@ -71,7 +71,7 @@ non-model defaults. This sends four generated prompts, two at a time. To send prompts of your own, list them in `prompts`, for example `"prompts": ["Explain unified memory in two sentences."]`. One request is sent -per listed prompt, and `count` is ignored. +for each listed prompt, and `count` is ignored. Pass the file with `--config path/to/config.json` or `INFERENCE_DISPATCHER_CONFIG`. Every scalar option also has an @@ -105,12 +105,13 @@ compact tab-separated error trail. ## What is Never Recorded -Neither log, nor stdout, ever contains prompt text or response bodies. Prompts +Neither log nor stdout ever contains prompt text or response bodies. Prompts are identified by a `sha256: len=` digest and responses by their byte count and their own digest, which is enough to correlate and compare jobs without putting inference content on disk. An upstream failure is recorded as -its HTTP status plus the size and digest of the body, never the body text — a -non-2xx body from an OpenAI-compatible endpoint can echo the request back. +its HTTP status plus the size and digest of the body, never the body text, +because a non-2xx body from an OpenAI-compatible endpoint can echo the request +back. This is a hard rule for every Personal AI Router component, not a setting. There is no flag that turns it off. diff --git a/docs/known-issues.mdx b/docs/known-issues.mdx index 21e37675..9dd4a04d 100644 --- a/docs/known-issues.mdx +++ b/docs/known-issues.mdx @@ -3,7 +3,7 @@ SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All SPDX-License-Identifier: Apache-2.0 */} -# Known issues +# Known Issues This page lists significant limitations, with what to expect and what to do about them. This is not a bug tracker. Report anything not listed here as an @@ -17,17 +17,17 @@ depends on the platform: | System | What PAIR Reports | | --- | --- | -| Grace-Blackwell, including DGX Spark | **Correct.** PAIR reports total and used system memory as the GPU's memory, which is what unified memory means on that hardware. | +| Grace Blackwell, including DGX Spark | **Correct.** PAIR reports total and used system memory as the GPU's memory, which is what unified memory means on that hardware. | | Windows with an integrated or unified-memory GPU | **Understated.** PAIR counts only memory dedicated to the GPU, so a GPU that draws on system RAM can appear to have almost none. | | Linux without an NVIDIA driver (AMD or Intel graphics) | **Missing.** PAIR lists the GPU by name with no memory figure and no utilization. | -PAIR handles the Grace-Blackwell case deliberately. The driver reports no GPU +PAIR handles the Grace Blackwell case deliberately. The driver reports no GPU memory total on that architecture, so PAIR substitutes system memory instead. The other two cases are the gap. **The impact is display, not routing.** These numbers appear on a node's card -and in the GPU chart. Routing does not consider VRAM at all — the scheduler -weighs queued work and GPU utilization, never memory — so a wrong figure does +and in the GPU chart. Routing does not consider VRAM at all (the scheduler +weighs queued work and GPU utilization, never memory), so a wrong figure does not send work to the wrong place. It misleads *you* when you decide which node should hold a large model, so on those systems judge capacity from the machine's own tools rather than from PAIR. @@ -58,7 +58,7 @@ still running and no longer responding. The symptom is state that quietly stops updating, such as nodes that never refresh or jobs that never complete. Restarting from **Settings > Service** resolves it. -## macOS Can Stop Answering LAN Connections While Unclustered +## macOS Can Stop Answering LAN Connections while Unclustered On macOS, PAIR can stop answering connections from other machines on the network after it has been left unclustered for a while. Other nodes stop being able to diff --git a/docs/log-collection.mdx b/docs/log-collection.mdx index 0b30a3cb..9ac037f9 100644 --- a/docs/log-collection.mdx +++ b/docs/log-collection.mdx @@ -3,7 +3,7 @@ SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All SPDX-License-Identifier: Apache-2.0 */} -# Collecting and sanitizing logs +# Collecting and Sanitizing Logs `scripts/collect-logs.sh` (and its Windows twin `scripts/collect-logs.ps1`) reads Personal AI Router logs and replaces the identifiers in them, so you can @@ -31,7 +31,7 @@ It does not require Node.js, so it runs on a machine that only builds the servic > [Designing the gathering step](#designing-the-gathering-step) at the end of this > document. -## What It Reads, and What It Writes +## What It Reads and What It Writes The script reads only the log itself: `nvpair.jsonl`, its rotated `nvpair.1.jsonl`, or an exported `nvpair-logs-*.txt`. It touches nothing else in @@ -112,8 +112,8 @@ when it produces a copy. That ordering follows from three things. There is one implementation instead of one for each process. Replacing identifiers while logging would mean the same rules in Go for the 12 services and -again in TypeScript for the desktop logger, because JSON-RPC traffic the Go side -cannot rewrite feeds the desktop log file, which is functional data. Two +again in TypeScript for the desktop logger. JSON-RPC traffic feeds the desktop log +file, and the Go side cannot rewrite that traffic because it is functional data. Two implementations of the same rules drift, and nodes on different app versions would then produce different tokens for the same address. @@ -198,7 +198,7 @@ apply a fix and repeat the run. On status 1 the source logs are untouched, so the fix is to correct detection and run again over the same copies. Keep them until the run succeeds. -## Collecting from More Than One Node +## Collecting from More than One Node Until retrieval exists, staging is manual. Copy each node's log onto one machine and run the sanitizer there, once, over all of them. @@ -258,7 +258,7 @@ snapshot survive into the sanitized file as useful context. **Fallback — raw JSONL.** A bare `nvpair.jsonl` has no header, so the script cannot identify the producer and the output falls back to `source-1.jsonl`, -`source-2.jsonl`. The script still fully sanitizes the logs and tokens still +`source-2.jsonl`, and so on. The script still fully sanitizes the logs and tokens still reconcile. You cannot tell which file came from which machine, which defeats much of the point of gathering them. @@ -280,8 +280,8 @@ Copy each log **exactly as it is on disk**: warns when it cannot learn a host name. Treat that warning as a failed collection. - **Keep rotation order.** `nvpair.1.jsonl` is older than `nvpair.jsonl`, and the - script must read it first. Passing the directory handles this. If listing files - explicitly, preserve that order. + script must read it first. Passing the directory handles this. If you list + files explicitly, preserve that order. - **Never write to the source log.** A node's log must keep accumulating so you can collect the same node again later. @@ -309,7 +309,7 @@ The script trusts host names only when they come from a structured field, either the bundle metadata or an object that also yielded a UUID or an address. No shape test separates a host name from a model name or a JSON-RPC method name, so accepting one from free text alone would corrupt unrelated text. A full exported -bundle always carries those structured fields. A hand-trimmed fragment may not, +bundle always carries those structured fields. A hand-trimmed fragment might not, and the run warns when it cannot learn a host name. The script leaves certificate fingerprints as they are. It recognizes them only diff --git a/docs/overview.mdx b/docs/overview.mdx index 2053efcb..936f0549 100644 --- a/docs/overview.mdx +++ b/docs/overview.mdx @@ -39,7 +39,7 @@ These terms have specific meanings in PAIR: - **Cluster** — the set of nodes you have paired together. A node belongs to at most one cluster, and it must leave before it can join another. - **Engine** — the local inference server that runs models: Ollama, LM Studio, - or llama.cpp. PAIR can install, start, stop, and update these engines; it can + or llama.cpp. PAIR can install, start, stop, and uninstall these engines. It can also adopt supported existing Ollama and LM Studio installations. - **Model** — what you prepare on each node. Nodes do not share models, so a node can serve a request only for a model it already holds. Preparing the same diff --git a/docs/release-notes.mdx b/docs/release-notes.mdx index 1c134cf1..00a878fa 100644 --- a/docs/release-notes.mdx +++ b/docs/release-notes.mdx @@ -5,9 +5,9 @@ SPDX-License-Identifier: Apache-2.0 # Release Notes -What changed in each release of PAIR. If you are updating from an earlier -version, read the [Upgrade Guide](upgrade-guide.mdx) as well: it covers the -steps some machines need. +This page lists what changed in each release of PAIR. If you are updating from +an earlier version, read the [Upgrade Guide](upgrade-guide.mdx) as well. It +covers the steps some machines need. Notes for releases before 1.0.0 are on [GitHub Releases](https://github.com/NVIDIA/Personal-AI-Router/releases). @@ -41,11 +41,13 @@ its public development and contribution workflow on GitHub. ### Compatibility Notes - **Browser clients.** PAIR's local endpoints now follow each engine's CORS - policy instead of allowing every browser origin. A browser-based client may - need the engine configured to allow it — for Ollama, set `OLLAMA_ORIGINS` in - the engine's launch options. + policy instead of allowing every browser origin. A browser-based client might + need the engine configured to allow it. For Ollama, set `OLLAMA_ORIGINS` in + the engine's launch options. For llama.cpp, add the origin to + `--cors-origins`, because managed llama.cpp allows no browser origins until + you add one. - **The macOS app is now NVIDIA PAIR.** The new `NVIDIA PAIR.app` installs - beside the old `PAIR.app`, so remove the old one to keep just one copy. Your + beside the old `PAIR.app`, so remove the old one to keep only one copy. Your settings, cluster membership, and downloaded models carry over. - **LM Studio installed by an earlier PAIR.** It keeps working as before. PAIR now removes only engines it has a record of installing, and earlier versions @@ -60,7 +62,7 @@ The [Upgrade Guide](upgrade-guide.mdx) walks through each of these. - Uninstalling LM Studio from PAIR, or uninstalling PAIR with its data, no longer deletes the models you downloaded. They stay in `~/.lmstudio/models`, - just as Ollama's stay in `~/.ollama` and llama.cpp's in `~/.llamacpp`. + as Ollama's stay in `~/.ollama` and llama.cpp's in `~/.llamacpp`. ### Community Contributions @@ -90,7 +92,7 @@ The [Upgrade Guide](upgrade-guide.mdx) walks through each of these. - Updated the contributor instructions to branch from and rebase onto `develop`. Thanks [@somtri](https://github.com/somtri)! ([#126](https://github.com/NVIDIA/Personal-AI-Router/pull/126)) -- Fixed node cards showing only the first GPU on multi-GPU systems; they now +- Fixed node cards showing only the first GPU on multi-GPU systems. They now show every GPU. Thanks [@CryptoJones](https://github.com/CryptoJones)! ([#34](https://github.com/NVIDIA/Personal-AI-Router/pull/34)) diff --git a/docs/terminal-interface.mdx b/docs/terminal-interface.mdx index ce289bba..35e18afb 100644 --- a/docs/terminal-interface.mdx +++ b/docs/terminal-interface.mdx @@ -3,7 +3,7 @@ SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All SPDX-License-Identifier: Apache-2.0 */} -# Using the PAIR terminal interface +# Using the PAIR Terminal Interface `nvpair-tui` is a full-screen keyboard-driven interface for operating one PAIR host. @@ -36,7 +36,7 @@ An installed PAIR puts `nvpair` on your `PATH` as a small wrapper that runs **Do not count on `nvpair` existing on a headless machine.** Only the Debian package writes the wrapper at install time. On Windows and macOS the desktop application writes it when it starts, so on a machine where you never open the -desktop application — the case this interface exists for — it may never appear. +desktop application (the case this interface exists for), it might never appear. On macOS it is also skipped when `/usr/local/bin` is not writable. Run `nvpair-tui` directly when `nvpair` is not found, and do the same for a source build, which has no wrapper at all. Refer to @@ -64,24 +64,24 @@ cd | --- | --- | | `--broker-path ` | Use a service binary that is not beside `nvpair-tui` | | `--log-level ` | Verbosity of the terminal interface's own logging: `debug`, `info`, `warn`, or `error`. PAIR also reads this from `NVPAIR_LOG_LEVEL` | -| `--appearance ` | `auto`, `light`, or `dark`. See below | +| `--appearance ` | `auto`, `light`, or `dark`. Refer to [If the Colors Are Hard to Read](#if-the-colors-are-hard-to-read) | | `--version` | Print the version and exit | While the interface is running, its own log lines appear on the **Logs** tab beside the service logs, so nothing is written over the display. Before it starts and after it exits, they go to stderr. -### If the Colours Are Hard to Read +### If the Colors Are Hard to Read -PAIR picks its colours from your terminal's background. It detects that by +PAIR picks its colors from your terminal's background. It detects that by asking the terminal and reading the reply, which works in most terminals, -including over SSH — the question and the answer travel the connection like -anything else. +including over SSH, because the question and the answer travel the connection +like anything else. Two cases cannot be detected. Inside `tmux` or `screen` the question is not asked at all, because one session can be attached to several terminals at once and there is no single background to report. A terminal that does not implement -the query simply never answers. Both fall back to assuming a dark background, +the query never answers. Both fall back to assuming a dark background, which on a light theme gives washed-out text. Tell it instead: @@ -101,7 +101,7 @@ with it. Press `q` to quit. Quitting shuts the services down cleanly rather than leaving them running. -### Keep It Running After You Disconnect +### Keep It Running after You Disconnect Quitting stops the services, and losing an SSH connection ends the program the same way. A node you reach over SSH therefore stops serving requests when your @@ -126,11 +126,11 @@ provides. The top line shows `NVPAIR` and the service state on the right: `starting service...`, `service ready v`, or `service disconnected`. -The second line is the numbered tab bar; when anything has gone wrong, the count +The second line is the numbered tab bar. When anything has gone wrong, the count appears on the **Errors** tab's own label. The footer shows the keys available where you are, starting with the ones that always work. When a newer release of PAIR exists, a notice sits between the tab bar and the content on every tab -until you dismiss it — see [Update Notices](#update-notices). +until you dismiss it. Refer to [Update Notices](#update-notices). If you see `starting...`, the interface is waiting for the terminal size. If the header stays on `starting service...`, the service tree did not come up, and the @@ -150,13 +150,13 @@ header stays on `starting service...`, the service tree did not come up, and the Within a table, `j` / `k` (or `↓` / `↑`) move the selection, `f` / `b` page, and `home` / `end` jump to the first and last row. `h` / `l` and `←` / `→` move between -panes rather than between tabs. Where a tab binds one of those letters to its own -verb — on the Nodes list `d` declines a pending pairing request and `f` finds a -machine by address, and on a node's detail screen `d` deletes a model in the -Models pane while `u` uninstalls an engine in the Engines pane — that verb wins -where it applies, and the paging key works elsewhere. The -Service tab's settings list is not a table; it moves with `j` / `k` or the -arrows. On the Nodes list `l` leaves the cluster rather than moving right, so +panes rather than between tabs. Some tabs bind one of those letters to their own +verb. On the Nodes list, `d` declines a pending pairing request and `f` finds a +machine by address. On a node's detail screen, `d` deletes a model in the Models +pane and `u` uninstalls an engine in the Engines pane. Where a verb applies, it +wins, and the paging key works elsewhere. The Service tab's settings list is not +a table. It moves with `j` / `k` or the arrows. On the Nodes list, `l` leaves the +cluster rather than moving right, so it is armed and asks you to confirm. While you are typing into a field, such as a PIN, an address, a port, or a model @@ -167,9 +167,9 @@ keys while a field is open, so it always reflects what actually works. Anything that destroys something asks first. Removing a machine from the cluster, leaving the cluster, deleting a model, uninstalling an engine, and resetting all data each arm on the first press and act only when you then press -`y`; any other key cancels. The prompt names what will happen and stays on +`y`. Any other key cancels. The prompt names what happens next and stays on screen until you answer it. Forgetting a machine you added with `f` does not -ask, because nothing is lost: you can find it again. +ask, because nothing is lost. You can find it again. The terminal needs to be at least 40 by 12. Below that PAIR says so and waits for you to resize rather than drawing something unreadable. Near that minimum @@ -183,11 +183,11 @@ are. | --- | --- | --- | | 1 | **Nodes** | Every machine PAIR knows about, whether reachable, and its cluster standing. `enter` opens one machine's engines, models, and hardware | | 2 | **Jobs** | The endpoints your clients connect to, and inference work as it runs. `t` sends a minute of test traffic | -| 3 | **Service** | Uptime and version, an `ok` / `DOWN` / `?` row per worker, settings, and a data reset | +| 3 | **Service** | Uptime and version, an `ok` / `DOWN` / `?` row for each worker, settings, and a data reset | | 4 | **Errors** | Active service errors by severity, age, node, and message | | 5 | **Logs** | Service log output, with a filter and save-to-file | -The **Errors** tab shows its own count in the tab bar — `Errors (2)` — so you can +The **Errors** tab shows its own count in the tab bar, such as `Errors (2)`, so you can see there is something to look at from any other tab. ## Pair This Machine with Another @@ -208,13 +208,13 @@ Everything below is on the **Nodes** tab (1). This is the easier path, because there is no address to type. Prefer it whenever PAIR has already discovered the machine you want. -The PIN stays on screen until the pairing is answered, then tells you the outcome -— joined, declined, or expired. A machine that is already a member, or that -belongs to another cluster, cannot be invited: PAIR says so instead of sending an +The PIN stays on screen until the pairing is answered, then tells you the outcome: +joined, declined, or expired. A machine that is already a member, or that +belongs to another cluster, cannot be invited. PAIR says so instead of sending an invite that would be refused. **To pair with a machine by address**, press `n`, type the other machine's -host — or `host:port` if it is not on the default pairing port — and press +host (or `host:port` if it is not on the default pairing port), and press `enter`. Use this when discovery has not found the machine, for example on a network that filters multicast. @@ -225,11 +225,11 @@ network that filters multicast. 3. Type the PIN displayed on the other machine and press `enter`. Press `d` instead to decline. After pairing, the peer's `CLUSTER` column reads -`Member`. If the PIN was wrong, the status line says so and you will need a -fresh invite — a PIN can only be used once. +`Member`. If the PIN was wrong, the status line says so, and you need a fresh +invite because a PIN can only be used once. If more than one request arrives, they wait their turn and the prompt says how -many are waiting. A machine can only be in one cluster, so once you accept one, +many are waiting. A machine can only be in one cluster, so after you accept one, the rest are declined and their senders are told. ![The terminal interface Nodes tab with a discovered machine selected and the pairing PIN on the status line.](assets/onboarding/terminal-interface/02-tui-pairing.png) @@ -246,8 +246,8 @@ the rest are declined and their senders are told. | `l` | Leave the cluster (asks you to confirm) | | `/` | Filter the list by name or address; `esc` clears it | -Cancel an invite with `c` if you read the PIN to the wrong person, or simply -change your mind. That invalidates the PIN straight away, so it can no longer be +Cancel an invite with `c` if you read the PIN to the wrong person, or change +your mind. That invalidates the PIN straight away, so it can no longer be used to join, and tells the other machine to drop its prompt. An invite you leave alone expires on its own, but until it does it stays answerable. @@ -290,7 +290,7 @@ still changes that peer's proxy port. A port is a request rather than a promise. If a running engine already holds the port you asked for, PAIR binds elsewhere and tells you which port it ended -up on — so a change you did not get is never reported as one you did. +up on, so a change you did not get is never reported as one you did. ### Startup Arguments @@ -303,23 +303,23 @@ OLLAMA_KEEP_ALIVE=5m OLLAMA_ORIGINS="https://example.com" --flag "two words" Environment assignments come first, then arguments. Quoting works the way it does in a shell for single and double quotes, but nothing is expanded: `$HOME`, -`~` and `%USERPROFILE%` are passed through as text, and pipes, redirects and +`~`, and `%USERPROFILE%` are passed through as text, and pipes, redirects, and command substitution are rejected rather than interpreted. The engine's own executable and its start subcommand are not editable. Press `enter` to check and save. PAIR validates before it saves anything, so a -mistake comes back as a message rather than an engine that will not start. If +mistake comes back as a message rather than an engine that does not start. If the change requires restarting the engine, it says so and waits for `y`. The port fields and the arguments are saved together, so a port written into the arguments and the port in the field cannot drift apart. Editing the field -rewrites the argument; editing the argument moves the field. If the two +rewrites the argument, and editing the argument moves the field. If the two disagree in a way PAIR cannot settle, it names both numbers and asks you to make them agree. This works on a peer as well as on this machine. Whether a particular engine -accepts the change is the engine's own answer: one PAIR did not start — one -that was already running when PAIR found it — reports that it cannot be +accepts the change is the engine's own answer. An engine PAIR did not start (one +that was already running when PAIR found it) reports that it cannot be configured, and says why. **Models** — the models each engine holds, and which are loaded into memory: @@ -364,8 +364,8 @@ machine you were last looking at. A node can serve a request only when it is online, a compatible engine is running, and the requested model is present on that node. To route across several -machines, download the same model on each — the Nodes tab's `MODELS` column, and -each machine's model list, are how you check. +machines, download the same model on each. The Nodes tab's `MODELS` column and +each machine's model list are how you check. ## Check Routing and Health @@ -379,32 +379,32 @@ see routing actually happening. An active job that has not been placed yet reads ![The terminal interface Jobs tab with the Ollama, LM Studio, and llama.cpp endpoints on top, and jobs listed by model, engine, state, the machine each arrived at, and the machine that ran it.](assets/onboarding/terminal-interface/03-tui-workloads.png) -Routing is not something you set. PAIR chooses a node per request from pending +Routing is not something you set. PAIR chooses a node for each request from pending work and GPU pressure, so the proxies always run in automatic mode. ### Generate Some Traffic If you have no client sending work yet, press `t` on the **Jobs** tab to run the -same sixty-second test the desktop application's **Test** button runs. It sends +same 60-second test the desktop application's **Test** button runs. It sends synthetic requests to your local endpoints, and the jobs appear in the table -below as they are placed — including on peers, if the cluster has any. +below as they are placed, including on peers if the cluster has any. A progress line shows how many requests have gone out and how long is left. -Press `t` again to stop early; requests already sent still finish. Nothing is -sent after sixty seconds. +Press `t` again to stop early. Requests already sent still finish. Nothing is +sent after 60 seconds. It needs at least one engine running with a model installed. If nothing is available, the tab says so rather than starting. The test is local to the machine you run it on. Running it here does not start -one anywhere else, and a peer running one does not show up as progress here — +one anywhere else, and a peer running one does not show up as progress here, only as jobs. The **Service** tab (3) reports whether each worker is up. Worker status is best-effort: `DOWN` means a worker reported a crash. The error reporting worker -is the one that collects those reports, so it cannot report its own — its row +is the one that collects those reports, so it cannot report its own. Its row says so. If the service itself is not answering, every row reads `?` rather than -`ok`: a worker is only known good because the crash stream said nothing about +`ok`. A worker is only known good because the crash stream said nothing about it, and that stream comes through the service. ## Update Notices @@ -414,25 +414,25 @@ bar naming the new version, the one you are running, and where to get it. It checks shortly after startup and every six hours, the same cadence the desktop application uses. -The notice is on every tab, not just **Service**, and stays until you press +The notice is on every tab, not only **Service**, and stays until you press `ctrl+x` to dismiss it. Dismissing it clears it everywhere. A release newer than the one you dismissed brings it back, since that is not the version you acknowledged. On a narrow terminal the line shortens rather than pushing the dismiss key off the screen. It only tells you. It does not download or install anything, because the -terminal interface is not a single program — it runs a broker and ten -workers from the directory beside it, and swapping some of them for a newer set -while they are serving inference is how you get a client talking to workers that -no longer understand it. Install the release the way you installed this one. +terminal interface is not a single program. It runs a broker and 10 workers from +the directory beside it. Swapping some of them for a newer set while they are +serving inference is how you get a client talking to workers that no longer +understand it. Install the release the way you installed this one. -If you installed PAIR as the desktop application, you do not need to: the -terminal interface ships inside the application and its updater replaces this +If you installed PAIR as the desktop application, you do not need to. The +terminal interface ships inside the application, and its updater replaces this binary along with everything else, so `nvpair` picks up the new version the next time you run it. -Nothing is reported when the check cannot reach the network — an unreachable -feed is not news. To stop it checking at all, set `NVPAIR_NO_UPDATE_CHECK=1` +Nothing is reported when the check cannot reach the network, because an +unreachable feed is not news. To stop it checking at all, set `NVPAIR_NO_UPDATE_CHECK=1` before starting `nvpair`. A copy you build yourself with `go build` never checks. One built with `services/build.sh` or `services/build.bat` does, because those scripts record the release number in it. @@ -441,21 +441,21 @@ those scripts record the release number in it. On **Errors** (4), press `c` to clear the selected entry. The tab label carries the count, so you do not have to go looking. The line under the table describes -whichever entry you have selected — which engine and operation it came from, the +whichever entry you have selected, which engine and operation it came from, the model if there was one, and what the service suggests doing about it. Errors are shared across the cluster, but clearing one only works on the machine -that reported it — the `NODE` column tells you which that is. Select an entry +that reported it. The `NODE` column tells you which that is. Select an entry from another machine and `c` is withdrawn from the footer, with a line naming the node to clear it from. Clearing there removes it everywhere. On **Logs** (5), scroll with `j` / `k` and the page keys. Press `/` to filter to lines containing a string, `c` to clear the filter, and `t` to toggle tailing -the newest line — scrolling up releases follow on its own. This is the first place +the newest line. Scrolling up releases follow on its own. This is the first place to look when something has not started. Press `s` to write the log buffer to a timestamped file in your home directory. -The whole buffer is saved, not just what the filter shows. +The whole buffer is saved, not only what the filter shows. Set how much detail the services log on the **Service** tab, not here. @@ -463,21 +463,21 @@ Set how much detail the services log on the **Service** tab, not here. On **Service** (3), move with `j` / `k` and press `enter`. The cluster name opens as a text field, with `enter` to save and `esc` to cancel. Ports are not changed -here — they belong to a machine, so they live on its detail screen. +here. They belong to a machine, so they live on its detail screen. **Cluster name** is a label for your own benefit and is stored on the machine you set it on. It is not shared with the rest of the cluster, so setting it here does not rename anything on your peers — set it on each machine you want it to show -on. Nothing operational depends on it; the cluster's real identity is an id you +on. Nothing operational depends on it. The cluster's real identity is an ID you never have to type. -**Service log level** opens a picker showing all four levels — debug, info, -warn, and error — with the one currently in force highlighted. Move with `←` / +**Service log level** opens a picker showing all four levels (debug, info, +warn, and error), with the one currently in force highlighted. Move with `←` / `→` (or `h` / `l`), press `enter` to apply, and `esc` to leave it alone. The -level applies to every service, not just one. +level applies to every service, not only one. Ports and startup arguments are not changed here. They live on the engine they -belong to, on that machine's node detail screen — press `enter` on a machine on +belong to, on that machine's node detail screen. Press `enter` on a machine on the **Nodes** tab. Refer to [Ports](getting-started.mdx#7-connecting-your-agents-and-port-information) for how PAIR arranges ports. @@ -490,7 +490,7 @@ services are still running, and the services are then stopped before the files are deleted, so nothing rewrites them on the way out. If an engine PAIR installed cannot be removed, the reset stops before deleting anything, names the engine on the status line, and leaves you in the interface to try again. You -will need to pair again afterwards. +need to pair again afterward. ## What the Terminal Interface Cannot Do @@ -499,14 +499,14 @@ It is an operations tool, not a full replacement for the desktop application: - It has no built-in way to send an inference request of your own. Use `curl` or another client against the endpoint port, as in [Getting Started](getting-started.mdx#6-run-your-first-inference). The - synthetic test traffic on the **Jobs** tab is not a substitute: you do not - choose the prompt and you never see the reply. -- It cannot install PAIR updates. It tells you when a release is available; - installing one is the same job it was before. + synthetic test traffic on the **Jobs** tab is not a substitute. You do not + choose the prompt, and you never see the reply. +- It cannot install PAIR updates. It tells you when a release is available. + Installing one is the same job it was before. - It does not survive losing your connection. Use a terminal multiplexer, as above. -## See Also +## Other References These documents cover related topics: diff --git a/docs/troubleshooting.mdx b/docs/troubleshooting.mdx index dee380e0..9b42b21e 100644 --- a/docs/troubleshooting.mdx +++ b/docs/troubleshooting.mdx @@ -79,18 +79,18 @@ If **Endpoints** cannot find a usable engine or model: 4. Select **Add model**, download a model, and load it if the engine requires an explicit load step. -PAIR may adopt an existing local Ollama or LM Studio installation instead of +PAIR can adopt an existing local Ollama or LM Studio installation instead of installing a new copy. An engine must be running before you can use models. To test routing across multiple nodes, prepare the same model on each node that should serve requests. -For a fuller walkthrough of install, start, stop, update, uninstall, and what +For a fuller walkthrough of install, start, stop, uninstall, and what PAIR restores after relaunch, refer to [Managing Engines](engine-lifecycle.mdx). ## An Application Cannot Reach PAIR -**First, is the application on the same machine as PAIR?** If it is somewhere else +**First, check that the application is on the same machine as PAIR.** If it is somewhere else on the network, that is the problem, and it is deliberate. An endpoint accepts plaintext requests from loopback only. PAIR refuses a request from another machine with `403` and a message saying the endpoint is loopback-only. Install @@ -154,7 +154,7 @@ Routing confirmation happens on **Overview**: 1. Use the **Jobs** filter in the left column to show active, completed, or failed requests. 2. Open a job card and read **Running on** or **Ran on** to see which node - served the request. Once a job has been sent to a node, its card shows + served the request. After a job has been sent to a node, its card shows **Sent to** until the job starts, and keeps it if the job ends without starting. @@ -201,7 +201,7 @@ Refer to [Collecting and Sanitizing Logs](log-collection.mdx) for the options. When a problem involves more than one machine, copy each node's log onto one machine and run the sanitizer once over all of them. That is the only way the same node reads the same way in every file. Refer to -[Collecting from More Than One Node](log-collection.mdx#collecting-from-more-than-one-node). +[Collecting from More than One Node](log-collection.mdx#collecting-from-more-than-one-node). ## Getting More Help