LIM: Stateful Local Inference Manager

LIM (Local Inference Manager) is a C++ terminal-based LLM controller built on llama.cpp. It provides a persistent, stateful session where the KV-cache is never discarded: each turn's tokens are simply appended, so new input costs only O(input tokens), not O(total history). LIM has native filesystem tools for reading, searching, editing, and writing files, plus web searching and PDF reading.

How It Works

Every mainstream chatbot (and most local LLM frontends using server APIs) follows the reprocess-every-turn model: each time you send a message, the entire conversation history is re-transmitted in full and re-tokenized from scratch. The per-turn decode cost grows linearly with context length, making long conversations progressively slower.

LIM avoids this by design: it runs locally as a single persistent process where the KV-cache is never discarded. Each turn simply continues from where the last one left off. This is the same approach used by llama-cli in interactive mode. LIM includes an integrated tool system, session save/restore, interactive undo, and LaTeX-aware browser output.

---

Architecture at a Glance

+-------------+     +---------------+     +---------------+
|  Your TTY   |<--->|   lim (C++)   |<--->|  llama.cpp    |
| (readline)  |     |  controller   |     |  inference    |
+-------------+     +-------+-------+     +---------------+
                            |
              +-------------+-------------+
              |             |             |
        +-----v-----+ +----v-----+ +-----v------+
        | Filesystem| | Web      | |  Browser   |
        |  Tools    | | Search   | | (FIFO)     |
        |           | | & PDFs   | |            |
        +-----------+ +----------+ +------------+

• lim: The C++ binary. Handles the REPL, KV-cache management, tool dispatch, and signal handling.
• limServer.py: An optional Python WebSocket server that streams output to a browser via /tmp/lim.fifo.

---

Prerequisites

1. A GPU with CUDA support (NVIDIA recommended) and the CUDA toolkit installed. For CPU-only builds, run make GGML_CUDA=off.
2. Python 3 with aiohttp for the browser server: pip3 install aiohttp.
3. A GGUF model file (e.g., Qwen, Llama, Mistral).
4. Optional: SearXNG for web search and Docling for PDF reading. LIM auto-starts them on demand; override the commands with LIM_SEARXNG_CMD / LIM_DOCLING_CMD (see Web Search & PDF Setup below).

Note: llama.cpp is bundled as a git subrepo with a LIM-specific patch for recurrent state checkpointing (required for instant /undo on hybrid models). The patched code is built automatically by the Makefile. A PR upstream to llama.cpp is pending.

---

Building

git clone https://github.com/statefullm/lim.git
cd lim
make
./lim --help

The build will automatically detect your GPU (if any) and compile llama with the appropriate architecture flags. No manual setup is required.

Custom Build Options

All auto-detected values can be overridden via environment variables:

Variable	Purpose	Example
CUDA_ARCH_FLAGS	Override GPU architecture detection	make CUDA_ARCH_FLAGS=120a
LIM_LLAMA_BUILD_DIR	Use pre-built llama from external directory	export LIM_LLAMA_BUILD_DIR=/path/to/llama/build
LIM_LLAMA_CMAKE_FLAGS	Extra cmake flags passed to the llama build	make LIM_LLAMA_CMAKE_FLAGS="-DGGML_AVX512=off"
GGML_CUDA	Force CUDA on/off	make GGML_CUDA=off for CPU-only
GGML_HIPBLAS	Enable ROCm/HIP build	make GGML_HIPBLAS=on

To clean only the llama build artifacts without touching lim:

make llama-clean

Installing LIM

After building, install the binary and config files. This must be run as $LIM_AI_USER (e.g., su - $LIM_AI_USER) so that files are installed into that user's home directory, which is where LIM will look for them at runtime:

su - $LIM_AI_USER
make install

This installs:

• lim binary to ~/bin/lim (ensure ~/bin is in your PATH via .bashrc)
• System prompt to ~/.config/lim/prompt
• Reincarnate instructions to ~/.config/lim/reincarnate
• Search cache directory at ~/.config/lim/searchCache
• Browser server files (limServer.py, viewer.html, libs/) to ~/.config/lim/

All LIM configuration files live under ~/.config/lim/ by default. Override with the LIM_CONFIG_DIR environment variable (set in $LIM_AI_USER's .bashrc) to place them elsewhere:

# As $LIM_AI_USER
export LIM_CONFIG_DIR=/opt/lim/config
make install

The LIM_CONFIG_DIR is read at runtime, so no recompilation is needed.

---

User Setup

The dedicated LLM user is controlled by the LIM_AI_USER environment variable (defaults to ai). All references below use $LIM_AI_USER. The group name always matches the user name.

1. Creating the AI User

LIM must run as a dedicated user. The binary enforces this at startup: it checks getuid() and refuses to run otherwise. This is a security measure: the LLM has filesystem write access and shell execution capabilities, so isolating it behind a dedicated user limits blast radius.

Create the user and group if they don't exist:

export LIM_AI_USER=ai
sudo groupadd -f $LIM_AI_USER
sudo useradd -m -g $LIM_AI_USER -s /bin/bash $LIM_AI_USER

Then add your personal user to the $LIM_AI_USER group so you can read and write files the LLM creates without needing sudo chown:

sudo usermod -aG $LIM_AI_USER $USER

Log out and back in (or run newgrp $LIM_AI_USER) for the group change to take effect.

2. Git Safe Directories

By default, Git refuses to operate in repositories owned by a different user. Since LIM runs as $LIM_AI_USER but your project directories are owned by you, add the [safe] section to your personal ~/.gitconfig:

[safe]
    directory = /home/$LIM_AI_USER

This prevents "fatal: detected dubious ownership in repository" errors when the LLM runs git commands.

3. Directory Permissions

Your project directories should be group-writable by the $LIM_AI_USER group so the LLM can read and write files. Because you added your personal user to the $LIM_AI_USER group (step 1), access is symmetric: you can read files the LLM creates, and the LLM can read your files. The recommended layout for /home/$LIM_AI_USER:

$ ls -ld /home/$LIM_AI_USER
drwxrwsr-x. 42 $USER $LIM_AI_USER 4096 Jan  1 00:00 /home/$LIM_AI_USER/

The setgid bit (s) ensures new files inherit the $LIM_AI_USER group.

To set permissions in any project sandbox, copy aishare from this repository to your personal /home/$USER/bin/:

cp aishare /home/$USER/bin/

Then run it on any directory you want the AI to access:

aishare /home/$LIM_AI_USER/project

It sets ownership to $USER:$LIM_AI_USER, grants group read/write and setgid, and clears the sticky bit. When run on /home/$LIM_AI_USER, it fixes permissions on the home directory itself plus all top-level entries, skipping .ssh.

4. Connecting and Running LIM

LIM must be run as user $LIM_AI_USER. The simplest approach is to SSH into the host as that user, then launch lim directly. This keeps the workflow consistent whether you're using VS Code or a standalone terminal.

Add this to the ~/.bashrc of $LIM_AI_USER:

alias coder='lim ~/models/Qwen3.6-27B-UD-Q5_K_XL.gguf'

Replace the model path with whichever GGUF you want to use. Then, before running coder, connect to the LIM server $LIM_HOST:

ssh $LIM_AI_USER@$LIM_HOST
coder

If running locally (no SSH needed), switch to the user first:

su - $LIM_AI_USER
coder

Core Pinning (taskset)

LIM automatically detects P-cores and E-cores on hybrid CPUs (Intel Alder Lake, Raptor Lake, etc.) by comparing thread_siblings_list from sysfs. It pins background services accordingly:

Workload	Default Cores	Description
SearxNG search server	E-cores	Light background Python process
Browser WebSocket server	E-cores	Light background Python process
Docling PDF converter	P-cores	Heavy ML inference workload

On non-hybrid CPUs (all cores identical), no taskset pinning is applied. If the pinning command isn't found on $PATH (e.g., macOS has no taskset), pinning is silently skipped.

Override the auto-detected layout with LIM_TASKSET:

# Format: LIM_TASKSET="P_CORES:E_CORES"
export LIM_TASKSET="0-15:16-23"   # Classic i9-12900K layout
export LIM_TASKSET="0-7:"         # P-cores 0-7, no E-core pinning
export LIM_TASKSET=":8-15"        # No P-core pinning, E-cores 8-15
export LIM_TASKSET="::"           # Disable all taskset pinning

macOS / non-Linux systems: macOS has no taskset binary. To enable manual core pinning, install an alternative and point LIM_TASKSET_CMD at it:

# Install numactl via Homebrew, then use it as the pinning backend
brew install numactl
export LIM_TASKSET="0-3:4-7"
export LIM_TASKSET_CMD="numactl --cpunodebind"

Or write a custom wrapper script and set LIM_TASKSET_CMD to its path. If the command isn't on $PATH, pinning is silently skipped -- services still start normally, just unpinned.

The detected topology and pinning status are logged to stderr at startup.

5. Setting Up $HOME/.bashrc for $LIM_AI_USER

Add these lines to /home/$LIM_AI_USER/.bashrc:

# --- /home/$LIM_AI_USER/.bashrc ---

umask 0002                    # Group-writable files by default

# Track current working directory so the LLM knows where you are.
cd() {
    builtin cd "$@" && pwd > $HOME/.cwd
}

# Block git add -A / --all and git add . to prevent the LLM from staging untracked files
git() {
    if [ "$1" = "add" ]; then
        for arg in "$@"; do
            if [ "$arg" = "-A" ] || [ "$arg" = "--all" ] || [[ "$arg" =~ ^-[a-zA-Z]*A[a-zA-Z]*$ ]] || [ "$arg" = "." ]; then
                echo "ERROR: git add -A / --all / . is disabled: use git commit -a or specify files explicitly" >&2
                return 1
            fi
        done
    fi
    command git "$@"
}

# Optional: add custom paths
export PATH="$HOME/bin:$PATH"

The cd override writes the current directory to $HOME/.cwd, which LIM reads at startup so it knows your working directory. The umask 0002 ensures files created by $LIM_AI_USER are group-readable/writable.

6. The System Prompt

The system prompt lives at ~/.config/lim/prompt. A default prompt file ships with this repository and is installed to that location by make install. This file is read once at startup and baked into the KV-cache. It defines the LLM's behavior: available tools, editing workflow, formatting rules, etc. You can customize it for different use cases (coding assistant, writer, researcher, etc.).

7. Message Shortcuts

Create /home/$LIM_AI_USER/.lim_aliases to define shorthand expansions at the >>> prompt:

# ~/.lim_aliases: one key=value per line; # comments are ignored
# Keys must start with / to distinguish them from regular messages

/test=Test the filesystem tools and clean up after.
/message=Show me a 1-sentence concise git commit message relative to HEAD.
/commit=run git commit -a -m "<message>"
/make=run make
/amend=Amend the previous commit.
/remind=Read the system prompt from ~/.config/lim/prompt and follow it strictly!
/prompt=Follow the system prompt strictly!
/tools=Use our robust filesystem tools.

At the >>> prompt, typing /commit expands to run git commit -a -m "<message>" before being sent to the LLM. Lines starting with # are treated as comments. Blank lines are skipped. Built-in command names (e.g., /quit, /clear) cannot be used as alias keys. This is loaded fresh each session.

---

Output Modes

Set via LIM_OUTPUT:

Value	stdout	Browser	Description
3	✅	✅	Both stdout and browser
2 (default)	❌	✅	Browser only
1	✅	❌	Stdout only
0	❌	❌	No output (system stderr still works)

---

Environment Variables

Variable	Default	Description
LIM_AI_USER	ai	Username that LIM must run as. Used by the binary for the user check, and by the VS Code extension for SSH.
LIM_CACHE_DIR	.cache	Directory (relative to CWD) for fast restore KV-cache files.
LIM_CONFIG_DIR	~/.config/lim	Directory for LIM config and server files (prompt, reincarnate, userprompt, searchCache, limServer.py, viewer.html, libs/). The C++ binary launches python3 $LIM_CONFIG_DIR/limServer.py when browser output is enabled. Override to place everything elsewhere.
LIM_LOG_DIR	log	Directory (relative to CWD) for session logs, token traces, and autosave files.
LIM_SAVE_DIR	.	Directory prepended to relative paths in /save, /load, /delete, and the CLI restore argument. Absolute paths (starting with /) are unaffected.
LIM_HOST	unset	Hostname or IP of your LIM server. Used for SSH connection and browser viewer URL.
LIM_PORT	8765	Port for the browser WebSocket server
LIM_OUTPUT	2	Output mode: 0 = none, 1 = stdout only, 2 = browser only (default), 3 = both
LIM_VIEWER_URL	(auto)	Override the auto-generated viewer URL
LIM_WEB_CONTEXT_FRACTION	0.75	Fraction of LIM_CTX reserved for fetched web content budget. The per-file limit (LIM_WEB_FILE_MAX) defaults to 25% of this budget, so ~3-4 files fill it. Set between 0.0 and 1.0.
LIM_WEB_FILE_MAX	(auto)	Max characters per fetched file before middle-drop truncation. Defaults to 25% of the session web budget (LIM_CTX * LIM_WEB_CONTEXT_FRACTION * 4). Set explicitly to override.
LIM_WEB_HTML_MAX	500000	Max bytes to buffer when downloading HTML/text pages via curl
LIM_WEB_PDF_MAX	50000000	Max bytes to buffer when downloading PDFs via curl (50 MB)
LIM_WEB_TIMEOUT	600	HTTP request timeout in seconds for fetches and searches
LIM_SEARCH_COOLDOWN	3	Minimum seconds between web searches to avoid rate-limiting SearxNG
LIM_DOCLING_CMD	~/venv/bin/docling-serve run --enable-ui	Command to start the Docling PDF service. Override if installed elsewhere (e.g., via Docker or a different venv).
LIM_SEARXNG_CMD	cd ~/searxng && python -m searx.webapp	Command to start the SearxNG search service. Override if installed elsewhere.
LIM_DEBUG	0	Set to 1 for verbose token-level logging in $LIM_LOG_DIR/<N>.tokens
LIM_EOG_RESAMPLE_MAX	64	Maximum resampling attempts when a spurious EOG is detected. When the model emits an EOG token but hasn't finished its response, LIM resamples up to this many times trying to recover a non-EOG token. Increase if you see premature turn endings with Qwen3.6 or similar models.
LIM_GPU_LAYERS	-1	Number of layers offloaded to GPU (-1 = auto-fit all layers). When set explicitly, bypasses auto-fitting. For MoE models that exceed VRAM, auto-fit uses partial layer offloading (dense weights on GPU, sparse expert weights on CPU) for optimal throughput.
LIM_HONEST_SPEED	0	Set to 1 for "honest" wall-clock speed diagnostic -- tokens / total generate() call time, including all CPU-side overhead (sampling, output rendering, tool-call detection). Default 0 reports generation time matching llama-cli's "Generation: X t/s" -- measured from first-token sample+sync to last-token sample+sync, covering N sampling operations and (N-1) decode cycles.
LIM_SPEED_INTERVAL	100	Number of tokens between in-loop speed diagnostic updates.
LIM_USE_MLOCK	1	Lock model in RAM to prevent swapping
LIM_USE_MMAP	0	Use memory-mapped model loading (faster startup, more RAM pressure)
LIM_BATCH	2048	Batch size for token feeding
LIM_CTX	262144	Context window size (KV-cache token capacity)
LIM_THREADS	(auto)	Threads for inference (physical core count)
LIM_THREADS_BATCH	(auto)	Threads for batch processing (physical core count)
LIM_UBATCH	512	Unbatched size
LIM_MIN_P	0.0	Minimum probability threshold: keep tokens where P >= min_p * P(top)
LIM_PENALTY_FREQ	0.0	Frequency penalty: discourages overused tokens proportional to frequency
LIM_PENALTY_PRESENT	1.5	Presence penalty: discourages repeating previously used tokens
LIM_PENALTY_REPEAT	1.0	Repetition penalty multiplier (1.0 = no penalty)
LIM_SEED	(auto)	Random seed for reproducibility (default is time-based)
LIM_TEMP	0.7	Sampling temperature (set to 0 for deterministic/greedy decoding)
LIM_THINKING	1	Set to 0 to suppress thinking blocks via a pre-filled stub for faster throughput. Not recommended for math or complex reasoning tasks, as it can cause incorrect answers by skipping intermediate steps.
LIM_ESCAPE_CONTRACT	0	Set to 1 to include the reserved-token escape contract in the system prompt. The escape mechanism itself is always active; this only controls whether the LLM sees the explicit rules.
LIM_INLINE_LATEX	1	Set to 0 to disable inline KaTeX rendering in the browser viewer. Block-level $$...$$ math still renders via katex.renderToString. Useful when $HOME, $PATH, and other env vars appear as bare text and get mangled by LaTeX processing.
LIM_TOP_K	20	Keep only the top_k most likely tokens before applying other samplers
LIM_TOP_P	0.8	Nucleus sampling: consider tokens with cumulative probability <= top_p
LIM_CACHE_TYPE_K	Q8_0	KV-cache key storage type (F16, Q4_0, Q5_0, Q5_1, Q8_0, Q8_1)
LIM_CACHE_TYPE_V	Q8_0	KV-cache value storage type (F16, Q4_0, Q5_0, Q5_1, Q8_0, Q8_1)
LIM_CHATBOT_MODE	0	Benchmarking mode: 0 = LIM normal (persistent KV-cache, same as llama-cli interactive), 1 = standard chatbot (re-decode full history each turn), 2 = cache-aware prefix match (emulates llama-server). Modes 1 and 2 force honest speed measurement.
LIM_EXEC_TRUNCATION	32768	Maximum bytes of exec_shell output before truncation
LIM_MAX_AUTO_CONTINUE	500	Maximum depth of automatic tool-call chaining
LIM_TURN_TIMEOUT	300	Maximum seconds per generation turn before auto-abort
LIM_TASKSET	(auto)	Format: "P_CORES:E_CORES" (e.g., "0-15:16-23"). Auto-detected on hybrid CPUs. Set to "::" to disable all pinning.
LIM_TASKSET_CMD	taskset -c	Override the core-pinning command. On macOS (no taskset), install numactl via Homebrew and set to numactl --cpunodebind. If the command isn't on $PATH, pinning is silently skipped.

Sampling Behavior

Between user turns, LIM resets the sampler chain (penalties ring buffer and RNG seed). This means repetition penalties apply only to tokens generated during the current turn, not to stale tokens from previous responses, and not to the user's input. This differs from llama-cli, which feeds user input tokens into the sampler chain: in a chat interface, penalizing words the user explicitly used in their message would actively harm response quality. During auto-continue (tool-call chains, /continue), the sampler state is preserved so generation continues seamlessly.

---

The AI Sandbox

The $LIM_AI_USER operates in a sandboxed environment:

• File access is limited to directories where the $LIM_AI_USER group has write permission. Use aishare to grant access to new project directories. Files created by the LLM are owned by $LIM_AI_USER:$LIM_AI_USER with group read/write permissions (via umask 0002), so your personal user can access them directly since it is a member of the $LIM_AI_USER group.
• Shell commands executed via the exec_shell tool run as user $LIM_AI_USER. They inherit that user's PATH and environment.
• Git integration: The sandbox repo is a separate git repository that is writable by $LIM_AI_USER using a dedicated AI GitHub account. This allows the LLM to commit, push, and manage version control autonomously without needing your personal credentials.

Set up an SSH key:

# As `$LIM_AI_USER`, generate a dedicated key
ssh-keygen -N ""

# Add the public key to your Git hosting provider
cat ~/.ssh/id_ed25519.pub

---

Usage

Starting a Session

SSH in as $LIM_AI_USER, then run your coder alias. You'll see the >>> prompt. Type your request and press Enter.

Available Tools

The LLM has access to six tools. You don't call them directly; just describe what you want and the LLM handles the rest:

• read_files: Read text files, PDFs, or URLs
• search_file: Search for text within a file, with optional line range
• edit_file: Replace exact text within a file (surgical editing)
• write_file: Write or overwrite a file
• exec_shell: Run a shell command as $LIM_AI_USER
• web_search: Search the web via SearXNG

Tools are invoked via XML-structured tags in the LLM's output. Results are fed back as continuation tokens into the KV-cache, avoiding the overhead of re-tokenizing the full conversation history.

Multi-line Input with Ctrl+J

The readline interface binds Ctrl+J to insert a literal newline into your input, while Enter (Return) submits the line. This lets you compose multi-line messages:

>>> Here's a function I need fixed:
(ctrl-j)
(ctrl-j)
def foo(x):
(ctrl-j)
    return x + 1
(ctrl-j)
(ctrl-j)
Please fix the type hints.      <-- Enter submits

The binding is restored after each input cycle so it doesn't leak into your shell session.

Readline Keybindings

The prompt uses GNU readline in callback mode with select() polling instead of blocking I/O. This allows Ctrl+C interrupts to be handled immediately during user input, not just during generation. All standard Emacs-style bindings are available:

Keys	Action
Ctrl+A / Ctrl+E	Beginning / end of line
Ctrl+B / Ctrl+F	Backward / forward character
Alt+B / Alt+F	Backward / forward word
Ctrl+K / Ctrl+U	Kill to end / start of line
Ctrl+W	Kill word backward
Ctrl+Y	Yank (paste)
Ctrl+_	Undo
Tab	Completion (if configured)
Up / Down	History navigation
Ctrl+R	Reverse search history

In-Session Commands

Command	Effect
/quit or /exit	Auto-save the current state to $LIM_LOG_DIR/<N>.save, then exit the session
/clear	Auto-save the current state to $LIM_LOG_DIR/<N>-clear.save, then clear the KV-cache (reset to system prompt only). The auto-saved file lets you restore if you change your mind. Use /save <name> to create a permanent restore point before clearing.
/undo	Interactive undo: auto-saves first to $LIM_LOG_DIR/<N>-clear.save, then presents an Undo> prompt listing all checkpoints (most recent first). Use up/down arrows to navigate, Enter to confirm. Ctrl+C, Ctrl+D, /quit, or /exit cancel the undo and return to the user prompt without losing your session. Selecting a checkpoint restores the session to the end of the turn associated with that prompt. Falls back to clear+re-decode only if no checkpoint is available. Readline history is updated to reflect the restored session state.
/continue	Resume generation after an interruption. If interrupted mid-tool-call, resumes from the exact point of interruption
/reset	Reset terminal, loop detector, and web search. Useful for recovering from a corrupted terminal or disabled web search after an interrupt or connection failure
/reincarnate	Ask the LLM to compose a new prompt in ~/.config/lim/userprompt, then clear and restart with it
/save	Save the full session state to $LIM_LOG_DIR/<N>.save, overwriting any previous save for this session
/save <path>	Save the full session state to <path>.save. The path can be relative or absolute. If it already ends in .save, no extra extension is added. Use this to create named restore points at meaningful moments in your session.
/load <path>	Load a saved session from within the current session. Must be used immediately after /clear: fails if any conversation tokens have been added since the clear. Accepts the same path format as /save: .save is appended automatically if not already present. Uses the fast cache when available, falling back to full re-decode with checkpoint regeneration.
/delete <path>	Delete a save file and its associated fast restore cache entry (if any). Accepts the same path format as /save: .save is appended automatically if not already present.
/help	Display a summary of all available commands

Save and Restore

You can save a running session and restore it later with zero context loss:

Save: Type /save at the >>> prompt to save to $LIM_LOG_DIR/<N>.save (overwrites any previous save for this session). Use /save <path> to create named checkpoints: e.g., /save cats saves to cats.save, and /save /tmp/checkpoint saves to /tmp/checkpoint.save. If the path already ends in .save, no extra extension is added. The save file contains only the conversation token sequence, keeping it small and model-agnostic. Named saves also cache the full KV-cache for fast future restores.

Restore: Pass a save file as the last argument to coder. The .save extension is added automatically if not already present, matching /save behavior:

coder $LIM_LOG_DIR/5.save    # explicit extension
coder $LIM_LOG_DIR/5         # .save appended automatically
coder cats          # restores from cats.save

This restores the session exactly as it was: the full conversation, KV-cache position, and generation state. The LLM continues generating from where it left off. Typing /clear after a restore resets to a fresh system prompt with the current date and working directory (but first auto-saves the restored state).

In-session load: You can also restore from within a running session using /load <path>. This must be used immediately after /clear; if any conversation tokens have been added since the clear, the command will fail. It uses the same path conventions as /save (.save appended automatically) and takes advantage of the fast cache when available. Typical workflow:

>>> /clear
[Context Cleared Successfully]
>>> /load cats
[Restoring session from cats.save... (75432 tokens, from cache)]
[Session restored: 75432 tokens loaded]

Partial restore via checkpoints: Save files record a checkpoint at the end of each conversation turn, storing your prompt text and the token position. On restore, if a fast-format cache is not available, LIM offers a choice of checkpoints before decoding. Use up/down arrow keys to navigate through your prompts (most recent first). Press Enter to confirm. If no checkpoint matches your input, all tokens are restored by default. Press Ctrl+C or Ctrl+D to cancel the restore and start a fresh session. Type /quit or /exit to exit LIM entirely. Restoring to a checkpoint replays tokens only up to the end of that turn -- as if you had just typed that prompt and received the response, and the session is ready for your next message. The available checkpoints accumulate across restore/save cycles: restoring from a save file carries over its checkpoints, and new turns add more.

Checkpoint restore: Add --checkpoints to skip the fast-format cache and trigger the checkpoint selection prompt. The flag can appear before or after the save file:

coder cats --checkpoints
coder --checkpoints cats

Fast restore cache: On first restore, tokens are decoded through the model to rebuild the KV-cache. During this decode, recurrent-state checkpoints are regenerated at each prompt boundary so that /undo works instantly for hybrid models (Qwen3.5/3.6) right after restore. The rebuilt cache is then automatically written to $LIM_CACHE_DIR/<hash> so all subsequent restores from the same save file are fast. Named saves (e.g., /save cats) also write the fast-format cache immediately for faster future restores. The name prefix in the cache filename is purely informational; LIM identifies cache files by their content hash, not their name. If you run /save cats followed by /save dogs with identical session content, only one cache file is written since both resolve to the same hash. Unnamed /save and auto-saves from /quit, /clear, and /reincarnate skip the fast cache to save disk space, relying on the automatic cache built on first restore. For fast cache restores, recurrent checkpoints build up naturally as conversation turns complete to support the instant /undo feature after the first new turn. The $LIM_CACHE_DIR/ directory is safe to delete at any time to reclaim space; it will be regenerated on the next restore.

Auto-save on clear and quit: Before clearing the context, LIM automatically saves the current state to $LIM_LOG_DIR/<N>-clear.save. Before exiting, it saves to $LIM_LOG_DIR/<N>.save. These use different filenames so neither clobbers the other: if you clear and then exit, both the pre-clear and post-clear states are preserved. To keep a permanent checkpoint at any point, use /save <name>.

Interrupting Generation

Press Ctrl+C during generation to interrupt. The partial output is preserved in the KV-cache. Type /continue to resume seamlessly: the LLM doesn't even know it was interrupted. This works because the KV-cache still holds all generated tokens up to the interruption point.

Browser Output

By default (LIM_OUTPUT=2), LIM streams output to a browser via WebSocket. Start your LIM session first, then load:

http://<hostname>:8765/viewer.html

This ensures the server is already running when the browser connects, avoiding any need to reload the page.

The Python server (limServer.py) is auto-started by the C++ binary when browser output is enabled. It runs on efficiency cores (auto-detected, or all cores on non-hybrid CPUs). The server reads from a named FIFO at /tmp/lim.fifo and broadcasts to all connected WebSocket clients.

VS Code Extension

The LIM Workspace extension provides a convenient way to set up your workspace: it creates an integrated terminal and opens a browser viewer once the server is ready.

The extension does two things:

• Opens a browser pointing to $LIM_HOST (or localhost) on the configured port, after waiting for the server to respond.
• Creates a terminal at $HOME. If $LIM_HOST is set and remote, it SSHs into that host as user $LIM_AI_USER; otherwise it opens a local shell.

You then run your coder alias in that terminal as usual.

Install:

make vscode             # builds and packages the extension
make install-vscode     # installs the extension into VS Code

To install everything (binary, config files, and VS Code extension) in one step:

make install-all        # runs make install && make install-vscode
make uninstall-all      # removes installed files and uninstalls the extension

Then in VS Code, click the LIM rocket icon in the status bar (or Ctrl+Shift+P -- LIM: Open Workspace). This opens a terminal panel and waits for the browser server. Run your coder alias in that terminal, and the viewer will connect automatically once the server starts.

To rebuild both the C++ binary and the extension together:

make all

---

Benchmarking

LIM supports benchmarking modes controlled by LIM_CHATBOT_MODE to compare its persistent KV-cache approach against standard chatbot and cache-aware decoding. Each mode writes a TPS log (log/<N>.tps) recording context position and tokens-per-second for every turn.

Value	Mode	Description
0 (default)	LIM normal	KV-cache persists across turns. Each token is decoded once and never re-decoded. Same approach as llama-cli interactive mode.
1	Standard chatbot	Clears cache each turn, reconstructs conversation text from stored tokens, re-tokenizes from scratch, and re-decodes through the model. Simulates a real chat API that receives the full conversation as text each request. TPS includes re-tokenize + re-decode (detokenization is excluded -- it's a LIM implementation detail, not part of chatbot behavior).
2	Cache-aware prefix match	Emulates llama-server behavior: KV-cache stays in memory, but each turn re-tokenizes the full conversation text and compares against the cached prefix to find where to resume decoding. Illustrates that prefix matching overhead is negligible compared to decode cost.

Chatbot modes (1 and 2) automatically enforce LIM_HONEST_SPEED=1. The TPS reported in logs includes the full re-decode overhead. This ensures the benchmark numbers reflect the true wall-clock cost of each approach.

Note: For fair comparisons, run benchmarks with an empty system prompt (remove ~/.config/lim/prompt) to inhibit tool calls. In mode 1, previous tool calls would be both re-decoded and re-executed on every turn, causing massive slowdowns plus unwanted side effects.

---

Web Search & PDF Setup

LIM can start SearXNG and Docling automatically when needed, but you must install them first:

SearXNG (Web Search)

cd ~
git clone https://github.com/searxng/searxng.git
cd searxng
pip install --no-build-isolation -e .

Then edit searx/settings.yml: set a value for server.secret_key (required). The defaults for bind_address (127.0.0.1) and port (8888) are already correct for LIM. You may need to disable some search engines that block automated queries -- see the SearXNG docs for details. Override the start command with LIM_SEARXNG_CMD if installed elsewhere.

Docling (PDF Reading)

cd ~
python3 -m venv venv
source venv/bin/activate
pip3 install docling-serve
deactivate

LIM expects the Docling binary at ~/venv/bin/docling-serve and starts it on port 5001. Override the start command with LIM_DOCLING_CMD if installed elsewhere.

---

Troubleshooting

"FIFO not found" or browser output fails

The FIFO at /tmp/lim.fifo is created automatically when LIM starts. If it was removed manually, restart your LIM session. Ensure aiohttp is installed: pip3 install aiohttp.

"SearxNG connection failed"

LIM tries to connect to SearXNG on 127.0.0.1:8888. Make sure SearXNG is installed at ~/searxng/ and configured to listen on that port in settings.yaml. You can also run SearXNG externally and point LIM to it -- just ensure the server is already listening before your first web search.

"Docling Error" when reading PDFs

Ensure Docling is installed. If it's not at ~/venv/bin/docling-serve, set LIM_DOCLING_CMD to the correct command, e.g.:

export LIM_DOCLING_CMD="/opt/docling/bin/docling-serve run --enable-ui"

"No GPU detected and CUDA_ARCH_FLAGS not set"

If you don't have an NVIDIA GPU, build for CPU only:

make GGML_CUDA=off

Browser viewer shows nothing / won't connect

1. Start LIM first, then open the viewer URL. The server must be running before the browser connects.
2. Check that nothing else is using port 8765 (or your custom LIM_PORT).
3. If behind a firewall or SSH tunnel, set LIM_VIEWER_URL to the correct address.

"detected dubious ownership in repository"

See Git Safe Directories in the User Setup section. Add [safe] directory = /home/$LIM_AI_USER to your personal ~/.gitconfig.

---

Session History

Input history is persisted to .lim_history in the current working directory and survives across sessions.

---

Logging

Each session writes a numbered log file in $LIM_LOG_DIR/<N>. With LIM_DEBUG=1, a companion token-level trace is written to $LIM_LOG_DIR/<N>.tokens showing every token fed into and generated by the model, with labels like FEED SYSTEM_PROMPT_INIT, FEED USER_INPUT, etc. A TPS log is also written to $LIM_LOG_DIR/<N>.tps recording context token count and tokens-per-second at each speed diagnostic update.

---

License

See LICENSE for details.

---