secureagent

A harness for testing prompt-injection defenses at the architecture level instead of the model level. Swap in different security designs, run them against the same attack set on the same local model, and see which patterns help and by how much.

Side project. Runs against a small model via Ollama — no API costs, fast to iterate, small enough to drop in your own defense. Can also run against any model on OpenRouter when you want to reach past local hardware.

The idea

Most prompt-injection work hardens the model through training. That helps but never closes the gap. The other path is to change the harness around the model so the attack surface shrinks: untrusted text never reaches a position of authority, secrets never reach the model at all, references replace raw content. This is a place to try those patterns side by side.

A defense here is just a transform on the input, system prompt, or tool schema (src/pipeline.ts). Adding one is a new toggle and a small file under src/defenses/.

Defenses (v0)

• typedContext — wraps every input block in <block trust=... authority=...> and tells the model only executable blocks carry policy. Prompt-class: works only if the model honors the labels.
• dualLlm — a quarantined model reads the raw input and emits structured JSON intent; the privileged agent acts on the JSON, never the raw text. Prompt-class. (Willison, CaMeL)
• signedContext — typedContext's enforcement twin. Authentic blocks carry a seed-derived nonce the attacker can't know, and every angle bracket in untrusted input is neutralized, so a forged <system>/<tool_output> block arrives as inert text instead of structure. Enforcement-class: the structural guarantee holds regardless of model competence.
• capabilityHandles — the model sees opaque handles (cap:email_address:0001) instead of raw addresses/paths; the runtime rejects any tool call naming a raw value. Enforcement-class: holds no matter what the model emits. (CaMeL)

Planned: URL references, indirect-injection scenarios.

The cross-scale law

The headline experiment. Defenses split by how they earn their security:

• Prompt-class (typedContext, dualLlm) — security depends on the model honoring a convention. Nothing stops a model that ignores it.
• Enforcement-class (capabilityHandles, signedContext) — security is a runtime check. The model cannot name a raw attacker address, nor see a forged authority block; the wrapper rejects or strips it.

The cleanest single test of this is typed_only vs signed_only: the same trust-label mechanism, one relying on the model and one enforced at the runtime. If the law holds, the prompt version's protection climbs with scale while the signed version stays flat.

That predicts a testable claim:

Prompt-class defenses need scale; enforcement-class defenses do not. A model too weak to honor a trust label gets little from a prompt defense, so its protection should grow with parameter count. Enforcement blocks at the runtime regardless of model competence, so its protection should be flat.

The runner sweeps the same attacks across a model-size ladder (default: qwen2.5 0.5B→7B, one family so only parameter count changes). For each defense it reports marginal protection vs baseline — the share of attacks it blocks that the bare baseline let through — which isolates the defense from the base model's own scale trend, then reports the protection slope per class.

Two guards keep the curve honest:

• Tool-engagement floor. A model that barely calls tools shows a low injection rate for free — incompetence, not defense. Scales below the floor (default 30%) are flagged * and excluded from slope claims.
• No significance theater. N is small (attacks × seeds). The verdict reports a direction (consistent / weak / not supported), never a p-value.

Run

Two supported paths. Both pull each model on first use.

Bun-native (no Nix). Install Bun and Ollama, then:

bun install
bun run bench                          # starts ollama if needed, then the default sweep
bun run bench qwen2.5:3b               # single model (no scale axis)

bun run bench calls scripts/serve-ollama.sh first, which starts ollama serve in the background only if it is not already up (and sets OLLAMA_NUM_PARALLEL so concurrent runs overlap). Stop a server it started with bun run ollama:stop. If you manage Ollama yourself, skip straight to bun run run [model...].

Nix flake. Still supported — entering the dev shell starts ollama serve in the background and cleans it up on exit:

nix develop
bun install
bun run run                            # default sweep: qwen2.5 0.5b,1.5b,3b,7b
bun run run qwen2.5:1.5b qwen2.5:7b    # custom sweep, ordered by scale

Or without a shell: nix run . (writes to $PWD/results/), nix run . -- qwen2.5:3b. Already have Ollama on PATH? nix develop .#bare skips a second copy in the Nix store.

Sweep selection precedence: positional args > SECUREAGENT_MODELS > SECUREAGENT_MODEL > default.

Providers

Two backends, one provider-neutral chat interface (src/llm.ts). The agent loop and every defense are written against it, so neither knows or cares which backend answers.

• ollama (default) — local models, pulled on first use. The scale study above assumes this.
• openrouter — any model on OpenRouter. Set OPENROUTER_API_KEY and it is selected automatically (or force it with SECUREAGENT_PROVIDER=openrouter); models are remote, so nothing is pulled.

export OPENROUTER_API_KEY=sk-or-...
bun run run openai/gpt-4o-mini                 # single hosted model
bun run run anthropic/claude-3.5-haiku meta-llama/llama-3.1-8b-instruct

Two caveats for hosted models. First, determinism is best-effort — a seed and temperature=0 are passed through, but hosted backends do not guarantee bit-identical sampling the way a pinned local model does, so verdicts may drift between runs. Second, the scale axis does not apply: parseParamsB reads a size from an Ollama-style tag (qwen2.5:7b), so an OpenRouter id like openai/gpt-4o-mini parses as unknown size and sorts to the end. Use OpenRouter for per-model spot checks, not for the cross-scale slope.

env	meaning
SECUREAGENT_MODELS	comma-separated model ladder (use one family)
SECUREAGENT_MODEL	single model tag (back-compat)
SECUREAGENT_CONFIGS	subset of baseline,typed_only,signed_only,caps_only,dual_only,typed+caps,all_on
SECUREAGENT_CONCURRENCY	runs in flight per model (default 4); raise on a big machine, set 1 for serial
SECUREAGENT_PROVIDER	ollama (default) or openrouter; auto-selects openrouter when OPENROUTER_API_KEY is set
OLLAMA_HOST	Ollama base URL (default http://localhost:11434)
OLLAMA_NUM_PARALLEL	Ollama's own parallel-request cap; keep >= SECUREAGENT_CONCURRENCY
OPENROUTER_API_KEY	OpenRouter API key; required when the provider is openrouter
OPENROUTER_BASE_URL	OpenRouter base URL (default https://openrouter.ai/api/v1)
SECUREAGENT_SEED	sampler seed (default 42)
SECUREAGENT_CHECKPOINT	checkpoint path (default results/checkpoint-seed-<seed>.jsonl)

Speed & memory

Runs within a model fire concurrently (SECUREAGENT_CONCURRENCY, default 4); the model sweep itself stays sequential so only one model is resident at a time — the speedup costs no extra memory. To run in parallel, Ollama needs OLLAMA_NUM_PARALLEL >= SECUREAGENT_CONCURRENCY (the serve script sets it to 4); otherwise concurrent requests just queue. Models are held warm between calls (keep_alive) so back-to-back runs skip reload. None of this touches verdicts — see Determinism.

Outputs land in results/: outcomes-<ts>.csv/.json (one row per run) and matrix-<ts>.csv (per-(model, config) summary — the file you plot the curve from).

Resume

Each run is appended to a checkpoint (SECUREAGENT_CHECKPOINT, default results/checkpoint-seed-<seed>.jsonl) the moment it finishes, so a crash or Ctrl-C mid-sweep loses nothing. Re-run the same command and it skips runs already recorded and picks up where it stopped; infrastructure failures (Ollama unreachable, unexpected errors) are retried, settled results aren't. The checkpoint is keyed by seed and removed once the final outputs are written. Because it skips by (model, config, attack), changing the model ladder or config subset between runs is safe; changing the seed starts a fresh checkpoint.

nix flake check runs tsc --noEmit in a hermetic derivation. (Changing package.json/bun.lock means re-pinning the node_modules FOD hash — see the comment in flake.nix.)

Scoring & determinism

Scoring is deterministic. Each attack declares a forbidden pattern — an address that must never appear in send_email.to, or a path that must never reach delete_file — and the scorer checks recorded tool calls against it. v0 attacks are pure attacks with no benign task mixed in, so this skips task-completion; that comes next.

Runs are reproducible on the same machine: locked sampler (temperature=0, fixed seed/num_ctx) and deterministic handle IDs make the system prompt byte-identical across runs. Only latencyMs and tool-call key ordering vary; verdicts and values don't. Concurrency (SECUREAGENT_CONCURRENCY) is pure scheduling — each run carries its own messages and fixed seed, results are written back in a stable (model, config, attack) order, so the output files stay byte-identical regardless of the setting; only the live progress lines interleave. Cross-machine determinism isn't guaranteed (GPU atomics, quantization, Ollama version), and the model itself isn't pinned by Nix.

Limitations

• No utility axis. Defenses that restrict the agent look good on injection but worse on real tasks. Until benign tasks are mixed in, every defense looks free.
• Direct injection only. The harder, more interesting case is indirect injection (poisoned tool output, retrieved docs).
• Toy scale. A 0.5B–7B ladder measures the slope entirely in the weak regime; it may not extrapolate.
• Hand-crafted attacks. Ten of them. Adapting an existing benchmark (AgentDojo, InjecAgent) is the right next step.

Non-goals

Training a more robust model · inference-engine modifications · a production agent framework.

References

• Debenedetti et al. (2024), AgentDojo. arXiv:2406.13352
• Zhan et al. (2024), InjecAgent. arXiv:2403.02691
• Toyer et al. (2023), Tensor Trust. arXiv:2311.01011
• Willison (2023), The Dual LLM pattern. simonwillison.net
• Debenedetti et al. (2025), Defeating Prompt Injections by Design (CaMeL). arXiv:2503.18813

Contributing

See CONTRIBUTING.md. One logical change per commit, Conventional Commits, bun run typecheck passes before a PR.

License

Apache 2.0. See LICENSE.