POSIX tool surface and memory evals (SPEC-47 A/B, BEAM, xAFS)

[phernandez]

Goal

Implement the SPEC-47 POSIX tool surface behind a config flag, and build the eval infrastructure to measure whether it beats the existing rich MCP tools on token usage and accuracy — plus add external memory benchmarks (BEAM, xAFS) so those same evals measure the SPEC-82/SPEC-89 roadmap.

Why now

• SPEC-47 claims 80%+ token savings from POSIX-shaped partial reads; that claim has never been measured against our own tools.
• BEAM (ICLR 2026, arXiv:2510.27246) shows every memory system — long-context, RAG, LIGHT, and SOTA commercial (Exabase M-1 ~60%) — is weakest at contradiction resolution, with temporal reasoning and knowledge update close behind. Those are exactly SPEC-89 and SPEC-82. A per-ability BEAM baseline before those specs land turns them into measurable deltas.
• supermemory's xAFS (CC-BY-4.0) is a surface-agnostic agent-filesystem retrieval eval whose headline metric is tokens per correct answer — the exact question the POSIX A/B asks. Vendor-made, so verify before trusting, but the shape is right.
• Their MemoryBench staging (INGEST → SEARCH → ANSWER → EVALUATE → REPORT, checkpointed) matches our benchmarks/ package stages; prior art, not a dependency.

Design decisions (settled)

1. POSIX tools are off by default, enabled by an enable_posix_tools config flag gated at the MCP composition root. Existing tools unchanged.
2. v1 is read-side only: cat, grep, ls, find, tail, man — thin translations over existing API endpoints per SPEC-47's table. Write verbs wait.
3. v1 ships without section indexing and gets measured anyway; section reads (SPEC-47 component 1) land second so the token delta is a measured result, not an assumption.
4. Eval fairness follows the existing benchmarks/ contract: same tasks, same model, same budget — only the tool surface varies (rich-tools vs posix-tools as the provider axis).
5. BEAM starts at the 100K/500K tiers with raw transcript-as-notes ingestion (cheap baseline) before agent-curated ingestion (the real product story). Expect the raw baseline to be unflattering; the curated delta is the result.

Sub-issues

• POSIX read-side MCP tools behind enable_posix_tools
• BEAM integration in the benchmarks package
• Agent-task eval: rich vs POSIX tool surfaces
• xAFS dataset adapter
• Section indexing and range reads (SPEC-47 component 1)
• bm CLI POSIX verbs

References

• SPEC-47: AGENT-FILESYSTEM (specs project) — tool shapes, flags, S3-facade cut rationale
• SPEC-82 (temporal), SPEC-89 (claims/contradictions) — the abilities BEAM measures externally
• BEAM code/data: https://github.com/mohammadtavakoli78/BEAM (paper's runner); Exabase forked Hindsight's open runner — evaluate both before building ours
• mem0 on BEAM: https://mem0.ai/blog/what-is-beam-memory-benchmark-the-paper-that-shows-1m-context-window-isnt-enough
• LoCoMo cross-system leaderboard: https://benchmarks.mnemoverse.com/
• Existing harness: benchmarks/ package (LoCoMo cats 1–4 headline, fairness contract, retrieval/qa/review/diagnose/rejudge stages)

🤖 Generated with Claude Code

https://claude.ai/code/session_014pmKq6bqCi6Zp6BTHuZjrp

[claude]

Claude finished @phernandez's task in 49s —— View job

---

Triage

Type: Enhancement — this is a tracking/epic issue for new functionality (POSIX-shaped MCP tool surface behind a flag, plus eval infrastructure), not a defect report.

Component: No dedicated repo label applies. In prose, this spans MCP tools (src/basic_memory/mcp/), the benchmarks/ eval package, and (per sub-issue #1404) the CLI. Not Cloud-specific.

Priority: Medium — roadmap/research work with no user-facing breakage or urgency; it's explicitly gated behind an off-by-default enable_posix_tools flag, so existing tools are unaffected until the eval results justify further investment.

Complexity: Complex — a multi-part epic spanning a new MCP tool surface, two external benchmark integrations (BEAM, xAFS), a rich-vs-POSIX eval harness, and follow-on CLI/section-indexing work.

Note on scope: This issue's checklist enumerates six sub-tasks that already exist as separate open issues in this repo, each matching a checklist item verbatim:

• POSIX read-side MCP tools behind enable_posix_tools flag #1399 — POSIX read-side MCP tools behind enable_posix_tools flag
• BEAM benchmark integration in the benchmarks package #1400 — BEAM benchmark integration in the benchmarks package
• Agent-task eval: rich vs POSIX tool surfaces #1401 — Agent-task eval: rich vs POSIX tool surfaces
• xAFS dataset adapter #1402 — xAFS dataset adapter
• Section indexing and range reads (SPEC-47 component 1) #1403 — Section indexing and range reads (SPEC-47 component 1)
• bm CLI POSIX verbs #1404 — bm CLI POSIX verbs

This issue appears to function as the parent/tracking issue for that set rather than a duplicate of any single one — worth confirming with @phernandez that #1398 is meant to stay open as the umbrella and the linked issues are where implementation work should land.