Integrate a SPACE benchmark task into the agent-eval framework

[dimos-bot]

Mirror of GitHub issue #3399 (by CC).

Context

DimOS PR #3378 introduces the initial agent-evaluation framework and a direct frozen-memory QA case. This follow-up uses that framework to evaluate the DimOS agent on an existing external benchmark.

SPACE (Spatial Perception And Cognition Evaluation) tests spatial reasoning through text, images, and interactive tasks — both large-scale environmental reasoning and small-scale object/layout reasoning.

Objective

Integrate one representative SPACE task into the framework from PR #3378 and run the DimOS agent on a bounded, reproducible subset of that task.

Scope

• Build on the contracts and execution path introduced by PR feat: add frozen recording agent evaluation #3378
• Select one SPACE task that can be integrated and run end to end within bounded effort
• Preserve the benchmark question, input, answer, and scoring semantics
• Use SPACE official evaluation and scoring implementation as-is
• Provide a direct command that another developer can run and test
• Run a fixed subset and summarize accuracy, invalid responses, and infrastructure failures
• Produce an evaluation report analyzing agent performance

Deliverables

1. Benchmark/task selection note with reasoning and known limitations
2. Working integration using the PR feat: add frozen recording agent evaluation #3378 framework
3. Reproducible setup and run instructions
4. Evaluation report covering results, strengths, failure modes, and improvement directions
5. Focused automated tests

Data and licensing

SPACE code: Apple Sample Code License. Dataset: CC BY-NC-ND 4.0. Must not vendor/modify/redistribute without confirming compliance.

References

• Paper
• SPACE repo
• Evaluation framework PR #3378
• GitHub issue #3399

---

Synced from DIM-1412