Feature idea: EvalPort adapter for portable Agent-Diff Bench suites & results

[adhabnr-ux]

Summary

Agent-Diff Bench has a genuinely portable eval data model — declarative, diff-based assertions (added/removed/changed predicates over Postgres inserts/updates/deletes) that don't depend on your Docker/Postgres runtime to be meaningful. Right now that model is legible only inside this repo's own schema (examples/<service>/testsuites/*.json + backend/src/platform/evaluationEngine/dsl_schema.json). I think it's worth a standalone adapter to/from EvalPort — an Apache-2.0 JSON interchange format for eval datasets and results (spec: spec/SPEC.md) — so a slack_bench_v2.json-shaped suite becomes readable by any EvalPort-aware tool, and a run's {passed, score, failures} output becomes a portable ResultSet other harnesses can ingest, diff, or plot.

Precedent for this exact shape of contribution: autogen-openeval-adapter, crewai-openeval-adapter, and opik-openeval-adapter are all standalone pip packages built purely against each project's existing public data shapes — zero changes to the host repo. Same playbook here: an agent-diff-openeval-adapter package that reads/writes examples/**/testsuites/*.json and evaluator output; nothing in backend/ or sdk/ needs to change.

Concrete field mapping

Your test suite (per docs/test-suite-extension-guide.md) —

{
  "id": "test_12",
  "name": "Post a hello message",
  "prompt": "Send 'hello' to #general",
  "type": "actionEval",
  "seed_template": "slack_bench_v2",
  "impersonate_user_id": "U01AGENBOT9",
  "assertions": [
    {
      "diff_type": "added",
      "entity": "messages",
      "where": {"channel_id": {"eq": "C01ABCD1234"}, "message_text": {"contains": "hello"}},
      "expected_count": 1
    }
  ]
}

— maps onto EvalPort's TestCase + Grader like this. Your assertions don't fit any of EvalPort's 11 "well-known" grader types (exact_match, contains, llm_judge, etc.) — they're structural diff predicates — so this uses the spec's typed escape hatch: a non-well-known type string is valid as long as params.handler is set, so a runner that doesn't understand "agent_diff.diff_assertion" skips it gracefully instead of guessing at its semantics (spec §"Grader Type System" / "Type openness"):

{
  "id": "tc_test_12",
  "input": "Send 'hello' to #general",
  "graders": ["gr_test_12_a0"],
  "metadata": {
    "agent_diff": {
      "service": "slack",
      "seed_template": "slack_bench_v2",
      "impersonate_user_id": "U01AGENBOT9",
      "type": "actionEval"
    }
  }
}

{
  "id": "gr_test_12_a0",
  "type": "agent_diff.diff_assertion",
  "params": {
    "handler": "agent_diff.assertion_engine",
    "diff_type": "added",
    "entity": "messages",
    "where": {"channel_id": {"eq": "C01ABCD1234"}, "message_text": {"contains": "hello"}},
    "expected_count": 1
  }
}

And a run's output maps onto ResultSet — one GraderResult per assertion, Result.passed = your top-level passed, summary.pass_rate = score.percent / 100. I read AssertionEngine.evaluate (backend/src/platform/evaluationEngine/assertion.py) to get this part right: failures there is a flat list[str], each message prefixed "assertion#{idx} ..." with idx 1-based (enumerate(assertions_list, start=1)) — there's no structured {index, message} object, even though the engine already tracks a failed_indexes: set[int] internally, it's just never returned. So the adapter below recovers per-grader pass/fail by regexing that prefix, which works but is one field-parse away from being exact.

Adapter sketch

# agent_diff_openeval_adapter/__init__.py
import re

def suite_to_openeval(suite: dict) -> dict:
    """examples/<service>/testsuites/*.json -> EvalPort EvalSuite."""
    graders, test_cases = [], []
    for test in suite["tests"]:
        assertions = test.get("assertions") or test.get("expected_output", {}).get("assertions", [])
        grader_ids = []
        for i, a in enumerate(assertions):
            gid = f"gr_{test['id']}_a{i}"
            grader_ids.append(gid)
            graders.append({
                "id": gid,
                "type": "agent_diff.diff_assertion",
                "params": {"handler": "agent_diff.assertion_engine", **a},
            })
        test_cases.append({
            "id": f"tc_{test['id']}",
            "input": test["prompt"],
            "graders": grader_ids,
            "metadata": {"agent_diff": {
                "service": suite.get("service"),
                "seed_template": test["seed_template"],
                "impersonate_user_id": test["impersonate_user_id"],
                "type": test.get("type"),
            }},
        })
    return {
        "$schema": "https://evalport.org/schema/suite.json",
        "version": "1.0.0",
        "id": suite["id"],
        "name": suite.get("name"),
        "description": suite.get("description"),
        "graders": graders,
        "test_cases": test_cases,
    }

def run_result_to_openeval(suite_id: str, run_id: str, test_id: str, eval_result: dict, started_at: str) -> dict:
    """AssertionEngine.evaluate() output -> EvalPort ResultSet with one Result.

    eval_result is exactly {"passed": bool, "failures": list[str], "score": {"passed", "total", "percent"}}
    as returned by AssertionEngine.evaluate(). Each failures[] entry starts "assertion#{N} "
    with N 1-based; grader ids below were built 0-based in suite_to_openeval, hence the +1.
    """
    total = eval_result["score"]["total"]
    failed_idx = {
        int(m.group(1))
        for f in eval_result.get("failures", [])
        if (m := re.match(r"assertion#(\d+) ", f))
    }
    grader_results = [
        {
            "grader_id": f"gr_{test_id}_a{i}",
            "type": "agent_diff.diff_assertion",
            "score": 0.0 if (i + 1) in failed_idx else 1.0,
            "passed": (i + 1) not in failed_idx,
            "reason": next(
                (f for f in eval_result.get("failures", []) if f.startswith(f"assertion#{i+1} ")),
                None,
            ),
        }
        for i in range(total)
    ]
    return {
        "$schema": "https://evalport.org/schema/resultset.json",
        "version": "1.0.0",
        "suite_id": suite_id,
        "run_id": run_id,
        "started_at": started_at,
        "results": [{
            "test_case_id": f"tc_{test_id}",
            "grader_results": grader_results,
            "passed": eval_result["passed"],
        }],
        "summary": {
            "total": total,
            "passed": eval_result["score"]["passed"],
            "pass_rate": eval_result["score"]["percent"] / 100,
        },
    }

If it's welcome, returning the already-computed failed_indexes alongside failures in AssertionEngine.evaluate's output (backend/src/platform/evaluationEngine/assertion.py) would let this — and any other structured consumer of run results — drop the regex entirely. Happy to send that as its own tiny PR regardless of what happens with the adapter.

Why bother

The 224-task Agent-Diff Bench dataset is already on the Hub as flat JSONL/parquet — this doesn't compete with that, it's a second, structured export path. Teams already standardized on EvalPort-consuming tooling (DeepEval, Promptfoo, Ragas, LangSmith, MLflow, Opik, and others have adapters/native support) could pull your suites in and push graded results back out without hand-rolling a converter, while your own DSL and assertion engine stay exactly as they are.

I'd build this as a standalone package under adapters/agent-diff-openeval-adapter in the EvalPort repo (same as the precedents above), not asking for anything to land here — just wanted to check it's actually useful to you first, and flag the failures[] format finding above in case it's a quick fix regardless.

— Sahi, independent contributor (not affiliated with this project)