Skip to content

feat(benchmark): add generic evaluation and comparison harness - #3640

Open
wuyuanfr wants to merge 23 commits into
ModelEngine-Group:developfrom
liudfgoo:feature/benchmark-observability-new
Open

feat(benchmark): add generic evaluation and comparison harness#3640
wuyuanfr wants to merge 23 commits into
ModelEngine-Group:developfrom
liudfgoo:feature/benchmark-observability-new

Conversation

@wuyuanfr

@wuyuanfr wuyuanfr commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

不会更改当前开发分支中 SDK/Nexent/Core 相关的内容,不更改Nexent 前后端系统服务。通过后端执行或者部署webhook服务后导入结果到Langfuse等Open Telemetry平台观测traces和实验结果。

  • 主代码在sdk/benchmark/generic下,支持导入通用的agent评测集,包括不限于开源公开benchmark或者自定义数据。
  • 评测指标支持 EM、F1、ACC等基础短答案QA评分器;支持缓存命中率、压缩节省tokens等多种指标观测,具体见markdown文档。

Summary

  • add a generic Langfuse-backed benchmark runner with evaluator, manifest, integrity, web-evidence, and replay support
  • add paired P/C and cross-worktree Legacy/P comparison runners with score synchronization and report generation
  • relocate Langfuse infrastructure and ctx_debugger under sdk/benchmark
  • add standard benchmark configs, bilingual documentation, and benchmark regression tests
  • remove the unused production parity snapshot public entrypoint and exporter

Scope

The final diff against develop is limited to benchmark code/tests and the expected sdk/ctx_debugger relocation cleanup. It does not change sdk/nexent/core relative to the current develop branch.

Validation

  • test/sdk/benchmark: 122 passed
  • generic benchmark and test Ruff: passed
  • ctx_debugger and LongMemEval Ruff: passed
  • git diff --check: passed

Evidence

  • gms8k benchmark-run screenshot
image
  • Legacy/P comparison screenshot
    GAIA数据集实验结果显示上下文管理相比Legacy 旧版本代码提示整体提升准确率
image
  • P/C comparison and compression screenshot

实验结果显示上下文管理提升adaptive compact策略提升缓存命中率
image

Comment thread sdk/benchmark/generic/tools/export_agent_config.py Fixed
Comment thread sdk/benchmark/generic/integrations/langfuse/webhook_server.py Fixed
@wuyuanfr
wuyuanfr marked this pull request as ready for review August 13, 2026 06:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants