Skip to content

fix(benchmark): keep key_map columns aligned when a record is incomplete - #512

Merged
xhd0728 merged 1 commit into
OpenBMB:mainfrom
chiruu12:fix/benchmark-key-alignment
Sep 16, 2026
Merged

xhd0728 merged 1 commit into
OpenBMB:mainfrom
chiruu12:fix/benchmark-key-alignment

Conversation

@chiruu12

Copy link
Copy Markdown
Contributor

Fixes #511.

_load_from_local filtered each output column on its own key. A record missing one mapped key was dropped from that column and kept in the others, so the columns ran off by one. compute_metrics pairs them by position, which means questions were scored against another row's ground truth, and the trailing one was dropped by zip. With shuffle: true the same data raised IndexError instead, because the shuffle branch indexes every column with the first column's range.

This decides which records are usable once, across all the keys in key_map, and builds every column from that list. The number of skipped records is logged, so a malformed file is visible rather than showing up as a slightly lower score. Well-formed data loads exactly as before.

Tests are under tests/servers/benchmark/. They lift _load_from_local out with ast, the way tests/servers/evaluation does, because importing the module pulls in the MCP app and pandas. Four of the seven fail on main: the misalignment, the unequal column lengths, the missing warning, and the IndexError. The other three cover behaviour that must not change and pass either way.

Copilot AI lite review requested due to automatic review settings September 15, 2026 08:50

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@xhd0728

xhd0728 commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the fix. I verified that the issue is reproducible on main: incomplete records can misalign benchmark columns, and shuffling can raise an IndexError.

The change correctly filters records once across all required key_map fields, keeping every output column aligned. All seven regression tests pass, and CI is green. This looks good to merge.

@xhd0728
xhd0728 merged commit 3a709a2 into OpenBMB:main Sep 16, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Benchmark key_map drops a record from one column only, misaligning questions and ground truths

3 participants