fix(benchmark): keep key_map columns aligned when a record is incomplete - #512
Merged
Merged
Conversation
xhd0728
approved these changes
Sep 16, 2026
Collaborator
|
Thanks for the fix. I verified that the issue is reproducible on The change correctly filters records once across all required |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #511.
_load_from_localfiltered each output column on its own key. A record missing one mapped key was dropped from that column and kept in the others, so the columns ran off by one.compute_metricspairs them by position, which means questions were scored against another row's ground truth, and the trailing one was dropped byzip. Withshuffle: truethe same data raisedIndexErrorinstead, because the shuffle branch indexes every column with the first column's range.This decides which records are usable once, across all the keys in
key_map, and builds every column from that list. The number of skipped records is logged, so a malformed file is visible rather than showing up as a slightly lower score. Well-formed data loads exactly as before.Tests are under
tests/servers/benchmark/. They lift_load_from_localout withast, the waytests/servers/evaluationdoes, because importing the module pulls in the MCP app and pandas. Four of the seven fail on main: the misalignment, the unequal column lengths, the missing warning, and theIndexError. The other three cover behaviour that must not change and pass either way.