Give tied objects the same rank - #357
Open
appleweiping wants to merge 1 commit into
Open
appleweiping wants to merge 1 commit into
appleweiping wants to merge 1 commit into
Conversation
Ranker assigned ranks by position in the sorted frame:
df[rank] = [idx + 1 for idx, _ in enumerate(df.index)]
so objects with identical scores received different ranks, and which one came
out ahead depended on the order pandas happened to produce. Ranking a corpus
where many objects share a score — a common case for engagement counts, vote
totals and similar integer metrics — therefore reported a precision the data
does not support, as reported in CornellNLP#320.
Both `transform` and `transform_objs` now derive the rank from the score with
`rank(ascending=False, method="min")`, standard competition ranking: tied
objects share the best rank they collectively occupy, and the next distinct
score resumes after the positions the tie consumed. Scores of 10, 10, 10, 5
now rank 1, 1, 1, 4 rather than 1, 2, 3, 4. Ranks stay integers, and objects
with distinct scores are unaffected.
Adds tests covering a tie under both methods and a distinct-score case to pin
the unchanged behaviour. Both tie tests fail against the current implementation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
|
Hi @appleweiping ! Thanks so much for looking into this! I'll review your PR within this coming week. |
Author
|
Focused validation is complete on the current head d5f816d.
The tied-rank behavior and its regression coverage are ready for review. The existing reviewer note from Sep 4 said a review would follow this week. Please take another look when available. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #320.
Rankerassigned ranks by position in the sorted frame:Objects with identical scores therefore received different ranks, and which one came out ahead depended on the order pandas happened to produce. For integer-valued metrics like engagement counts or vote totals, where ties are common, the output reported a precision the data does not support.
Both
transformandtransform_objsnow derive the rank from the score column withrank(ascending=False, method="min")— standard competition ranking. Tied objects share the best rank they collectively occupy, and the next distinct score resumes after the positions the tie consumed:Ranks remain integers, and objects with distinct scores are unaffected.
Testing
Adds
convokit/tests/ranker/test_ranker.pycovering a tie through bothtransformandtransform_objs, plus a distinct-score case pinning the unchanged behaviour. Verified against the released 4.1.2: the two tie tests fail (assert np.int64(1) == np.int64(2)) and pass with this change; the distinct-score test passes either way.convokit/tests/run_all_tests.pydiscovers all three.The new test directory includes an
__init__.py, matching the other test packages.🤖 Generated with Claude Code