Skip to content

Give tied objects the same rank - #357

Open
appleweiping wants to merge 1 commit into
CornellNLP:masterfrom
appleweiping:fix/320-ranker-ties
Open

appleweiping wants to merge 1 commit into
CornellNLP:masterfrom
appleweiping:fix/320-ranker-ties

Conversation

@appleweiping

Copy link
Copy Markdown

Fixes #320.

Ranker assigned ranks by position in the sorted frame:

df[self.rank_attribute_name] = [idx + 1 for idx, _ in enumerate(df.index)]

Objects with identical scores therefore received different ranks, and which one came out ahead depended on the order pandas happened to produce. For integer-valued metrics like engagement counts or vote totals, where ties are common, the output reported a precision the data does not support.

Both transform and transform_objs now derive the rank from the score column with rank(ascending=False, method="min") — standard competition ranking. Tied objects share the best rank they collectively occupy, and the next distinct score resumes after the positions the tie consumed:

scores before after
10, 10, 10, 5 1, 2, 3, 4 1, 1, 1, 4
3, 2, 1 1, 2, 3 1, 2, 3

Ranks remain integers, and objects with distinct scores are unaffected.

Testing

Adds convokit/tests/ranker/test_ranker.py covering a tie through both transform and transform_objs, plus a distinct-score case pinning the unchanged behaviour. Verified against the released 4.1.2: the two tie tests fail (assert np.int64(1) == np.int64(2)) and pass with this change; the distinct-score test passes either way. convokit/tests/run_all_tests.py discovers all three.

The new test directory includes an __init__.py, matching the other test packages.

🤖 Generated with Claude Code

Ranker assigned ranks by position in the sorted frame:

    df[rank] = [idx + 1 for idx, _ in enumerate(df.index)]

so objects with identical scores received different ranks, and which one came
out ahead depended on the order pandas happened to produce. Ranking a corpus
where many objects share a score — a common case for engagement counts, vote
totals and similar integer metrics — therefore reported a precision the data
does not support, as reported in CornellNLP#320.

Both `transform` and `transform_objs` now derive the rank from the score with
`rank(ascending=False, method="min")`, standard competition ranking: tied
objects share the best rank they collectively occupy, and the next distinct
score resumes after the positions the tie consumed. Scores of 10, 10, 10, 5
now rank 1, 1, 1, 4 rather than 1, 2, 3, 4. Ranks stay integers, and objects
with distinct scores are unaffected.

Adds tests covering a tie under both methods and a distinct-score case to pin
the unchanged behaviour. Both tie tests fail against the current implementation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@cristiandnm
cristiandnm requested a review from laerdon September 4, 2026 19:48
@laerdon

laerdon commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Hi @appleweiping ! Thanks so much for looking into this! I'll review your PR within this coming week.

@appleweiping

appleweiping commented Sep 7, 2026 •

Copy link
Copy Markdown
Author

Focused validation is complete on the current head d5f816d.

  • The ranker regression suite passed all 3 tests.
  • The repository-wide runner reached a database mode corpus test that requires a live MongoDB. I stopped it after confirming that the block was environmental, not a ranker assertion.

The tied-rank behavior and its regression coverage are ready for review. The existing reviewer note from Sep 4 said a review would follow this week. Please take another look when available.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ranker defaults to a ranking order given the same values

2 participants