All pull requests

Pull requests

• fix(arena): mark over-length samples as candidate losses

  #1534·rwinkelman opened Sep 28, 2026·

  ·
  1

  2

• fix: raise a clear error on empty eval group config files

  #1532·Ashfaqbs opened Sep 16, 2026·

  ·
  1

  2

• Apply BFCL Java/JavaScript preprocessing to simple_java and simple_javascript categories

  #1528·azrabano23 opened Sep 4, 2026·

  ·
  1

  1

• Restore boxed-first default for relaxed answer extraction, add opt-in regex_first

  #1527·azrabano23 opened Sep 4, 2026·

  ·
  1

  4

• Prefer explicit "Answer: X" line over letters inside boxed expressions in MCQ eval

  #1526·azrabano23 opened Sep 4, 2026·

  ·
  1

  3

• Add GenCorrect recipe

  #1525·SeanNaren opened Sep 4, 2026Collaborator·

  ·
  4

• docs: point Papers & Releases link at docs/releases/index.md

  #1522·MohammedAlkindi opened Aug 31, 2026·

  ·
  1

• Update SHERLOC teaser figure

  #1521·tamohannes opened Aug 28, 2026Collaborator·

  ·
  

• fix: group majority@k and reward@k by answer string (#1257)

  #1519·FazeelUsmani opened Aug 25, 2026·

  ·
  1

  1

• senior-swe-bench benchmark evaluation integration

  #1515·wasiahmad opened Aug 14, 2026Collaborator·

  ·
  11

• swe-atlas-qna benchmark evaluation integration

  #1513·wasiahmad opened Jul 30, 2026Collaborator·

  ·
  11

• CodeRabbit Generated Unit Tests: Add unit tests for PR changes

  #1508·coderabbitai[bot] opened Jul 9, 2026Bot·

  ·
  

• fix(security): floor litellm/wandb/lxml past known CVEs; relax stale click cap

  #1507·nicgupta-nvidia opened Jul 9, 2026Collaborator·

  ·
  9

  

• High-throughput async inference: native aiohttp fast-path, scalable async loop, and request-routing keys

  #1502·gwarmstrong opened Jul 1, 2026Collaborator·

  ·
  17

• Quality: yaml.safe_load returns None for empty YAML files, propagating as None to callers expecting dict

  #1498·kumburovicbranko682-boop opened Jun 27, 2026·

  ·
  1

  3

• [WIP]Add CS-FLEURS code-switched ASR benchmark with Mixed Error Rate scoring

  #1494·Slyne opened Jun 22, 2026·

  ·
  13

• Bump litellm from 1.83.14 to 1.84.0 in /core

  dependenciespython
  #1492·dependabot[bot] opened Jun 20, 2026Bot·

  ·

• Add ntt-complex.format_ast benchmark scaffold

  #1491·vmendelev opened Jun 18, 2026Collaborator·

  ·

• Add HiL-Bench SWE support

  #1490·agarkovv opened Jun 15, 2026·

  ·
  32

• Standardize audio manifest container paths

  #1489·DongjiGao opened Jun 15, 2026Collaborator·

  ·
  1

• fix: honor skip_special_tokens from extra_body on the vLLM completion path

  #1488·DongjiGao opened Jun 15, 2026Collaborator·

  ·
  1

• Speed up local vLLM ASR client requests

  #1487·DongjiGao opened Jun 15, 2026Collaborator·

  ·
  6

• feat(speech): add AppTek Call-Center Dialogues ASR benchmark

  #1486·pzelasko opened Jun 15, 2026Collaborator·

  ·
  6

• fix(audio): harden AudioBench and MMAU evaluation

  #1485·pzelasko opened Jun 15, 2026Collaborator·

  ·
  1

• fix(pipeline): harden eval runtime plumbing

  #1484·pzelasko opened Jun 15, 2026Collaborator·

  ·
  4