feat: add OrcaRouter as a first-class provider in the LiteLLM backend

[kuswardhanietidims-svg]

Add OrcaRouter as a first-class provider in the LiteLLM backend

Lighteval's lighteval endpoint litellm backend already treats a few gateways as named providers (OpenRouter gets special handling in litellm_model.py for context-length estimation). This PR adds OrcaRouter the same way: a first-class orcarouter provider, instead of making users point an anonymous base_url at it.

OrcaRouter is an OpenAI-compatible AI gateway built for both models and agents. Like OpenRouter, it exposes a provider/model namespace across many models — but it also combines adaptive routing, automatic failover, zero-markup inference, observability, guardrails, and agent-tool governance behind the same endpoint. Adding orcarouter as a first-class provider means this project's users can use that stack directly, without treating OrcaRouter as an anonymous custom base URL. It also runs gateway-level, zero-trust security for AI agents on the same endpoint — screening every prompt/response and governing every tool call on a default-deny basis, with no application code changes.

Changes

• src/lighteval/models/endpoints/litellm_model.py
  • LiteLLMModelConfig + LiteLLMClient: when provider="orcarouter" and no base_url is given, default to https://api.orcarouter.ai/v1 (mirrors how OpenRouter is wired through the same backend).
  • __call_api: route OrcaRouter completions through litellm as an OpenAI-compatible endpoint (custom_llm_provider="openai"), keeping the orcarouter/... model prefix for model selection.
  • _estimate_orcarouter_context_length: fetch the model context length from OrcaRouter's /models/{id} API, mirroring the existing OpenRouter context-length estimation.
  • max_length: use the OrcaRouter estimator when provider="orcarouter".
• examples/model_configs/orcarouter_litellm_model.yaml: example config showing the orcarouter provider in use.

Validation

• ruff format --check . and ruff check . pass on the whole repo.
• Live-tested against the real OrcaRouter API using the new provider path (provider=orcarouter,model_name=orcarouter/fusion-flash): chat completion returned a normal response and the context-length lookup returned 200000.

Usage

lighteval endpoint litellm "provider=orcarouter,model_name=orcarouter/fusion-flash" gsm8k

---

I'm an engineer on the OrcaRouter team.

Discord: discord.gg/YEubt8enRA · X: https://x.com/OrcaRouter