diff --git a/CHANGELOG.md b/CHANGELOG.md index 0e987ef86..54a2349b7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,11 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Added +- **`timeout_ms` on `[llm_clients.]`** — one deadline covers all attempts, + retry delays, and the complete response, including stream reads. Unset leaves + the wait unbounded; `0` is rejected. A timeout returns `504` without trying another + model, or a framed error if the final answer has already started streaming. + Timed-out attempts are counted in metrics. - **Per-target `reasoning_effort`** — a target can force the reasoning effort of every request it serves, replacing the caller's value (`reasoning.effort` on the Responses wire, `reasoning_effort` on Chat Completions), so a strong @@ -89,6 +94,11 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Changed +- **HTTP client errors stop routing** — after the configured retries, the Rust + runner stops the request instead of letting the routing algorithm choose a + fallback. This also applies when `timeout_ms` is unset or an advisor has + `fail_open = true`. The runner collects streams used during routing and preserves + provider events for replay. Invalid judge verdicts keep their existing fallback. - **`Algorithm::route` returns `Result`** — instead of the bare final `Result`, so callers observe the full routing outcome (see #458 for the design). (#459) diff --git a/crates/libsy-llm-client/README.md b/crates/libsy-llm-client/README.md index bad3f86d0..bb3b38caf 100644 --- a/crates/libsy-llm-client/README.md +++ b/crates/libsy-llm-client/README.md @@ -36,8 +36,8 @@ It depends on `switchyard-libsy`, `switchyard-protocol`, and `anthropic-version`). - **Model rewrite.** The resolved [`ModelId`] is both the map key and the model id sent upstream — it overwrites whatever `model` the request arrived with. -- **Streaming is chosen by the request.** If the encoded body has `stream: true` - (i.e. `request.llm_request.stream`), you get `LlmResponse::Stream`; otherwise +- **Streaming is chosen by the encoded request body.** If the body has + `stream: true` after `extra_body` is applied, you get `LlmResponse::Stream`; otherwise `LlmResponse::Agg`. OpenAI Chat streaming requests default `stream_options.include_usage` to `true`; an explicit caller value is preserved. @@ -69,7 +69,9 @@ fn build_client() -> switchyard_llm_client::Result { forward_auth: false, extra_headers: BTreeMap::new(), extra_body: BTreeMap::new(), + reasoning_effort: None, max_retries: 2, + timeout: None, }; let models = [ModelConfig::new( @@ -247,13 +249,23 @@ fn build_multi_format_client( transport failures, timeouts, HTTP 408/429, and 5xx responses. Buffered body transport failures are retried; streaming body failures are not replayed after the response has been returned. - -Retries replay the same upstream request to the same model. Each candidate's -`max_retries` budget is exhausted before candidate fallback advances to the next -model. The worst case is `candidates × (max_retries + 1)` upstream requests, and -total latency includes every candidate's capped `Retry-After` backoff. A transport -failure can duplicate a request that the provider processed but did not finish -returning. +- `HttpBackendConfig::timeout` bounds one complete response, including retries, + retry delays, and every stream read. Expiry returns `LlmClientError::Timeout`, + either from the call or from the returned stream, which then ends. `None` leaves + the wait unbounded. + +`run` and `decide` collect streams used during routing, including answers that an +algorithm must inspect, before returning them to the algorithm. They retain the +provider events for replay. After the configured retries, a client failure stops +`run` or `decide` before the algorithm can choose another routing candidate. This +also applies when `timeout` is `None` or an advisor has `fail_open = true`. +`libsy` and custom hosts that drive it directly are unchanged. + +Retries replay the same upstream request to the same model. After routing completes, +non-timeout failures may try another completion candidate. A timeout stops the call. +A transport failure can duplicate a request that the provider processed but did not +finish returning. Each attempt is counted once; expiry during a retry delay or +after a stream has started does not count another attempt. ## Errors diff --git a/crates/libsy-llm-client/src/backend.rs b/crates/libsy-llm-client/src/backend.rs index da43e2c63..a96a3e1fa 100644 --- a/crates/libsy-llm-client/src/backend.rs +++ b/crates/libsy-llm-client/src/backend.rs @@ -3,6 +3,7 @@ //! Per-provider backend configuration: wire format, upstream URL, and auth. +use std::time::Duration; use std::{collections::BTreeMap, fmt}; use reqwest::RequestBuilder; @@ -62,6 +63,9 @@ pub struct HttpBackendConfig { pub reasoning_effort: Option, /// Additional attempts after the initial upstream request. pub max_retries: u32, + /// Deadline for one complete response, including retries, retry delays, and stream reads. + /// `None` leaves the wait unbounded. + pub timeout: Option, } impl fmt::Debug for HttpBackendConfig { @@ -74,6 +78,7 @@ impl fmt::Debug for HttpBackendConfig { .field("extra_body_keys", &self.extra_body.keys()) .field("reasoning_effort", &self.reasoning_effort) .field("max_retries", &self.max_retries) + .field("timeout", &self.timeout) .finish() } } @@ -256,6 +261,11 @@ impl Backend { self.config().max_retries } + /// Deadline for all attempts and the complete response; `None` leaves the wait unbounded. + pub fn timeout(&self) -> Option { + self.config().timeout + } + /// Whether this backend speaks the Anthropic Messages wire format — the only /// one with a `count_tokens` endpoint. pub fn is_anthropic(&self) -> bool { @@ -348,6 +358,7 @@ mod tests { extra_body: BTreeMap::new(), reasoning_effort: None, max_retries: 0, + timeout: None, } } diff --git a/crates/libsy-llm-client/src/client.rs b/crates/libsy-llm-client/src/client.rs index f9580a537..dbb69b2ca 100644 --- a/crates/libsy-llm-client/src/client.rs +++ b/crates/libsy-llm-client/src/client.rs @@ -224,8 +224,9 @@ impl TranslatingLlmClient { /// forwarded headers plus the backend's static headers and auth, and return the /// successful upstream response. A /// buffered response is fully collected within the retry boundary; a streamed - /// response has its first decoded event checked within that boundary. A non-success - /// status maps to a typed error — a 400 is classified as a context-window + /// response has its first decoded event checked within that boundary. The backend's + /// deadline, when set, spans every attempt, retry delay, and subsequent stream read. A + /// non-success status maps to a typed error — a 400 is classified as a context-window /// overflow via the backend's provider rules. Shared by /// [`call_rewrite_model`](Self::call_rewrite_model) (which POSTs to the /// backend's completion URL and decodes a response) and @@ -269,15 +270,47 @@ impl TranslatingLlmClient { let url = endpoint.url(backend); record_gen_ai_request(&url, model, streaming); + self.send_with_retries(&url, backend, &body, metadata, model, streaming) + .await + } + + // Sends the encoded body, retrying retryable failures within the backend's retry budget. + async fn send_with_retries( + &self, + url: &str, + backend: &Backend, + body: &Value, + metadata: Option<&Metadata>, + model: &ModelId, + streaming: bool, + ) -> Result { let max_retries = u64::from(backend.max_retries()); let max_attempts = max_retries + 1; + let expires_at = backend + .timeout() + .map(|timeout| (tokio::time::sleep(timeout).deadline(), timeout)); + let deadline = async { + match expires_at { + Some((at, timeout)) => { + tokio::time::sleep_until(at).await; + timeout + } + None => std::future::pending().await, + } + }; + tokio::pin!(deadline); let mut attempt = 0_u64; loop { + if let Some((at, timeout)) = expires_at + && tokio::time::Instant::now() >= at + { + return Err(deadline_error(timeout)); + } let span = tracing::debug_span!( target: "libsy", "libsy.upstream_attempt", model = %model, - wire_format = %wire_format, + wire_format = %backend.wire_format(), attempt = attempt + 1, max_attempts, retry = attempt > 0, @@ -287,10 +320,22 @@ impl TranslatingLlmClient { will_retry = tracing::field::Empty, retry_delay_ms = tracing::field::Empty, ); - let result = self - .send_once(&url, backend, &body, metadata, model, streaming) - .instrument(span.clone()) - .await; + let mut attempt_started = false; + let result = tokio::select! { + biased; + timeout = &mut deadline => { + if attempt_started { + metrics::record_upstream_attempt(None); + } + span.record("outcome", "error"); + span.record("will_retry", false); + return Err(deadline_error(timeout)); + } + result = async { + attempt_started = true; + self.send_once(url, backend, body, metadata, model, streaming).await + }.instrument(span.clone()) => result, + }; // The retained handle updates this same attempt span with its outcome. match result { Ok(response) => { @@ -300,7 +345,18 @@ impl TranslatingLlmClient { if attempt > 0 { metrics::record_retry_recovered(); } - return Ok(response); + return Ok(match response { + EncodedResponse::Streaming { + status, + chunks, + upstream_headers, + } => EncodedResponse::Streaming { + status, + chunks: stream_with_deadline(chunks, expires_at), + upstream_headers, + }, + buffered => buffered, + }); } Err(failure) => { let will_retry = attempt < max_retries && failure.is_retryable(); @@ -317,7 +373,11 @@ impl TranslatingLlmClient { span.record("retry_delay_ms", duration_millis(delay)); // Close the attempt span before sleeping so backoff is not attempt latency. drop(span); - tokio::time::sleep(delay).await; + tokio::select! { + biased; + timeout = &mut deadline => return Err(deadline_error(timeout)), + _ = tokio::time::sleep(delay) => {} + } attempt += 1; } } @@ -653,6 +713,18 @@ struct AttemptFailure { retry_after: Option, } +fn deadline_error(timeout: Duration) -> LlmClientError { + LlmClientError::Timeout { + source: Box::new(std::io::Error::new( + std::io::ErrorKind::TimedOut, + format!( + "response did not finish within {} ms, retries included", + duration_millis(timeout) + ), + )), + } +} + impl AttemptFailure { fn is_retryable(&self) -> bool { match &self.error { @@ -665,6 +737,26 @@ impl AttemptFailure { } } +fn stream_with_deadline( + chunks: LlmResponseStream, + deadline: Option<(tokio::time::Instant, Duration)>, +) -> LlmResponseStream { + let Some((at, timeout)) = deadline else { + return chunks; + }; + stream::unfold(Some(chunks), move |chunks| async move { + let mut chunks = chunks?; + let chunk = tokio::select! { + biased; + _ = tokio::time::sleep_until(at) => return Some((Err(deadline_error(timeout)), None)), + chunk = chunks.next() => chunk?, + }; + let next = chunk.is_ok().then_some(chunks); + Some((chunk, next)) + }) + .boxed() +} + async fn prepare_response_stream( response: reqwest::Response, backend: &Backend, @@ -1199,6 +1291,7 @@ mod tests { extra_body: BTreeMap::new(), reasoning_effort: None, max_retries: 0, + timeout: None, } } @@ -2212,6 +2305,52 @@ mod tests { Ok(()) } + #[tokio::test] + async fn deadline_expires_before_send_or_stream_poll() + -> std::result::Result<(), Box> { + let server = MockServer::start().await; + Mock::given(method("POST")) + .respond_with(ResponseTemplate::new(200).set_body_raw( + "data: {\"id\":\"test\",\"model\":\"gpt\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"ok\"},\"finish_reason\":\"stop\"}]}\n\ndata: [DONE]\n\n", + "text/event-stream", + )) + .mount(&server) + .await; + for timeout in [Duration::ZERO, Duration::from_millis(200)] { + let mut backend = config(&server.uri()); + backend.timeout = Some(timeout); + let client = TranslatingLlmClient::new(&[ModelConfig::new( + "gpt", + Backend::OpenAiChat(backend), + None, + )])?; + let result = client + .call_rewrite_model(request_for(Some("gpt"), true), None) + .await; + if timeout.is_zero() { + assert!(matches!(result, Err(LlmClientError::Timeout { .. }))); + assert!( + server + .received_requests() + .await + .expect("recorded requests") + .is_empty() + ); + } else { + let LlmResponse::Stream(mut chunks) = result?.llm_response else { + return Err("expected a streaming response".into()); + }; + tokio::time::sleep(Duration::from_millis(300)).await; + assert!(matches!( + chunks.next().await, + Some(Err(LlmClientError::Timeout { .. })) + )); + assert!(chunks.next().await.is_none()); + } + } + Ok(()) + } + #[tokio::test] async fn timeout_is_retried_before_a_response_is_returned() -> std::result::Result<(), Box> { diff --git a/crates/libsy-llm-client/src/run.rs b/crates/libsy-llm-client/src/run.rs index c16922c44..2326c951c 100644 --- a/crates/libsy-llm-client/src/run.rs +++ b/crates/libsy-llm-client/src/run.rs @@ -10,8 +10,8 @@ //! //! libsy owns the stream mechanics; what this module adds is per-target request preparation, //! ordered candidate fallback, and the `libsy.client_call` span around each candidate. Each -//! candidate exhausts its backend retry budget before fallback advances, so the worst case is -//! `candidates × (max_retries + 1)` upstream attempts plus every candidate's backoff. +//! completion candidate exhausts its backend retry budget before fallback advances. Routing +//! calls stop on the first candidate's failure. A timeout stops either kind of call. //! //! A Responses continuation can refer to state held by one provider through //! `previous_response_id` or `conversation`. When completion targets use different clients, @@ -22,7 +22,7 @@ use std::collections::HashMap; use std::sync::Arc; use std::time::Instant; -use futures_util::StreamExt; +use futures::{StreamExt, TryStreamExt, stream}; use http::StatusCode; use parking_lot::Mutex; use serde_json::{Value, json}; @@ -30,8 +30,8 @@ use switchyard_libsy::{ Algorithm, CallModel, LibsyError, OutcomeMetadata, Result, RoutingOutcome, RuntimeModels, drive, }; use switchyard_protocol::{ - LlmClientError, LlmResponse, ModelId, Request, Response, RoutedLlmClient, - RoutingFallbackReason, WireFormat, + LlmClientError, LlmResponse, LlmResponseChunk, LlmResponseStream, ModelId, Request, Response, + RoutedLlmClient, RoutingFallbackReason, WireFormat, }; use switchyard_translation::prepare_request_for_target; @@ -48,8 +48,8 @@ use crate::{metrics, observability}; /// a per-call lookup, not one client for the whole run. Use /// [`ClientRouter::single`](ClientRouter::single) when one client serves every target. /// -/// Routing-time model failures are forwarded back into the algorithm. Once routing completes, -/// this client exhausts backend retries and then the outcome's ordered candidate fallbacks. +/// Routing calls are buffered; a client failure stops the request before the algorithm continues. +/// Once routing completes, non-timeout failures may try the outcome's ordered fallback candidates. pub async fn run( algorithm: Arc, clients: ClientRouter, @@ -98,7 +98,6 @@ pub async fn run( &algorithm_name, &outcome.request, &outcome.selected_model_ids, - CallPhase::Completion, &observe, ) .await; @@ -171,8 +170,7 @@ fn emit_routing_observations( /// Serve one offloaded call and fulfill its promise. /// -/// Errors only when the promise itself could not be fulfilled; a call that failed on every -/// candidate is forwarded to the algorithm as an `Err`. +/// A client failure stops the driver before the algorithm can issue another call. async fn serve( clients: ClientRouter, call: CallModel, @@ -183,21 +181,20 @@ async fn serve( observations.lock().push(observation); } }; - let result = call_first_available( + let target = call.models.first().ok_or(LibsyError::NoTargets)?; + let request = clients.prepare_routing_request(call.request.clone(), target); + let response = call_one( &clients, + target, + request, &call.algorithm, - &call.request, - &call.models, - CallPhase::Routing, &observe, + 0, + call.models.len(), + true, ) - .await; - call.respond(result) -} - -enum CallPhase { - Routing, - Completion, + .await?; + call.respond(Ok(response)) } /// Try candidates in order until one succeeds or a failure stops fallback. @@ -206,14 +203,10 @@ async fn call_first_available( algorithm: &str, request: &Request, models: &[ModelId], - phase: CallPhase, observe: &(dyn Fn(LlmCallObservation) + Send + Sync), ) -> Result { for (index, target) in models.iter().enumerate() { - let request = match phase { - CallPhase::Routing => clients.prepare_routing_request(request.clone(), target), - CallPhase::Completion => clients.prepare_completion_request(request.clone(), target), - }; + let request = clients.prepare_completion_request(request.clone(), target); match call_one( clients, target, @@ -222,6 +215,7 @@ async fn call_first_available( observe, index, models.len(), + false, ) .await { @@ -291,6 +285,7 @@ async fn call_one( index: usize, // count is for span log count: usize, + buffer: bool, ) -> Result { let span = tracing::Span::current(); observability::record_gen_ai_request(&span, &request.llm_request); @@ -305,10 +300,14 @@ async fn call_one( // the provider's, so it belongs in the routing overhead. let client = clients.route(model_id); let started = Instant::now(); - let result = match client { - Ok(client) => client.call(request).await, - Err(error) => Err(error), + let result = async { + let mut response = client?.call(request).await?; + if buffer && let LlmResponse::Stream(chunks) = response.llm_response { + response.llm_response = LlmResponse::Stream(buffer_routing_stream(chunks).await?); + } + Ok(response) } + .await .map_err(|source| LibsyError::client_call(model_id.clone(), source)); let duration = started.elapsed(); @@ -330,6 +329,31 @@ async fn call_one( result } +// Preserve signed provider events while checking the complete routing response. +async fn buffer_routing_stream( + mut chunks: LlmResponseStream, +) -> std::result::Result { + let mut events = Vec::new(); + while let Some(event) = chunks.try_next().await? { + for chunk in event.normalized() { + match chunk { + LlmResponseChunk::DecodeError { message } => { + return Err(LlmClientError::ResponseTranslation(message.clone())); + } + LlmResponseChunk::StreamError { message } => { + return Err(LlmClientError::UpstreamHttp { + status: StatusCode::BAD_GATEWAY, + body: message.clone(), + }); + } + _ => {} + } + } + events.push(event); + } + Ok(stream::iter(events.into_iter().map(Ok)).boxed()) +} + /// Whether a failed candidate is worth routing around. fn fallback_reason(error: &LibsyError) -> Option { let LibsyError::ClientCall { source, .. } = error else { @@ -337,9 +361,7 @@ fn fallback_reason(error: &LibsyError) -> Option { }; match source { LlmClientError::ContextWindowExceeded { .. } => Some(RoutingFallbackReason::ContextWindow), - LlmClientError::Transport { .. } | LlmClientError::Timeout { .. } => { - Some(RoutingFallbackReason::Unavailable) - } + LlmClientError::Transport { .. } => Some(RoutingFallbackReason::Unavailable), LlmClientError::UpstreamHttp { status, .. } if matches!( *status, @@ -967,6 +989,7 @@ mod tests { extra_body: BTreeMap::from([("store".to_string(), json!(store))]), reasoning_effort: None, max_retries: 0, + timeout: None, }; let backend = if responses { Backend::OpenAiResponses(config) @@ -1335,6 +1358,7 @@ mod tests { extra_body: BTreeMap::new(), reasoning_effort: None, max_retries: 2, + timeout: None, }) }; let client = Arc::new( @@ -1432,6 +1456,7 @@ mod tests { extra_body: BTreeMap::new(), reasoning_effort: None, max_retries: 0, + timeout: None, }) }; let client = Arc::new( diff --git a/crates/libsy-llm-client/tests/observability.rs b/crates/libsy-llm-client/tests/observability.rs index 473658281..26757982b 100644 --- a/crates/libsy-llm-client/tests/observability.rs +++ b/crates/libsy-llm-client/tests/observability.rs @@ -1380,14 +1380,21 @@ async fn upstream_body_is_redacted_from_the_client_call_span() -> switchyard_lib judge_model: JUDGE.into(), outcome: JudgeOutcome::CallFailure, }) as Arc; - run_classifier( + let result = run_classifier( JUDGE, "redaction-weak", "redaction-strong", client, classifier_request(), ) - .await?; + .await; + assert!(matches!( + result, + Err(LibsyError::ClientCall { + source: LlmClientError::UpstreamHttp { status, body }, + .. + }) if status == http::StatusCode::INTERNAL_SERVER_ERROR && body.contains(LEAKED_CONTENT) + )); let spans = store.spans(); let client_span = find_span(&spans, "libsy.client_call", "selected_model", JUDGE); @@ -1402,6 +1409,13 @@ async fn upstream_body_is_redacted_from_the_client_call_span() -> switchyard_lib .unwrap_or(""); assert!(error.contains("upstream HTTP 500"), "{client_span:?}"); assert!(!error.contains(LEAKED_CONTENT), "{client_span:?}"); + assert!(!spans.iter().any(|span| { + span.name == "libsy.client_call" + && matches!( + span.fields.get("selected_model").map(String::as_str), + Some("redaction-weak" | "redaction-strong") + ) + })); Ok(()) } @@ -1576,22 +1590,19 @@ async fn classifier_metrics_count_routing_and_answer_calls_once() -> switchyard_ } #[tokio::test] -async fn classifier_fail_open_records_each_failure_stage() -> switchyard_libsy::Result<()> { +async fn classifier_stops_on_client_errors_and_records_verdict_fallback() +-> switchyard_libsy::Result<()> { let _guard = serialize_test().lock().await; - let (_store, exporter, provider, _, _) = telemetry(); + let (_, exporter, provider, _, _) = telemetry(); let cases = [ - ("fo-call", JudgeOutcome::CallFailure, Some("upstream_5xx")), + ("fo-call", JudgeOutcome::CallFailure, None), ( "fo-parse", JudgeOutcome::Reply("not json at all"), Some("parse_error"), ), - ( - "fo-stream-decode", - JudgeOutcome::StreamDecodeFailure, - Some("invalid_response"), - ), + ("fo-stream-decode", JudgeOutcome::StreamDecodeFailure, None), ( "fo-valid", JudgeOutcome::Reply( @@ -1605,15 +1616,26 @@ async fn classifier_fail_open_records_each_failure_stage() -> switchyard_libsy:: let client = Arc::new(JudgeClient { judge_model: judge_model.into(), outcome, - }) as Arc; - run_classifier( + }); + let result = run_classifier( judge_model, "fo-weak", "fo-strong", - client, + client.clone(), classifier_request(), ) - .await?; + .await; + match client.outcome { + JudgeOutcome::CallFailure => assert!(result.is_err(), "{judge_model}"), + JudgeOutcome::StreamDecodeFailure => assert!(matches!( + result, + Err(LibsyError::ClientCall { + source: LlmClientError::ResponseTranslation { .. }, + .. + }) + )), + JudgeOutcome::Reply(_) => assert_eq!(result?.0.as_str(), "fo-strong"), + } let snapshots = flushed_metrics(exporter, provider); match expected_reason { @@ -1633,7 +1655,7 @@ async fn classifier_fail_open_records_each_failure_stage() -> switchyard_libsy:: &[("judge_model", judge_model)], ), None, - "a valid verdict was counted as a fail-open" + "client failures and valid verdicts must not increment switchyard.classifier_fail_open" ), } } diff --git a/crates/switchyard-runner/src/config.rs b/crates/switchyard-runner/src/config.rs index 2d258474f..58a8513ee 100644 --- a/crates/switchyard-runner/src/config.rs +++ b/crates/switchyard-runner/src/config.rs @@ -7,6 +7,7 @@ use std::collections::{BTreeMap, HashMap}; use std::fs; use std::path::Path; use std::sync::Arc; +use std::time::Duration; use libsy::RuntimeModels; use serde::de::DeserializeOwned; @@ -558,6 +559,8 @@ struct LlmClientConfig { extra_headers: BTreeMap, #[serde(default = "default_max_retries")] max_retries: u32, + /// Deadline in milliseconds for all attempts and the complete response. Unset is unbounded. + timeout_ms: Option, } #[derive(Debug, Deserialize)] @@ -634,6 +637,11 @@ fn build_backend( "llm client {client_name} max_retries must be at most {MAX_CONFIGURED_RETRIES}" ))); } + if config.timeout_ms == Some(0) { + return Err(RunnerError::configuration(format!( + "llm client {client_name} timeout_ms must be at least 1" + ))); + } if config.forward_auth && config.api_key_env.is_some() { return Err(RunnerError::configuration(format!( "llm client {client_name} cannot set both forward_auth and api_key_env" @@ -669,6 +677,7 @@ fn build_backend( extra_body: extra_body.clone(), reasoning_effort, max_retries: config.max_retries, + timeout: config.timeout_ms.map(Duration::from_millis), }; let backend = match config.format { ClientFormat::OpenAiChat => Backend::OpenAiChat(http), @@ -1087,17 +1096,22 @@ new = ["send_message"] #[test] fn rejects_invalid_unreferenced_llm_client() { - let invalid = format!( - "{VALID_CONFIG}\n\ - [llm_clients.unused]\n\ - format = \"openai_chat\"\n\ - base_url = \"not a url\"\n" - ); - let message = error_message(&invalid); - assert!( - message.contains("base_url must be an absolute HTTP(S) URL"), - "unexpected error: {message}" - ); + for (settings, expected) in [ + ( + "base_url = \"not a url\"", + "base_url must be an absolute HTTP(S) URL", + ), + ( + "base_url = \"https://example.test\"\ntimeout_ms = 0", + "timeout_ms must be at least 1", + ), + ] { + let invalid = format!( + "{VALID_CONFIG}\n[llm_clients.unused]\nformat = \"openai_chat\"\n{settings}" + ); + let message = error_message(&invalid); + assert!(message.contains(expected), "unexpected error: {message}"); + } } #[test] @@ -1581,6 +1595,27 @@ target = "azure" Ok(()) } + #[test] + fn client_deadline_defaults_and_rejects_zero() -> RunnerResult<()> { + for (setting, expected) in [ + ("", None), + ("timeout_ms = 1500", Some(1500)), + ("timeout_ms = 0", Some(0)), + ] { + let source = format!( + "format = \"openai_chat\"\nbase_url = \"https://example.test/v1\"\n{setting}" + ); + let config: LlmClientConfig = toml::from_str(&source).expect("valid deadline config"); + let backend = build_backend("test", &config, &BTreeMap::new(), None); + if expected == Some(0) { + assert!(backend.is_err()); + } else { + assert_eq!(backend?.timeout(), expected.map(Duration::from_millis)); + } + } + Ok(()) + } + #[test] fn retry_budget_defaults_and_accepts_an_override() -> RunnerResult<()> { let default: DeploymentConfig = toml::from_str(VALID_CONFIG).map_err(|error| { diff --git a/crates/switchyard-server/tests/client_deadline.rs b/crates/switchyard-server/tests/client_deadline.rs new file mode 100644 index 000000000..58bd06fc3 --- /dev/null +++ b/crates/switchyard-server/tests/client_deadline.rs @@ -0,0 +1,328 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Run deadline cases in one test because they share process-wide attempt counters. + +use std::convert::Infallible; +use std::error::Error; +use std::io::Write; +use std::sync::Arc; +use std::time::Duration; + +use axum::body::{Body, Bytes}; +use axum::extract::State; +use axum::http::{Request, StatusCode}; +use axum::response::{IntoResponse, Response}; +use axum::routing::post; +use axum::{Json, Router}; +use http_body_util::BodyExt; +use serde_json::{Value, json}; +use switchyard_server::{build_switchyard_router, config::load_server_state}; +use tokio::net::TcpListener; +use tokio::sync::Mutex; +use tokio::task::JoinHandle; +use tower::ServiceExt; + +type TestResult = Result>; + +// `Upstream::drop` stops the server task even when an assertion fails. +struct Upstream { + url: String, + calls: Arc>>, + task: JoinHandle>, +} + +impl Upstream { + async fn start() -> TestResult { + let calls = Arc::new(Mutex::new(Vec::new())); + let app = Router::new() + .route("/v1/chat/completions", post(upstream)) + .with_state(Arc::clone(&calls)); + let listener = TcpListener::bind("127.0.0.1:0").await?; + let url = format!("http://{}/v1", listener.local_addr()?); + let task = tokio::spawn(async move { axum::serve(listener, app).await }); + Ok(Self { url, calls, task }) + } +} + +impl Drop for Upstream { + fn drop(&mut self) { + self.task.abort(); + } +} + +async fn upstream( + State(calls): State>>>, + Json(body): Json, +) -> Response { + let model = body["model"].as_str().unwrap_or_default(); + let attempt = { + let mut calls = calls.lock().await; + calls.push(body.clone()); + calls.iter().filter(|call| call["model"] == model).count() + }; + if model == "backoff" || (model == "retry-body" && attempt == 1) { + if model == "retry-body" { + tokio::time::sleep(Duration::from_millis(300)).await; + } + let delay = if model == "backoff" { "1" } else { "0" }; + return ( + StatusCode::SERVICE_UNAVAILABLE, + [("retry-after", delay)], + "retry", + ) + .into_response(); + } + if model == "stalled" { + tokio::time::sleep(Duration::from_secs(3)).await; + } + if body["stream"] == true { + let model = model.to_string(); + let stream = async_stream::stream! { + let first = json!({"id":"test", "model":model, "choices":[{"index":0,"delta":{"role":"assistant","content":"PONG"},"finish_reason":null}]}); + yield Ok::<_, Infallible>(Bytes::from(format!("data: {first}\n\n"))); + let delay = match model.as_str() { "slow-body" => 3000, "streaming" => 150, _ => 5 }; + tokio::time::sleep(Duration::from_millis(delay)).await; + if model == "stream-error" { + yield Ok(Bytes::from_static(b"data: {\"error\":{\"message\":\"upstream stream failed\"}}\n\ndata: [DONE]\n\n")); + return; + } + let last = json!({"id":"test", "model":model, "choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}); + yield Ok(Bytes::from(format!("data: {last}\n\ndata: [DONE]\n\n"))); + }; + return ( + [("content-type", "text/event-stream")], + Body::from_stream(stream), + ) + .into_response(); + } + let content = if model == "retry-body" { + r#"{"crux":"bounded task","primary_rule":"SUP-1","capability_boundary":"supported","p_solve":0.9}"# + } else { + "PONG" + }; + let response = json!({"id":"test","object":"chat.completion","model":model, + "choices":[{"index":0,"message":{"role":"assistant","content":content},"finish_reason":"stop"}], + "usage":{"prompt_tokens":3,"completion_tokens":1,"total_tokens":4}}); + if model == "slow-body" || model == "retry-body" { + let delay = if model == "retry-body" { 450 } else { 3000 }; + let stream = async_stream::stream! { + yield Ok::<_, Infallible>(Bytes::from_static(b" ")); + tokio::time::sleep(Duration::from_millis(delay)).await; + yield Ok(Bytes::from(response.to_string())); + }; + return ( + [("content-type", "application/json")], + Body::from_stream(stream), + ) + .into_response(); + } + Json(response).into_response() +} + +fn app( + upstream: &Upstream, + mode: &str, + judge: &str, + weak: &str, + timeout: u64, +) -> TestResult { + let route = match mode { + "escalation" => { + "type = \"llm_classifier\"\nmode = \"escalation\"\nclassifier_target = \"judge\"\nstrong_target = \"strong\"\nweak_target = \"weak\"\nescalation = { confirmations = 1 }" + } + "candidates" => { + "type = \"random\"\ntargets = [\"weak\", \"strong\"]\nweights = [1000000, 1]\nseed = 17" + } + "terminal" => "type = \"passthrough\"\ntarget = \"weak\"", + _ => { + "type = \"llm_classifier\"\nclassifier_target = \"judge\"\nstrong_target = \"strong\"\nweak_target = \"weak\"\nbase_threshold = 0.5" + } + }; + let mut config = tempfile::Builder::new().suffix(".toml").tempfile()?; + write!( + config, + r#" +schema_version = 1 +[llm_clients.http] +format = "openai_chat" +base_url = "{}" +max_retries = 1 +timeout_ms = {timeout} +[targets] +judge = {{ id = "{judge}", llm_client = "http", extra_body = {{ stream = {judge_stream} }} }} +weak = {{ id = "{weak}", llm_client = "http" }} +strong = {{ id = "strong", llm_client = "http" }} +[routes.test] +id = "test" +{route} +"#, + upstream.url, + judge_stream = judge == "stream-error", + )?; + Ok(build_switchyard_router(load_server_state(config.path())?)) +} + +async fn send(app: &Router, path: &str, body: Option) -> TestResult<(StatusCode, String)> { + let (method, body) = match body { + Some(body) => ("POST", Body::from(serde_json::to_vec(&body)?)), + None => ("GET", Body::empty()), + }; + let response = tokio::time::timeout( + Duration::from_secs(2), + app.clone().oneshot( + Request::builder() + .method(method) + .uri(path) + .header("content-type", "application/json") + .body(body)?, + ), + ) + .await??; + let status = response.status(); + let bytes = tokio::time::timeout(Duration::from_secs(2), response.into_body().collect()) + .await?? + .to_bytes(); + Ok((status, String::from_utf8(bytes.to_vec())?)) +} + +fn metric(text: &str, code: &str) -> f64 { + text.lines() + .find(|line| { + line.starts_with("switchyard_upstream_attempts_total{") + && line.contains(&format!("code=\"{code}\"")) + }) + .and_then(|line| line.split_whitespace().last()) + .and_then(|value| value.parse().ok()) + .unwrap_or_default() +} + +async fn check( + upstream: &Upstream, + app: &Router, + endpoint: &str, + expected_calls: &[&str], + expected_counts: [f64; 3], +) -> TestResult<(StatusCode, String)> { + upstream.calls.lock().await.clear(); + let (_, before) = send(app, "/metrics", None).await?; + let mut body = if endpoint == "/v1/responses" { + json!({"model":"test", "input":"Say PONG", "stream":true}) + } else { + json!({"model":"test", "messages":[{"role":"user","content":"Say PONG"}], "max_tokens":32, "stream":true}) + }; + if endpoint == "/v1/decision" { + body = json!({"input_format":"openai_chat", "request":body}); + } + let result = send(app, endpoint, Some(body)).await?; + let calls = upstream.calls.lock().await.clone(); + let actual: Vec<_> = calls + .iter() + .filter_map(|call| call["model"].as_str()) + .collect(); + assert_eq!(actual, expected_calls, "{endpoint}: {}", result.1); + let (_, after) = send(app, "/metrics", None).await?; + // Count HTTP 200, HTTP 503, and pre-response failures separately. A stream + // that times out after HTTP 200 must not count as another attempt. + let counts = ["200", "503", "none"].map(|code| metric(&after, code) - metric(&before, code)); + assert_eq!(counts, expected_counts, "{endpoint}: {}", result.1); + Ok(result) +} + +#[tokio::test] +async fn client_deadline_stops_routing_and_counts_attempts() -> TestResult { + let upstream = Upstream::start().await?; + let stalled = app(&upstream, "classifier", "stalled", "weak", 100)?; + let streaming = app(&upstream, "terminal", "judge", "streaming", 500)?; + let slow_stream = app(&upstream, "terminal", "judge", "slow-body", 100)?; + for endpoint in [ + "/v1/chat/completions", + "/v1/messages", + "/v1/responses", + "/v1/decision", + ] { + let (status, body) = + check(&upstream, &stalled, endpoint, &["stalled"], [0., 0., 1.]).await?; + assert_eq!(status, StatusCode::GATEWAY_TIMEOUT, "{body}"); + assert!(body.contains("100 ms"), "{body}"); + if endpoint == "/v1/decision" { + continue; + } + let (status, body) = check( + &upstream, + &streaming, + endpoint, + &["streaming"], + [1., 0., 0.], + ) + .await?; + assert_eq!(status, StatusCode::OK, "{body}"); + assert!(body.contains("PONG"), "{body}"); + let success = match endpoint { + "/v1/messages" => "message_stop", + "/v1/responses" => "response.completed", + _ => "[DONE]", + }; + assert!(body.contains(success), "{body}"); + let (status, body) = check( + &upstream, + &slow_stream, + endpoint, + &["slow-body"], + [1., 0., 0.], + ) + .await?; + assert_eq!(status, StatusCode::OK, "{body}"); + assert!(body.contains("PONG") && body.contains("100 ms"), "{body}"); + assert!(!body.contains(success), "{body}"); + assert!(!body.contains("\"finish_reason\":\"stop\""), "{body}"); + } + + // These response and retry failures use the same client code for every endpoint. + for (judge, expected_calls, counts) in [ + ("slow-body", &["slow-body"][..], [0., 0., 1.]), + ("backoff", &["backoff"][..], [0., 1., 0.]), + // The first attempt waits 300 ms and the retry body waits 450 ms. + // Each fits the 600 ms deadline alone; together they exceed it. + ( + "retry-body", + &["retry-body", "retry-body"][..], + [0., 1., 1.], + ), + ] { + let app = app(&upstream, "classifier", judge, "weak", 600)?; + let (status, body) = check( + &upstream, + &app, + "/v1/chat/completions", + expected_calls, + counts, + ) + .await?; + assert_eq!(status, StatusCode::GATEWAY_TIMEOUT, "{judge}: {body}"); + assert!(body.contains("600 ms"), "{body}"); + } + let failed_stream = app(&upstream, "classifier", "stream-error", "weak", 100)?; + let (status, body) = check( + &upstream, + &failed_stream, + "/v1/chat/completions", + &["stream-error"], + [1., 0., 0.], + ) + .await?; + assert_eq!(status, StatusCode::BAD_GATEWAY, "{body}"); + for (mode, weak, counts) in [ + ("escalation", "slow-body", [1., 0., 0.]), + ("candidates", "stalled", [0., 0., 1.]), + ] { + let app = app(&upstream, mode, "judge", weak, 100)?; + let (status, body) = + check(&upstream, &app, "/v1/chat/completions", &[weak], counts).await?; + assert_eq!(status, StatusCode::GATEWAY_TIMEOUT, "{mode}: {body}"); + if mode == "escalation" { + assert_eq!(upstream.calls.lock().await[0]["stream"], true); + } + } + Ok(()) +} diff --git a/crates/switchyard-server/tests/server.rs b/crates/switchyard-server/tests/server.rs index 00295e7d9..98ea8387a 100644 --- a/crates/switchyard-server/tests/server.rs +++ b/crates/switchyard-server/tests/server.rs @@ -730,6 +730,7 @@ fn random_state_with_retries( extra_body: BTreeMap::new(), reasoning_effort: None, max_retries, + timeout: None, }); let target_models = routes .iter() @@ -4635,7 +4636,7 @@ fn gate_count(stats: &Value, path: &[&str]) -> u64 { // process-global, so this is the only test that emits redo / consult-failure // metrics and the only one that may assert their exact counts. #[tokio::test] -async fn advisor_route_redo_fail_open_and_stats_projection() -> TestResult { +async fn advisor_route_redo_client_error_and_stats_projection() -> TestResult { let upstream = MockUpstream::start().await?; let app = build_switchyard_router(advisor_state_no_retry(&upstream.base_url)?); let before = send(&app, "GET", "/v1/stats", None).await?.json()?; @@ -4673,7 +4674,7 @@ async fn advisor_route_redo_fail_open_and_stats_projection() -> TestResult { assert!(feedback.starts_with("A senior reviewer examined your work")); assert!(feedback.ends_with("run the tests")); - // Fail-open: the advisor 503s once (no retries) and the turn still flows. + // A failed HTTP advisor call stops the request before the algorithm can approve it. let response = send_with_headers( &app, "POST", @@ -4682,8 +4683,8 @@ async fn advisor_route_redo_fail_open_and_stats_projection() -> TestResult { &[("proxy_x_session_id", "fail-flow")], ) .await?; - assert_eq!(response.status, StatusCode::OK); - assert_eq!(response.json()?["choices"][0]["message"]["content"], "ok"); + assert_eq!(response.status, StatusCode::SERVICE_UNAVAILABLE); + assert_eq!(response.json()?["error"]["type"], "upstream_error"); assert_eq!( upstream.models().await, [ @@ -4696,9 +4697,8 @@ async fn advisor_route_redo_fail_open_and_stats_projection() -> TestResult { ); let stats = send(&app, "GET", "/v1/stats", None).await?.json()?; - // State-owned accumulator: two client-visible executor answers; the discarded REDO attempt - // is routing work. The failed advisor consult is counted separately. - assert_eq!(stats["models"]["model/executor"]["calls"], 2); + // Only the successful redo request returns an executor answer. + assert_eq!(stats["models"]["model/executor"]["calls"], 1); assert_eq!(stats["classifier"]["total_errors"], 1); // Projection deltas for the metrics only this test emits. let redo = gate_count(&stats, &["reviews", "redo", "total"]) @@ -4725,10 +4725,10 @@ async fn advisor_route_redo_fail_open_and_stats_projection() -> TestResult { gate_count(&stats, &["discarded", "tokens", "output"]), gate_count(&before, &["discarded", "tokens", "output"]) + 2 ); - // The 503 maps to the bounded upstream_5xx reason label. + // The host stops before the algorithm records a fail-open advisor decision. assert_eq!( gate_count(&stats, &["consult_failures", "upstream_5xx"]), - gate_count(&before, &["consult_failures", "upstream_5xx"]) + 1 + gate_count(&before, &["consult_failures", "upstream_5xx"]) ); // Reset re-baselines the projection: the redo/discard counts this test diff --git a/docs/reference/toml_schema.md b/docs/reference/toml_schema.md index e595e7cef..d5c78c17e 100644 --- a/docs/reference/toml_schema.md +++ b/docs/reference/toml_schema.md @@ -54,10 +54,23 @@ route reaches no upstream. A file without a `[targets]` table is rejected with | `forward_auth` | No | `false` | Forward the caller's provider credential and application headers. All backends reachable through the route must use the same provider. | | `extra_headers` | No | `{}` | Custom HTTP headers sent to the model server. Set credentials with `api_key_env` or `forward_auth`; the server rejects headers owned by the selected auth mode. Header names are case-insensitive. | | `max_retries` | No | `2` | Retry budget, `0`–`10`. | +| `timeout_ms` | No | unset | Deadline in milliseconds for all attempts, retry delays, and the complete response, including stream reads. Must be at least `1`. Unset leaves the wait unbounded. | The TOML never contains the secret itself. `api_key_env` names a variable that must exist and be non-empty when the server loads. +`timeout_ms` applies separately to every call through the client, including judge +verdicts and answers. To give a judge a short deadline without limiting the +answering models, put the judge on its own `[llm_clients]` entry; two entries may +share a `base_url`. When the deadline expires, the server returns `504` without +trying another target. If the final answer has already started streaming, the +server sends a framed error and ends the stream without a success marker. + +The Rust runner collects streams used during routing before the algorithm +continues, preserving provider events for replay. After the configured retries, +an HTTP client failure stops routing. This also applies when `timeout_ms` is +unset or an advisor has `fail_open = true`. + Set `forward_auth = true` to use each caller's credential instead of a server-owned key: @@ -222,8 +235,9 @@ Existing configurations that contain `escalation` but omit `mode` remain valid. Custom mode validates the judge's JSON against `response_schema`, resolves the policy selector, and routes to a runtime model group. A verdict names a group and -the first model in it serves the turn; if that call fails the client falls through -the rest of that group, then through whatever `models.any` adds. +the first model in it serves the turn. An eligible non-timeout failure tries the +rest of that group, then any remaining models in `models.any`. A timeout stops +the request without trying another model. The `[routes..models]` table takes any group name you choose. `any` and `judge` are reserved and required; `capable` and `efficient` are reserved for the @@ -232,12 +246,12 @@ how one route chooses between more than two models. | Key | Required | Default | Meaning | |---|:---:|---|---| -| `models.any` | Yes | — | Every selectable completion target, in last-resort fallback order. Every other group's targets must also appear here; one that does not is rejected at configuration load. | -| `models.judge` | Yes | — | One or more ordered judge candidates. Not a completion destination. | -| `models.capable` | No | — | Ordered capable-tier models. A `capable` verdict selects the first and falls through the rest in order. | -| `models.efficient` | No | — | Ordered efficient-tier models. An `efficient` verdict selects the first and falls through the rest in order. | -| `models.` | No | — | A group you name. A verdict naming it selects its first model and falls through the rest in order. | -| `default_target` | Yes | — | Group used when the judge fails or its verdict cannot be routed. Any group except `judge`, and it must contain at least one target. | +| `models.any` | Yes | — | Every selectable completion target, in fallback order for eligible non-timeout failures. Every other group's targets must also appear here; one that does not is rejected at configuration load. | +| `models.judge` | Yes | — | One or more ordered judge candidates. The Rust runner calls the first and stops on a client error after retries. Not a completion destination. | +| `models.capable` | No | — | Ordered capable-tier models. A `capable` verdict selects the first; eligible non-timeout failures try the rest in order. | +| `models.efficient` | No | — | Ordered efficient-tier models. An `efficient` verdict selects the first; eligible non-timeout failures try the rest in order. | +| `models.` | No | — | A group you name. A verdict naming it selects its first model; eligible non-timeout failures try the rest in order. | +| `default_target` | Yes | — | Group used when the judge's verdict cannot be parsed or routed. HTTP client failures stop the request after retries. Any group except `judge`, and it must contain at least one target. | | `prompt` | Yes | — | Judge system prompt. The configured inner schema is sent separately as structured-output configuration. | | `response_schema` | Yes | — | Inner JSON Schema encoded as a TOML string. Switchyard adds the provider wrapper. | | `policy` | Yes | — | Policy table. `target_selector` accepts a JSON Pointer such as `/decision/target`. | diff --git a/docs/routing_algorithms/advisor_gate_routing.md b/docs/routing_algorithms/advisor_gate_routing.md index 610530cb9..5af0077b3 100644 --- a/docs/routing_algorithms/advisor_gate_routing.md +++ b/docs/routing_algorithms/advisor_gate_routing.md @@ -61,15 +61,20 @@ stamp every request of one evaluation with it, sub-agents included, so the budget means "reviews for this task" even behind a gateway shared by many tasks — then the session id resolved from harness headers (such as `x-switchyard-session-id`), and one scope per server when neither is present. -Failed consults refund the budget and count toward a separate cap of 3, which -bounds consult latency against a down advisor. An unparseable verdict also -refunds and passes the turn through as APPROVE. +For hosts that return client errors to the core algorithm, a failed consult refunds +the review budget and counts toward a separate cap of 3. The Rust runner stops on +HTTP client errors before returning them to the algorithm, so failed HTTP calls +do not refund this budget. An unparseable verdict still refunds the budget and +passes the turn through as APPROVE. Long sessions are truncated middle-out before the consult: the transcript keeps the task statement at the start and the most recent work at the end, marked `......`, capped at -`transcript_max_chars`. With `fail_open = true` (default) any advisor failure -degrades to APPROVE; `fail_open = false` surfaces it as a server error instead. +`transcript_max_chars`. `fail_open` controls failures returned to the core +algorithm by its host. The Rust runner stops the request after an HTTP client's +configured retries fail or its deadline expires, even with `fail_open = true`. +HTTP client errors also stop the request when `timeout_ms` is unset. A timeout +before the answer starts returns `504`. Gate behavior is observable at `/v1/stats` under `advisor_gate`: verdicts by trigger, consult failures by reason, and REDO-discarded turns with their token @@ -109,7 +114,7 @@ gate_min_tool_results = 3 | `advisor_max_tokens` | `2048` | Output cap for each advisor consult. | | `advisor_temperature` | unset | Sampling temperature for consults; omitted when unset. | | `transcript_max_chars` | `200000` | Middle-out cap on the serialized transcript (~50k tokens). | -| `fail_open` | `true` | Advisor failure passes the turn through instead of erroring. | +| `fail_open` | `true` | Failures returned to the core algorithm pass the turn through. The Rust runner stops HTTP client failures regardless of this setting. | | `reviewer_system_prompt` | built-in | Overrides the APPROVE/REDO reviewer contract. | | `redo_feedback_prefix` | built-in | Overrides the prefix injected before a REDO plan. | diff --git a/docs/routing_algorithms/escalation_router_routing.md b/docs/routing_algorithms/escalation_router_routing.md index 1ed9e8085..3fecddab6 100644 --- a/docs/routing_algorithms/escalation_router_routing.md +++ b/docs/routing_algorithms/escalation_router_routing.md @@ -86,9 +86,12 @@ flowchart LR class t,p,s,c,j,w,l box; ``` -A judge that times out, errors, or returns an unparseable verdict fails open: the -turn serves the buffered weak reply and the existing streak is held rather than -cleared. A judge failure never creates a strong-tier latch. +An unparseable verdict serves the buffered weak reply and holds the existing +streak. The Rust runner stops the request if an HTTP model call fails after +retries. To bound both the weak-model response and the judge response, set +`timeout_ms` on each `[llm_clients]` entry they use. The deadline applies separately +to each call, even when both models share a client. Expiry returns `504` without +calling another target or selecting the strong tier for subsequent session turns. ## Judge model compatibility diff --git a/docs/routing_algorithms/llm_classifier_routing.md b/docs/routing_algorithms/llm_classifier_routing.md index 329ccadc0..c1b3a3fbe 100644 --- a/docs/routing_algorithms/llm_classifier_routing.md +++ b/docs/routing_algorithms/llm_classifier_routing.md @@ -67,9 +67,19 @@ greater than or equal to the applicable threshold. Otherwise it routes to - `uncertain` and `unmatched` use `base_threshold + threshold_step`. - `unsupported` uses `base_threshold + 2 * threshold_step`. -An invalid, inconsistent, or unparseable verdict, or a judge failure, routes to +An invalid, inconsistent, or unparseable verdict routes to `strong_target`. Raising either knob sends more traffic to the strong model. +To stop waiting for a judge that accepts the request but never finishes its +response, set `timeout_ms` on the judge's `[llm_clients]` entry +(see the [TOML schema](../reference/toml_schema.md)); it covers the judge's +retries and the complete verdict body. When the deadline expires, the Rust server returns +`504` without calling `strong_target` or `weak_target`. Other HTTP client failures +also stop routing after retries. The deadline applies to every call +through that client. Give the judge its own entry if the answering models need a +different deadline, even when they use the same provider. Without a deadline, +the request can wait indefinitely for the judge. + ## Judge model compatibility The judge must return complete, schema-valid JSON in normal assistant `content`.