Decouple HTTP connection pool limits from max_parallel_requests at high concurrency

[andreatnvidia]

Priority Level

High (Major functionality broken)

Describe the bug

HttpModelClient derives HTTP transport limits directly from max_parallel_requests:

max_connections = max(32, 2 * max_parallel_requests)
max_keepalive_connections = max(16, max_parallel_requests)

At high request-admission ceilings this creates and retains a very large shared httpcore connection pool, even when the async scheduler can expose far fewer concurrent model requests. httpcore.AsyncConnectionPool._assign_requests_to_connections() and connection is_idle() checks scan Python connection state on the event-loop thread, so an oversized pool can become a dominant GIL cost.

The model request-admission ceiling and HTTP transport-pool capacity represent different concerns and should not be coupled linearly without a bound or independent control.

Steps/Code to reproduce bug

from data_designer.engine.models.clients.adapters.http_model_client import (
    ClientConcurrencyMode,
    HttpModelClient,
)


class ExampleClient(HttpModelClient):
    def _build_headers(self, extra_headers: dict[str, str]) -> dict[str, str]:
        return extra_headers


client = ExampleClient(
    provider_name="example",
    endpoint="http://127.0.0.1:8000",
    max_parallel_requests=8_192,
    concurrency_mode=ClientConcurrencyMode.ASYNC,
)

print(client.limits.max_connections)
print(client.limits.max_keepalive_connections)

This configures 16,384 total connections and 8,192 keepalive connections before any workload-specific client concurrency is considered.

For the runtime symptom, run a large asynchronous workload with a high model request ceiling and a lower scheduler task limit, then capture a GIL-only profile. Connection assignment and idle checks consume substantial event-loop CPU even though the useful concurrent request population is much smaller than the configured pool.

Expected behavior

Data Designer should size the HTTP pool from expected transport concurrency rather than directly from the model request-admission ceiling. A supported solution could provide independently configurable, validated transport limits with bounded defaults.

The implementation should:

• Preserve enough connections to sustain configured useful concurrency.
• Avoid retaining thousands of connections when scheduler demand is materially lower.
• Expose the effective transport limits for diagnostics.
• Preserve the correction made for max_parallel_requests has no effect on actual HTTP connection pool size #459, where configured limits previously did not reach the underlying transport.

Agent Diagnostic / Prior Investigation

No existing issue was found for the high-concurrency inverse of #459. That issue correctly ensured that max_parallel_requests affected the real transport pool, but the current direct 2x/1x sizing policy becomes expensive at much larger values.

In a controlled GIL-only profile, HTTP connection and transport work accounted for 52.45% of samples. After removing an independent request-admission queue bottleneck, it accounted for 75.39% and became the dominant remaining client cost. The largest leaves were _assign_requests_to_connections() and connection is_idle() variants.

The profile showed one event-loop thread consuming the client CPU while other host CPU capacity remained available. Request completion was balanced and successful, which points to client-side transport bookkeeping rather than endpoint routing or error handling.

Additional context

Related: #459. This issue is not requesting that the pool return to the old fixed limit. It requests a transport-capacity model that remains correct at both moderate and very high request-admission ceilings.

Checklist

• I reproduced this issue or provided a minimal example
• I searched the docs/issues myself, or had my agent do so
• If I used an agent, I included its diagnostics above

[andreatnvidia]

Follow-up from further experiments: the pool-size hypothesis needs some nuance.

Capping max_connections and max_keepalive_connections at 768 did not improve throughput. It reduced completed records by roughly 10-12% in comparable runs. The current evidence therefore does not support a smaller bound as the performance fix, and it may starve useful concurrency.

The successful PoC instead reduced connection-assignment bookkeeping in httpcore==1.0.9. It replaced the repeated scans in AsyncConnectionPool._assign_requests_to_connections() with a guarded single-pass assignment, changing the relevant work from O(N² + N×M) to O(N + M). With the request-admission heap fix also applied, this improved throughput by 36-42% over the heap-only two-node runs.

Independent transport configuration and effective-limit diagnostics would still be useful. For the throughput problem, though, the evidence points to linear connection assignment (or the corresponding upstream httpcore fix), not simply reducing the pool size. We may want to track that separately or revise this issue so decoupling does not imply that a smaller cap is the expected fix.