Honor model eos_token_id and bound local generation (v0.3.20)

[codelion]

Problem

Running a model whose ChatML end token differs from its tokenizer EOS through optillm's local inference caused runaway generation. Concretely for dhara-250m: its chat template ends at <|im_end|> (49154) but the tokenizer's eos_token is <|end_of_text|> (1).

optillm (inference.py) hardcoded eos_token_id = self.tokenizer.eos_token_id (so it waited for token 1, which chat responses never emit) and defaulted max_new_tokens to 4096. Result: the model never stopped early and generated the full 4096 tokens (~800s at ~5 tok/s) on every call that didn't send max_tokens — including approach-internal calls. Well-behaved models like Qwen2.5 were unaffected because their tokenizer eos_token is <|im_end|>.

This only affects the built-in local inference engine (OPTILLM_API_KEY=optillm); proxied providers handle stopping themselves.

Fixes

1. Honor the model's generation_config.eos_token_id. New _resolve_eos_token_ids() prefers the model's own generation-config EOS (which chat models set to their turn-end token) and merges the tokenizer EOS as a fallback. Applied to both PyTorch generate paths.

2. Env-configurable generation bound. OPTILLM_MAX_TOKENS overrides the default max_new_tokens (default stays 4096), so a single generation can be bounded even when the request sends no max_tokens. An explicit request max_tokens still takes precedence. Set to 128 in the CI jobs that run the small test model. This is the reliable stop for models like dhara that don't emit their EOS reliably at all.

Testing

• New unit tests (no model load) for _resolve_eos_token_ids and _default_max_new_tokens in test_batching.py (unit-tests job).
• End-to-end: an unbounded request (no max_tokens) to a dhara server with OPTILLM_MAX_TOKENS=64 now returns 64 tokens, finish_reason=stop, ~30s (was 800s+).
• README documents OPTILLM_MAX_TOKENS.