Skip to content

Add unit tests for VLLMModel core methods (#724) - #1377

Open
sneha4175 wants to merge 1 commit into
huggingface:mainfrom
sneha4175:add-vllm-model-unit-tests
Open

Add unit tests for VLLMModel core methods (#724)#1377
sneha4175 wants to merge 1 commit into
huggingface:mainfrom
sneha4175:add-vllm-model-unit-tests

Conversation

@sneha4175

Copy link
Copy Markdown

What does this PR do?

Adds unit tests for the VLLMModel class methods, as requested by @lewtun in #724 (unit tests to "catch subtle bugs like [#721] and also protect us a bit from silent, yet breaking changes on the vllm side").

The existing tests/unit/models/vllm/test_vllm_model.py only exercised tokenizer/chat-template wiring. This PR adds coverage for the pure-Python logic of the model, focusing on the boundary with vllm's SamplingParams (exactly the area #721 broke) and the loglikelihood math.

Methods now covered

  • _generate — the vllm boundary:
    • greedy scoring path (generate=False): temperature=0, prompt_logprobs=1, max_tokens=1, detokenize=False, and prompts wrapped via build_vllm_token_prompts
    • generation path: n, max_tokens, stop, and logprobs mapping (incl. returns_logits on/off)
    • the num_samples > 1 + temperature == 0 guard raising ValueError, and that sampling is allowed with a positive temperature
  • _create_auto_model — model-args wiring (max_model_len, seed casting, enforce_eager, tensor-parallel size), optional quantization/load_format forwarding, subfolderrevision handling, and data-parallel deferral (returns None, sets the ray executor backend)
  • _loglikelihood_tokens — continuation logprob slicing, summation, and rank-1 argmax bookkeeping, plus the resulting ModelResponse fields
  • max_length, add_special_tokens, tokenizer properties
  • build_vllm_token_prompts batching/empty-input behaviour

How vllm is mocked

vllm is a heavy GPU dependency and VLLMModel is gated behind @requires("vllm") (it degrades to a placeholder when vllm is absent). To keep these tests CPU-runnable while still exercising the model's own code, a vllm_model_module fixture injects a lightweight fake vllm (and ray) package into sys.modules — including a recording SamplingParams stand-in — patches is_package_available and reloads the module so the real class loads, then reloads again on teardown to restore the original import state. Instances are built with __new__ to skip __init__ (no real engine), setting only the attributes each method needs. When vllm is installed (the dev-gpu CI env), the fakes simply shadow it for the duration of each test.

Test / lint results

  • pytest tests/unit/models/vllm/test_vllm_model.py -k "Properties or GenerateSamplingParams or CreateAutoModel or LoglikelihoodTokens or TokenPromptsBatching"14 passed
  • ruff check and ruff format --check → clean

Closes #724 (or fixes part of it — happy to adjust scope).

Before submitting

Adds CPU-runnable unit tests for the pure-Python logic of VLLMModel, as
requested in huggingface#724, covering the vllm boundary and loglikelihood math:

- _generate: greedy scoring SamplingParams, generation-path field mapping,
  logprobs toggling, and the temperature=0 multi-sample guard
- _create_auto_model: model-args wiring, optional quantization/load_format,
  subfolder revision handling, and data-parallel deferral
- _loglikelihood_tokens: continuation logprob slicing, summation and argmax
- max_length / add_special_tokens / tokenizer properties
- build_vllm_token_prompts batching

vllm is mocked via a lightweight fake package injected into sys.modules with
a recording SamplingParams stand-in, so the tests run without a GPU.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FT] Add tests for VLLMModel base methods

1 participant