OpenVINO™ Model Server

High-performance model serving for Generative AI and classic deep learning — powered by OpenVINO and optimized for Intel hardware.

---

What is OVMS?

OpenVINO Model Server (OVMS) is a production-grade, C++ inference server that exposes ML models over standard network APIs. It serves both Generative AI, Agentic workloads (LLMs, VLMs, image generation, audio) and classic deep learning models (object detection, classification, OCR, and more).

• OpenAI-compatible API for text generation, embeddings, image generation, and audio
• KServe APIs for classic model inference
• Runs anywhere — Docker, bare metal, Kubernetes/OpenShift, Windows
• Intel-optimized — CPU, GPU, NPU acceleration via OpenVINO

---

Quick Start

Serve an LLM with OpenAI-compatible API

On Linux (Docker):

mkdir -p ${HOME}/models
# Model is downloaded automatically from HuggingFace to models folder
docker run --rm -p 8000:8000 \
  --user $(id -u):$(id -g) -v ${HOME}/models:/models:rw \
  openvino/model_server:latest \
  --source_model OpenVINO/Qwen3-4B-int4-ov \
  --model_repository_path /models \
  --rest_port 8000

For GPU acceleration, use the latest-gpu image tag and pass --device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1) to expose the Intel GPU device.

On Windows (binary package):

mkdir c:\models
ovms.exe --source_model OpenVINO/Qwen3-4B-int4-ov --model_repository_path c:\models --rest_port 8000

Query the model:

pip install openai

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
stream = client.chat.completions.create(
    model="OpenVINO/Qwen3-4B-int4-ov",
    messages=[{"role": "user", "content": "What are the 3 main tourist attractions in Paris?"}],
    stream=True,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}}
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

LLM QuickStart

Serve a Classic Model with KServe API

Download the model:

curl -L https://huggingface.co/OpenVINO/resnet50-int8-ov/resolve/main/resnet50.bin -O
curl -L https://huggingface.co/OpenVINO/resnet50-int8-ov/resolve/main/resnet50.xml -O

On Linux (Docker):

docker run --rm -d -u $(id -u) -v ${PWD}:/models -p 9000:9000 \
  openvino/model_server:latest \
  --model_name resnet --model_path /models/resnet50.xml \
  --mean "[123.675,116.28,103.53]" --scale "[58.395,57.12,57.375]" --layout "NHWC:NCHW" \
  --port 9000

For GPU acceleration, use the latest-gpu image tag and pass --device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1) to expose the Intel GPU device.

Windows (binary package):

ovms --model_name resnet --model_path resnet50.xml --mean "[123.675,116.28,103.53]" --scale "[58.395,57.12,57.375]" --layout "NHWC:NCHW" --port 9000

Run inference with a sample client

pip install numpy tritonclient[grpc]
curl -L -o image.jpeg https://github.com/openvinotoolkit/model_server/blob/main/demos/common/static/images/bee.jpeg?raw=true

import numpy as np
import tritonclient.grpc as grpcclient
with open("image.jpeg", "rb") as f:
    image_bytes = f.read()
client = grpcclient.InferenceServerClient(url="localhost:9000")
inputs = [grpcclient.InferInput("image", [1], "BYTES")]
inputs[0].set_data_from_numpy(np.array([image_bytes], dtype=object))
outputs = [grpcclient.InferRequestedOutput("output")]
result = client.infer(model_name="resnet", inputs=inputs, outputs=outputs)
output = result.as_numpy("output")  # (1, 1000) FP32
print("Top-1 class index:", int(np.argmax(output[0])))

Vision model QuickStart

---

Features

Generative AI

• LLM text generation — continuous batching, streaming, structured output, speculative decoding
• VLM (Vision Language Models)
• AI Agents with MCP servers
• Text embeddings — OpenAI-compatible /v1/embeddings
• Reranking — Cohere-compatible API
• Image generation — OpenAI-compatible /v1/images/generations
• Speech recognition and TTS — OpenAI-compatible audio API
• GGUF model support

Classic Models & Pipelines

• TensorFlow, ONNX, PaddlePaddle, OpenVINO IR model formats
• MediaPipe graphs
• Python execution nodes
• Dynamic input shapes

Deployment & Integration

• Docker, bare metal (Linux & Windows), Kubernetes / OpenShift
• Model repository: local storage, S3, GCS, Azure Blob, HuggingFace Hub
• Model versioning and hot-reload
• Prometheus-compatible metrics
• gRPC streaming
• C API for embedding OVMS in native applications

Hardware Acceleration

• CPU (x86, including Xeon), Intel integrated and discrete GPU, NPU
• See supported accelerators

→ Full feature list

---

Documentation

Topic	Link
Deployment	Deploying the server
Model repository	Preparing models
Client libraries	Writing client code
Demos & examples	Demos
Release notes	GitHub Releases

---

Get the Server

Docker images:

docker pull openvino/model_server:latest        # Intel CPU
docker pull openvino/model_server:latest-gpu    # Intel CPU,GPU,NPU

docker pull openvino/model_server:weekly        # pre-production version with all accelerators enabled

• Docker Hub
• Red Hat Ecosystem Catalog

Binary official packages (Linux & Windows): GitHub Releases

Binary pre-production packages (Linux & Windows): storage.openvinotoolkit.org

---

Contributing

Contributions are welcome! Please open an issue or pull request on GitHub.
See security policy for responsible disclosure.

---

References

• OpenVINO Toolkit
• Performance benchmarks
• GenAI with CPU optimization — Intel whitepaper
• RAG with OpenVINO Model Server — blog post
• AIPC turned into a mighty assistant

---

* Other names and brands may be claimed as the property of others.