-
Notifications
You must be signed in to change notification settings - Fork 269
Menu and documentation changes #4423
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
17 commits
Select commit
Hold shift + click to select a range
515db21
update menu and main page
dtrawins 6fb3660
updates
dtrawins 65e4436
fixes
dtrawins d5a289a
fix
dtrawins b6b44e3
fix1
dtrawins 4119058
menu fix
dtrawins 310047a
shorer demo
dtrawins fb639d1
review corrections
dtrawins bf5d41f
Merge branch 'releases/2026/3' into menu-rel
dtrawins 4a234a1
fix docs test flow
dtrawins 84fc38e
Merge branch 'menu-rel' of https://github.com/openvinotoolkit/model_s…
dtrawins a923b5d
fix home commands
dtrawins 97c6ea9
CVS-177198 LLM Memory Usage tests (#4360)
porlows1 61cbcfe
curl -l
dtrawins a2be74a
[validation_branch=20260827_test_fixes]
dtrawins 0df9ae8
merge
dtrawins 00b68ec
[validation_branch=20260827_test_fixes]
dtrawins File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,88 +1,197 @@ | ||
| # OpenVINO™ Model Server | ||
|
|
||
| Model Server hosts models and makes them accessible to software components over standard network protocols: a client sends a request to the model server, which performs model inference and sends a response back to the client. Model Server offers many advantages for efficient model deployment: | ||
| - Remote inference enables using lightweight clients with only the necessary functions to perform API calls to edge or cloud deployments. | ||
| - Applications are independent of the model framework, hardware device, and infrastructure. | ||
| - Client applications in any programming language that supports REST or gRPC calls can be used to run inference remotely on the model server. | ||
| - Clients require fewer updates since client libraries change very rarely. | ||
| - Model topology and weights are not exposed directly to client applications, making it easier to control access to the model. | ||
| - Ideal architecture for microservices-based applications and deployments in cloud environments – including Kubernetes and OpenShift clusters. | ||
| - Efficient resource utilization with horizontal and vertical inference scaling. | ||
| **High-performance model serving for Generative AI and classic deep learning — powered by [OpenVINO](https://github.com/openvinotoolkit/openvino) and optimized for Intel hardware.** | ||
|
|
||
|  | ||
|
|
||
| OpenVINO™ Model Server (OVMS) is a high-performance system for serving models. Implemented in C++ for scalability and optimized for deployment on Intel architectures. It uses the [generative API](https://docs.openvino.ai/2026/model-server/ovms_docs_clients_genai.html) like OpenAI and Cohere, [KServe](https://docs.openvino.ai/2026/model-server/ovms_docs_clients_kfs.html) while applying OpenVINO for inference execution. Inference service is provided via gRPC or REST API, making deploying new algorithms and AI experiments easy. | ||
| [](https://github.com/openvinotoolkit/model_server/blob/main/LICENSE) | ||
| [](https://hub.docker.com/r/openvino/model_server) | ||
| [](https://github.com/openvinotoolkit/model_server/releases) | ||
| [](https://docs.openvino.ai/2026/model-server/ovms_docs_deploying_server.html) | ||
|
|
||
|  | ||
|
|
||
| The models used by the server can be stored locally, hosted remotely by object storage services or pulled from HuggingFace Hub. For more details, refer to [Preparing Model Repository](https://docs.openvino.ai/2026/model-server/ovms_docs_models_repository.html) and [Deployment](https://docs.openvino.ai/2026/model-server/ovms_docs_deploying_server.html) documentation. | ||
| Model server works inside Docker containers, Bare Metal and in Kubernetes environment. | ||
| --- | ||
|
|
||
| Start using OpenVINO Model Server with a fast-forward serving example from the [QuickStart guide](https://docs.openvino.ai/2026/model-server/ovms_docs_quick_start_guide.html) or [LLM QuickStart guide](https://docs.openvino.ai/2026/model-server/ovms_docs_llm_quickstart.html). | ||
| ## What is OVMS? | ||
|
|
||
| Read [release notes](https://github.com/openvinotoolkit/model_server/releases) to find out what’s new. | ||
| OpenVINO Model Server (OVMS) is a production-grade, C++ inference server that exposes ML models over standard network APIs. It serves both **Generative AI**, **Agentic** workloads (LLMs, VLMs, image generation, audio) and **classic deep learning** models (object detection, classification, OCR, and more). | ||
|
|
||
| ### Key features: | ||
| - **[NEW]** [Speech Generation and Speech Recognition with OpenAI API](https://docs.openvino.ai/2026/model-server/ovms_demos_audio.html) | ||
| - **[NEW]** [Support for AI agents](https://docs.openvino.ai/2026/model-server/ovms_demos_continuous_batching_agent.html) | ||
| - **[NEW]** [Image generation compatible with OpenAI API](https://docs.openvino.ai/2026/model-server/ovms_demos_image_generation.html) | ||
| - Native Windows support. Check updated [deployment guide](https://docs.openvino.ai/2026/model-server/ovms_docs_deploying_server_baremetal.html) | ||
| - [Text Embeddings compatible with OpenAI API](https://docs.openvino.ai/2026/model-server/ovms_demos_embeddings.html) | ||
| - [Reranking compatible with Cohere API](https://docs.openvino.ai/2026/model-server/ovms_demos_rerank.html) | ||
| - [Efficient Text Generation via OpenAI API](https://docs.openvino.ai/2026/model-server/ovms_demos_continuous_batching.html) | ||
| - [Python code execution](docs/python_support/reference.md) | ||
| - [gRPC streaming](docs/streaming_endpoints.md) | ||
| - [MediaPipe graphs serving](docs/mediapipe.md) | ||
| - Model management - including [model versioning](docs/model_version_policy.md) and [model updates in runtime](docs/online_config_changes.md) | ||
| - [Dynamic model inputs](docs/shape_batch_size_and_layout.md) | ||
| - [Directed Acyclic Graph Scheduler](docs/dag_scheduler.md) along with [custom nodes in DAG pipelines](docs/custom_node_development.md) | ||
| - [Metrics](docs/metrics.md) - metrics compatible with Prometheus standard | ||
| - Support for multiple frameworks, such as TensorFlow, PaddlePaddle and ONNX | ||
| - Support for [AI accelerators](./docs/accelerators.md) | ||
| - **OpenAI-compatible API** for text generation, embeddings, image generation, and audio | ||
| - **KServe** APIs for classic model inference | ||
| - **Runs anywhere** — Docker, bare metal, Kubernetes/OpenShift, Windows | ||
| - **Intel-optimized** — CPU, GPU, NPU acceleration via OpenVINO | ||
|
|
||
| Check full list of [features](./docs/features.md) | ||
|  | ||
|
|
||
| **Note:** OVMS has been tested on RedHat, Ubuntu and Windows. | ||
| Public docker images are stored in: | ||
| - [Dockerhub](https://hub.docker.com/r/openvino/model_server) | ||
| - [RedHat Ecosystem Catalog](https://catalog.redhat.com/software/containers/intel/openvino-model-server/607833052937385fc98515de) | ||
| Binary packages for Linux and Windows are on [Github](https://github.com/openvinotoolkit/model_server/releases) | ||
| --- | ||
|
|
||
| ## Run OpenVINO Model Server | ||
| ## Quick Start | ||
|
|
||
| ### Serve an LLM with OpenAI-compatible API | ||
|
|
||
| **On Linux (Docker):** | ||
| ```bash | ||
| # Model is downloaded automatically from HuggingFace | ||
| docker run --rm -p 8000:8000 \ | ||
| openvino/model_server:latest \ | ||
| --source_model OpenVINO/Qwen3-4B-int4-ov \ | ||
| --model_repository_path /tmp/models \ | ||
| --rest_port 8000 | ||
| ``` | ||
| > For GPU acceleration, use the `latest-gpu` image tag and pass `--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)` to expose the Intel GPU device. | ||
|
|
||
| **On Windows (binary package):** | ||
| ```bat | ||
| mkdir c:\models | ||
| ovms.exe --source_model OpenVINO/Qwen3-4B-int4-ov --model_repository_path c:\models --rest_port 8000 | ||
| ``` | ||
|
|
||
| **Query the model:** | ||
| ```console | ||
| pip install openai | ||
| ``` | ||
| ```python | ||
| from openai import OpenAI | ||
|
|
||
| client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused") | ||
| stream = client.chat.completions.create( | ||
| model="OpenVINO/Qwen3-4B-int4-ov", | ||
| messages=[{"role": "user", "content": "What are the 3 main tourist attractions in Paris?"}], | ||
| stream=True, | ||
| extra_body={"chat_template_kwargs": {"enable_thinking": False}} | ||
| ) | ||
| for chunk in stream: | ||
| if chunk.choices[0].delta.content: | ||
| print(chunk.choices[0].delta.content, end="", flush=True) | ||
| ``` | ||
| > [LLM QuickStart](https://docs.openvino.ai/2026/model-server/ovms_docs_llm_quickstart.html) | ||
|
|
||
|
|
||
|
|
||
| ### Serve a Classic Model with KServe API | ||
|
|
||
| **Download the model:** | ||
| ```console | ||
| curl -L https://huggingface.co/OpenVINO/resnet50-int8-ov/resolve/main/resnet50.bin -O | ||
| curl -L https://huggingface.co/OpenVINO/resnet50-int8-ov/resolve/main/resnet50.xml -O | ||
| ``` | ||
|
|
||
| **On Linux (Docker):** | ||
| ```bash | ||
| docker run --rm -d -u $(id -u) -v ${PWD}:/models -p 9000:9000 \ | ||
| openvino/model_server:latest \ | ||
| --model_name resnet --model_path /models/resnet50.xml \ | ||
| --mean "[123.675,116.28,103.53]" --scale "[58.395,57.12,57.375]" --layout "NHWC:NCHW" \ | ||
| --port 9000 | ||
| ``` | ||
| > For GPU acceleration, use the `latest-gpu` image tag and pass `--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)` to expose the Intel GPU device. | ||
|
|
||
| **Windows (binary package):** | ||
| ```bat | ||
| ovms --model_name resnet --model_path resnet50.xml --mean "[123.675,116.28,103.53]" --scale "[58.395,57.12,57.375]" --layout "NHWC:NCHW" --port 9000 | ||
|
|
||
| ``` | ||
|
|
||
| Run inference with a sample client | ||
| ```console | ||
| pip install numpy tritonclient[grpc] | ||
| curl -L -o image.jpeg https://github.com/openvinotoolkit/model_server/blob/main/demos/common/static/images/bee.jpeg?raw=true | ||
| ``` | ||
| ```python | ||
| import numpy as np | ||
| import tritonclient.grpc as grpcclient | ||
| with open("image.jpeg", "rb") as f: | ||
| image_bytes = f.read() | ||
| client = grpcclient.InferenceServerClient(url="localhost:9000") | ||
| inputs = [grpcclient.InferInput("image", [1], "BYTES")] | ||
| inputs[0].set_data_from_numpy(np.array([image_bytes], dtype=object)) | ||
| outputs = [grpcclient.InferRequestedOutput("output")] | ||
| result = client.infer(model_name="resnet", inputs=inputs, outputs=outputs) | ||
| output = result.as_numpy("output") # (1, 1000) FP32 | ||
| print("Top-1 class index:", int(np.argmax(output[0]))) | ||
| ``` | ||
|
|
||
| > [Vision model QuickStart](https://docs.openvino.ai/2026/model-server/ovms_docs_quick_start_guide.html) | ||
|
|
||
| A demonstration on how to use OpenVINO Model Server can be found in our [quick-start guide for vision use case](docs/ovms_quickstart.md) and [LLM text generation](docs/llm/quickstart.md). | ||
| --- | ||
|
|
||
| Check also other instructions: | ||
| ## Features | ||
|
|
||
| ### Generative AI | ||
| - [LLM text generation](https://docs.openvino.ai/2026/model-server/ovms_demos_continuous_batching.html) — continuous batching, streaming, structured output, speculative decoding | ||
| - [VLM (Vision Language Models)](https://docs.openvino.ai/2026/model-server/ovms_demos_continuous_batching_vlm.html) | ||
| - [AI Agents with MCP servers](https://docs.openvino.ai/2026/model-server/ovms_demos_continuous_batching_agent.html) | ||
| - [Text embeddings](https://docs.openvino.ai/2026/model-server/ovms_demos_embeddings.html) — OpenAI-compatible `/v1/embeddings` | ||
| - [Reranking](https://docs.openvino.ai/2026/model-server/ovms_demos_rerank.html) — Cohere-compatible API | ||
| - [Image generation](https://docs.openvino.ai/2026/model-server/ovms_demos_image_generation.html) — OpenAI-compatible `/v1/images/generations` | ||
| - [Speech recognition and TTS](https://docs.openvino.ai/2026/model-server/ovms_demos_audio.html) — OpenAI-compatible audio API | ||
| - [GGUF model support](https://docs.openvino.ai/2026/model-server/ovms_demos_gguf.html) | ||
|
|
||
| ### Classic Models & Pipelines | ||
| - TensorFlow, ONNX, PaddlePaddle, OpenVINO IR model formats | ||
| - [MediaPipe graphs](https://docs.openvino.ai/2026/model-server/ovms_docs_mediapipe.html) | ||
| - [Python execution nodes](https://docs.openvino.ai/2026/model-server/ovms_docs_python_support_reference.html) | ||
| - [Dynamic input shapes](https://docs.openvino.ai/2026/model-server/ovms_docs_shape_batch_size_and_layout.html) | ||
|
|
||
| ### Deployment & Integration | ||
| - [Docker](docs/deploying_server_docker.md), [bare metal (Linux & Windows)](docs/deploying_server_baremetal.md), [Kubernetes / OpenShift](docs/deploying_server_kubernetes.md) | ||
| - [Model repository](https://docs.openvino.ai/2026/model-server/ovms_docs_models_repository.html): local storage, S3, GCS, Azure Blob, HuggingFace Hub | ||
| - [Model versioning](https://docs.openvino.ai/2026/model-server/ovms_docs_model_version_policy.html) and [hot-reload](https://docs.openvino.ai/2026/model-server/ovms_docs_online_config_changes.html) | ||
| - [Prometheus-compatible metrics](https://docs.openvino.ai/2026/model-server/ovms_docs_metrics.html) | ||
| - [gRPC streaming](https://docs.openvino.ai/2026/model-server/ovms_docs_streaming_endpoints.html) | ||
| - [C API](https://docs.openvino.ai/2026/model-server/ovms_docs_c_api.html) for embedding OVMS in native applications | ||
|
|
||
| ### Hardware Acceleration | ||
| - CPU (x86, including Xeon), Intel integrated and discrete GPU, NPU | ||
| - See [supported accelerators](https://docs.openvino.ai/2026/model-server/ovms_docs_accelerators.html) | ||
|
|
||
| [→ Full feature list](https://docs.openvino.ai/2026/model-server/ovms_docs_features.html) | ||
|
|
||
| [Preparing model repository](https://docs.openvino.ai/2026/model-server/ovms_docs_models_repository.html) | ||
| --- | ||
|
|
||
| [Deployment](https://docs.openvino.ai/2026/model-server/ovms_docs_deploying_server.html) | ||
| ## Documentation | ||
|
|
||
| [Writing client code](https://docs.openvino.ai/2026/model-server/ovms_docs_server_app.html) | ||
| | Topic | Link | | ||
| |---|---| | ||
| | Deployment | [Deploying the server](https://docs.openvino.ai/2026/model-server/ovms_docs_deploying_server.html) | | ||
| | Model repository | [Preparing models](https://docs.openvino.ai/2026/model-server/ovms_docs_models_repository.html) | | ||
| | Client libraries | [Writing client code](https://docs.openvino.ai/2026/model-server/ovms_docs_server_app.html) | | ||
| | Demos & examples | [Demos](https://docs.openvino.ai/2026/model-server/ovms_docs_demos.html) | | ||
| | Release notes | [GitHub Releases](https://github.com/openvinotoolkit/model_server/releases) | | ||
|
|
||
| [Demos](https://docs.openvino.ai/2026/model-server/ovms_docs_demos.html) | ||
| --- | ||
|
|
||
| ## Get the Server | ||
|
|
||
| **Docker images**: | ||
| ```text | ||
| docker pull openvino/model_server:latest # Intel CPU | ||
| docker pull openvino/model_server:latest-gpu # Intel CPU,GPU,NPU | ||
|
|
||
| ## References | ||
| docker pull openvino/model_server:weekly # pre-production version with all accelerators enabled | ||
| ``` | ||
|
|
||
| * [OpenVINO™](https://software.intel.com/en-us/openvino-toolkit) | ||
| - [Docker Hub](https://hub.docker.com/r/openvino/model_server) | ||
| - [Red Hat Ecosystem Catalog](https://catalog.redhat.com/software/containers/intel/openvino-model-server/607833052937385fc98515de) | ||
|
|
||
| * [ADVANCING GENAI WITH CPU OPTIMIZATION](https://cdrdv2-public.intel.com/864404/vFINAL_Intel%20SLM%20Whitepaper.pdf) | ||
| **Binary official packages** (Linux & Windows): [GitHub Releases](https://github.com/openvinotoolkit/model_server/releases) | ||
|
|
||
| * [Manage deep learning models with OpenVINO Model Server](https://developers.redhat.com/articles/2024/07/03/manage-deep-learning-models-openvino-model-server#) | ||
|
|
||
| * [RAG building blocks made easy and affordable with OpenVINO Model Server](https://medium.com/openvino-toolkit/rag-building-blocks-made-easy-and-affordable-with-openvino-model-server-e7b03da5012b) | ||
| Binary pre-production packages (Linux & Windows): [storage.openvinotoolkit.org](https://storage.openvinotoolkit.org/repositories/openvino_model_server/packages/weekly/) | ||
|
|
||
| * [Simple deployment with KServe API](https://blog.openvino.ai/blog-posts/kserve-api) | ||
| --- | ||
|
|
||
| * [Benchmarking results](https://docs.openvino.ai/2026/about-openvino/performance-benchmarks.html) | ||
| ## Contributing | ||
|
|
||
| Contributions are welcome! Please open an issue or pull request on GitHub. | ||
| See [security policy](security.md) for responsible disclosure. | ||
|
|
||
| ## Contact | ||
| --- | ||
|
|
||
| If you have a question, a feature request, or a bug report, feel free to submit a Github issue. | ||
| ## References | ||
|
|
||
| - [OpenVINO Toolkit](https://software.intel.com/en-us/openvino-toolkit) | ||
| - [Performance benchmarks](https://docs.openvino.ai/2026/about-openvino/performance-benchmarks.html) | ||
| - [GenAI with CPU optimization — Intel whitepaper](https://cdrdv2-public.intel.com/864404/vFINAL_Intel%20SLM%20Whitepaper.pdf) | ||
| - [RAG with OpenVINO Model Server — blog post](https://medium.com/openvino-toolkit/rag-building-blocks-made-easy-and-affordable-with-openvino-model-server-e7b03da5012b) | ||
| - [AIPC turned into a mighty assistant](https://medium.com/openvino-toolkit/ai-pc-turned-into-a-mighty-ai-assistant-with-local-models-and-openvino-model-server-1f41913252c9) | ||
|
|
||
| --- | ||
|
|
||
| \* Other names and brands may be claimed as the property of others. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe use: mkdir -p ${HOME}/models?