diff --git a/README.md b/README.md index 087a8a8c99..d17bfeb3bf 100644 --- a/README.md +++ b/README.md @@ -1,88 +1,197 @@ # OpenVINO™ Model Server -Model Server hosts models and makes them accessible to software components over standard network protocols: a client sends a request to the model server, which performs model inference and sends a response back to the client. Model Server offers many advantages for efficient model deployment: -- Remote inference enables using lightweight clients with only the necessary functions to perform API calls to edge or cloud deployments. -- Applications are independent of the model framework, hardware device, and infrastructure. -- Client applications in any programming language that supports REST or gRPC calls can be used to run inference remotely on the model server. -- Clients require fewer updates since client libraries change very rarely. -- Model topology and weights are not exposed directly to client applications, making it easier to control access to the model. -- Ideal architecture for microservices-based applications and deployments in cloud environments – including Kubernetes and OpenShift clusters. -- Efficient resource utilization with horizontal and vertical inference scaling. +**High-performance model serving for Generative AI and classic deep learning — powered by [OpenVINO](https://github.com/openvinotoolkit/openvino) and optimized for Intel hardware.** -![OVMS diagram](docs/ovms_diagram.png) - -OpenVINO™ Model Server (OVMS) is a high-performance system for serving models. Implemented in C++ for scalability and optimized for deployment on Intel architectures. It uses the [generative API](https://docs.openvino.ai/2026/model-server/ovms_docs_clients_genai.html) like OpenAI and Cohere, [KServe](https://docs.openvino.ai/2026/model-server/ovms_docs_clients_kfs.html) while applying OpenVINO for inference execution. Inference service is provided via gRPC or REST API, making deploying new algorithms and AI experiments easy. +[![Apache License](https://img.shields.io/badge/license-Apache%202.0-green.svg)](https://github.com/openvinotoolkit/model_server/blob/main/LICENSE) +[![Docker Pulls](https://img.shields.io/docker/pulls/openvino/model_server.svg)](https://hub.docker.com/r/openvino/model_server) +[![GitHub Release](https://img.shields.io/github/v/release/openvinotoolkit/model_server)](https://github.com/openvinotoolkit/model_server/releases) +[![Platform](https://img.shields.io/badge/platform-Linux%20%7C%20Windows-blue)](https://docs.openvino.ai/2026/model-server/ovms_docs_deploying_server.html) -![OVMS picture](docs/ovms_high_level.png) - -The models used by the server can be stored locally, hosted remotely by object storage services or pulled from HuggingFace Hub. For more details, refer to [Preparing Model Repository](https://docs.openvino.ai/2026/model-server/ovms_docs_models_repository.html) and [Deployment](https://docs.openvino.ai/2026/model-server/ovms_docs_deploying_server.html) documentation. -Model server works inside Docker containers, Bare Metal and in Kubernetes environment. +--- -Start using OpenVINO Model Server with a fast-forward serving example from the [QuickStart guide](https://docs.openvino.ai/2026/model-server/ovms_docs_quick_start_guide.html) or [LLM QuickStart guide](https://docs.openvino.ai/2026/model-server/ovms_docs_llm_quickstart.html). +## What is OVMS? -Read [release notes](https://github.com/openvinotoolkit/model_server/releases) to find out what’s new. +OpenVINO Model Server (OVMS) is a production-grade, C++ inference server that exposes ML models over standard network APIs. It serves both **Generative AI**, **Agentic** workloads (LLMs, VLMs, image generation, audio) and **classic deep learning** models (object detection, classification, OCR, and more). -### Key features: -- **[NEW]** [Speech Generation and Speech Recognition with OpenAI API](https://docs.openvino.ai/2026/model-server/ovms_demos_audio.html) -- **[NEW]** [Support for AI agents](https://docs.openvino.ai/2026/model-server/ovms_demos_continuous_batching_agent.html) -- **[NEW]** [Image generation compatible with OpenAI API](https://docs.openvino.ai/2026/model-server/ovms_demos_image_generation.html) -- Native Windows support. Check updated [deployment guide](https://docs.openvino.ai/2026/model-server/ovms_docs_deploying_server_baremetal.html) -- [Text Embeddings compatible with OpenAI API](https://docs.openvino.ai/2026/model-server/ovms_demos_embeddings.html) -- [Reranking compatible with Cohere API](https://docs.openvino.ai/2026/model-server/ovms_demos_rerank.html) -- [Efficient Text Generation via OpenAI API](https://docs.openvino.ai/2026/model-server/ovms_demos_continuous_batching.html) -- [Python code execution](docs/python_support/reference.md) -- [gRPC streaming](docs/streaming_endpoints.md) -- [MediaPipe graphs serving](docs/mediapipe.md) -- Model management - including [model versioning](docs/model_version_policy.md) and [model updates in runtime](docs/online_config_changes.md) -- [Dynamic model inputs](docs/shape_batch_size_and_layout.md) -- [Directed Acyclic Graph Scheduler](docs/dag_scheduler.md) along with [custom nodes in DAG pipelines](docs/custom_node_development.md) -- [Metrics](docs/metrics.md) - metrics compatible with Prometheus standard -- Support for multiple frameworks, such as TensorFlow, PaddlePaddle and ONNX -- Support for [AI accelerators](./docs/accelerators.md) +- **OpenAI-compatible API** for text generation, embeddings, image generation, and audio +- **KServe** APIs for classic model inference +- **Runs anywhere** — Docker, bare metal, Kubernetes/OpenShift, Windows +- **Intel-optimized** — CPU, GPU, NPU acceleration via OpenVINO -Check full list of [features](./docs/features.md) +![OVMS diagram](docs/ovms_diagram.png) -**Note:** OVMS has been tested on RedHat, Ubuntu and Windows. -Public docker images are stored in: -- [Dockerhub](https://hub.docker.com/r/openvino/model_server) -- [RedHat Ecosystem Catalog](https://catalog.redhat.com/software/containers/intel/openvino-model-server/607833052937385fc98515de) -Binary packages for Linux and Windows are on [Github](https://github.com/openvinotoolkit/model_server/releases) +--- -## Run OpenVINO Model Server +## Quick Start + +### Serve an LLM with OpenAI-compatible API + +**On Linux (Docker):** +```bash +# Model is downloaded automatically from HuggingFace +docker run --rm -p 8000:8000 \ + openvino/model_server:latest \ + --source_model OpenVINO/Qwen3-4B-int4-ov \ + --model_repository_path /tmp/models \ + --rest_port 8000 +``` +> For GPU acceleration, use the `latest-gpu` image tag and pass `--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)` to expose the Intel GPU device. + +**On Windows (binary package):** +```bat +mkdir c:\models +ovms.exe --source_model OpenVINO/Qwen3-4B-int4-ov --model_repository_path c:\models --rest_port 8000 +``` + +**Query the model:** +```console +pip install openai +``` +```python +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused") +stream = client.chat.completions.create( + model="OpenVINO/Qwen3-4B-int4-ov", + messages=[{"role": "user", "content": "What are the 3 main tourist attractions in Paris?"}], + stream=True, + extra_body={"chat_template_kwargs": {"enable_thinking": False}} +) +for chunk in stream: + if chunk.choices[0].delta.content: + print(chunk.choices[0].delta.content, end="", flush=True) +``` +> [LLM QuickStart](https://docs.openvino.ai/2026/model-server/ovms_docs_llm_quickstart.html) + + + +### Serve a Classic Model with KServe API + +**Download the model:** +```console +curl -L https://huggingface.co/OpenVINO/resnet50-int8-ov/resolve/main/resnet50.bin -O +curl -L https://huggingface.co/OpenVINO/resnet50-int8-ov/resolve/main/resnet50.xml -O +``` + +**On Linux (Docker):** +```bash +docker run --rm -d -u $(id -u) -v ${PWD}:/models -p 9000:9000 \ + openvino/model_server:latest \ + --model_name resnet --model_path /models/resnet50.xml \ + --mean "[123.675,116.28,103.53]" --scale "[58.395,57.12,57.375]" --layout "NHWC:NCHW" \ + --port 9000 +``` +> For GPU acceleration, use the `latest-gpu` image tag and pass `--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)` to expose the Intel GPU device. + +**Windows (binary package):** +```bat +ovms --model_name resnet --model_path resnet50.xml --mean "[123.675,116.28,103.53]" --scale "[58.395,57.12,57.375]" --layout "NHWC:NCHW" --port 9000 +``` + +Run inference with a sample client +```console +pip install numpy tritonclient[grpc] +curl -L -o image.jpeg https://github.com/openvinotoolkit/model_server/blob/main/demos/common/static/images/bee.jpeg?raw=true +``` +```python +import numpy as np +import tritonclient.grpc as grpcclient +with open("image.jpeg", "rb") as f: + image_bytes = f.read() +client = grpcclient.InferenceServerClient(url="localhost:9000") +inputs = [grpcclient.InferInput("image", [1], "BYTES")] +inputs[0].set_data_from_numpy(np.array([image_bytes], dtype=object)) +outputs = [grpcclient.InferRequestedOutput("output")] +result = client.infer(model_name="resnet", inputs=inputs, outputs=outputs) +output = result.as_numpy("output") # (1, 1000) FP32 +print("Top-1 class index:", int(np.argmax(output[0]))) +``` + +> [Vision model QuickStart](https://docs.openvino.ai/2026/model-server/ovms_docs_quick_start_guide.html) -A demonstration on how to use OpenVINO Model Server can be found in our [quick-start guide for vision use case](docs/ovms_quickstart.md) and [LLM text generation](docs/llm/quickstart.md). +--- -Check also other instructions: +## Features + +### Generative AI +- [LLM text generation](https://docs.openvino.ai/2026/model-server/ovms_demos_continuous_batching.html) — continuous batching, streaming, structured output, speculative decoding +- [VLM (Vision Language Models)](https://docs.openvino.ai/2026/model-server/ovms_demos_continuous_batching_vlm.html) +- [AI Agents with MCP servers](https://docs.openvino.ai/2026/model-server/ovms_demos_continuous_batching_agent.html) +- [Text embeddings](https://docs.openvino.ai/2026/model-server/ovms_demos_embeddings.html) — OpenAI-compatible `/v1/embeddings` +- [Reranking](https://docs.openvino.ai/2026/model-server/ovms_demos_rerank.html) — Cohere-compatible API +- [Image generation](https://docs.openvino.ai/2026/model-server/ovms_demos_image_generation.html) — OpenAI-compatible `/v1/images/generations` +- [Speech recognition and TTS](https://docs.openvino.ai/2026/model-server/ovms_demos_audio.html) — OpenAI-compatible audio API +- [GGUF model support](https://docs.openvino.ai/2026/model-server/ovms_demos_gguf.html) + +### Classic Models & Pipelines +- TensorFlow, ONNX, PaddlePaddle, OpenVINO IR model formats +- [MediaPipe graphs](https://docs.openvino.ai/2026/model-server/ovms_docs_mediapipe.html) +- [Python execution nodes](https://docs.openvino.ai/2026/model-server/ovms_docs_python_support_reference.html) +- [Dynamic input shapes](https://docs.openvino.ai/2026/model-server/ovms_docs_shape_batch_size_and_layout.html) + +### Deployment & Integration +- [Docker](docs/deploying_server_docker.md), [bare metal (Linux & Windows)](docs/deploying_server_baremetal.md), [Kubernetes / OpenShift](docs/deploying_server_kubernetes.md) +- [Model repository](https://docs.openvino.ai/2026/model-server/ovms_docs_models_repository.html): local storage, S3, GCS, Azure Blob, HuggingFace Hub +- [Model versioning](https://docs.openvino.ai/2026/model-server/ovms_docs_model_version_policy.html) and [hot-reload](https://docs.openvino.ai/2026/model-server/ovms_docs_online_config_changes.html) +- [Prometheus-compatible metrics](https://docs.openvino.ai/2026/model-server/ovms_docs_metrics.html) +- [gRPC streaming](https://docs.openvino.ai/2026/model-server/ovms_docs_streaming_endpoints.html) +- [C API](https://docs.openvino.ai/2026/model-server/ovms_docs_c_api.html) for embedding OVMS in native applications + +### Hardware Acceleration +- CPU (x86, including Xeon), Intel integrated and discrete GPU, NPU +- See [supported accelerators](https://docs.openvino.ai/2026/model-server/ovms_docs_accelerators.html) + +[→ Full feature list](https://docs.openvino.ai/2026/model-server/ovms_docs_features.html) -[Preparing model repository](https://docs.openvino.ai/2026/model-server/ovms_docs_models_repository.html) +--- -[Deployment](https://docs.openvino.ai/2026/model-server/ovms_docs_deploying_server.html) +## Documentation -[Writing client code](https://docs.openvino.ai/2026/model-server/ovms_docs_server_app.html) +| Topic | Link | +|---|---| +| Deployment | [Deploying the server](https://docs.openvino.ai/2026/model-server/ovms_docs_deploying_server.html) | +| Model repository | [Preparing models](https://docs.openvino.ai/2026/model-server/ovms_docs_models_repository.html) | +| Client libraries | [Writing client code](https://docs.openvino.ai/2026/model-server/ovms_docs_server_app.html) | +| Demos & examples | [Demos](https://docs.openvino.ai/2026/model-server/ovms_docs_demos.html) | +| Release notes | [GitHub Releases](https://github.com/openvinotoolkit/model_server/releases) | -[Demos](https://docs.openvino.ai/2026/model-server/ovms_docs_demos.html) +--- +## Get the Server +**Docker images**: +```text +docker pull openvino/model_server:latest # Intel CPU +docker pull openvino/model_server:latest-gpu # Intel CPU,GPU,NPU -## References +docker pull openvino/model_server:weekly # pre-production version with all accelerators enabled +``` -* [OpenVINO™](https://software.intel.com/en-us/openvino-toolkit) +- [Docker Hub](https://hub.docker.com/r/openvino/model_server) +- [Red Hat Ecosystem Catalog](https://catalog.redhat.com/software/containers/intel/openvino-model-server/607833052937385fc98515de) -* [ADVANCING GENAI WITH CPU OPTIMIZATION](https://cdrdv2-public.intel.com/864404/vFINAL_Intel%20SLM%20Whitepaper.pdf) +**Binary official packages** (Linux & Windows): [GitHub Releases](https://github.com/openvinotoolkit/model_server/releases) -* [Manage deep learning models with OpenVINO Model Server](https://developers.redhat.com/articles/2024/07/03/manage-deep-learning-models-openvino-model-server#) -* [RAG building blocks made easy and affordable with OpenVINO Model Server](https://medium.com/openvino-toolkit/rag-building-blocks-made-easy-and-affordable-with-openvino-model-server-e7b03da5012b) +Binary pre-production packages (Linux & Windows): [storage.openvinotoolkit.org](https://storage.openvinotoolkit.org/repositories/openvino_model_server/packages/weekly/) -* [Simple deployment with KServe API](https://blog.openvino.ai/blog-posts/kserve-api) +--- -* [Benchmarking results](https://docs.openvino.ai/2026/about-openvino/performance-benchmarks.html) +## Contributing +Contributions are welcome! Please open an issue or pull request on GitHub. +See [security policy](security.md) for responsible disclosure. -## Contact +--- -If you have a question, a feature request, or a bug report, feel free to submit a Github issue. +## References +- [OpenVINO Toolkit](https://software.intel.com/en-us/openvino-toolkit) +- [Performance benchmarks](https://docs.openvino.ai/2026/about-openvino/performance-benchmarks.html) +- [GenAI with CPU optimization — Intel whitepaper](https://cdrdv2-public.intel.com/864404/vFINAL_Intel%20SLM%20Whitepaper.pdf) +- [RAG with OpenVINO Model Server — blog post](https://medium.com/openvino-toolkit/rag-building-blocks-made-easy-and-affordable-with-openvino-model-server-e7b03da5012b) +- [AIPC turned into a mighty assistant](https://medium.com/openvino-toolkit/ai-pc-turned-into-a-mighty-ai-assistant-with-local-models-and-openvino-model-server-1f41913252c9) --- + \* Other names and brands may be claimed as the property of others. diff --git a/ci/build_test_OnCommit.groovy b/ci/build_test_OnCommit.groovy index 0435645689..5573d55ce6 100644 --- a/ci/build_test_OnCommit.groovy +++ b/ci/build_test_OnCommit.groovy @@ -340,7 +340,7 @@ pipeline { sh "pwd" def pwd = sh(returnStdout:true, script: "pwd").strip() def ovms_c_repo_path = sh(returnStdout:true, script: "cd .. && pwd").strip() - def test_doc_files_str = test_doc_files_linux.split('\n').join(' or ') + def test_doc_files_str = test_doc_files_linux.split('\n').collect { 'U-' + it }.join(' or ') sh "make create-venv && rm -f tests/functional && ln -s ${pwd}/../tests/functional tests/functional" def cmd_venv_activate = ". .venv/bin/activate" def cmd_export = "export TT_OVMS_C_REPO_PATH=../ && export TT_RUN_REGRESSION_TESTS=True && export TT_REGRESSION_WEEKLY_TESTS=True && export TT_TARGET_DEVICE=CPU,GPU,NPU && export TT_ENABLE_UAT_TESTS=True && export TT_ENABLE_SMOKE_TESTS=False && export TT_OVMS_C_REPO_PATH=${ovms_c_repo_path} && export TT_LOGGING_LEVEL_OVMS=DEBUG && export TT_WAIT_FOR_MESSAGES_TIMEOUT=1500 && export CORE_BRANCH=${env.CHANGE_BRANCH ?: 'main'}" @@ -403,7 +403,7 @@ pipeline { script { dir ('documentation_tests') { checkout scmGit(branches: [[name: validation_branch]], userRemoteConfigs: [[credentialsId: 'workflow-lab', url: 'https://github.com/intel-innersource/frameworks.ai.openvino.model-server.tests.git']]) - def test_doc_files_str = test_doc_files_windows.split('\n').join(' or ') + def test_doc_files_str = test_doc_files_windows.split('\n').collect { 'U-' + it }.join(' or ') def current_path = bat(returnStdout: true, script: 'cd').trim().split('\n').last().trim() def ovms_c_repo_path = bat(returnStdout: true, script: 'cd .. && cd').trim().split('\n').last().trim() def cmd_link_ovms = "(if exist ${current_path}\\tests\\functional rmdir ${current_path}\\tests\\functional) && mklink /D ${current_path}\\tests\\functional ${ovms_c_repo_path}\\tests\\functional" diff --git a/demos/README.md b/demos/README.md index a67370ab07..f14bd6fff4 100644 --- a/demos/README.md +++ b/demos/README.md @@ -5,88 +5,86 @@ maxdepth: 1 hidden: --- -ovms_demos_continuous_batching -ovms_demos_integration_with_open_webui -ovms_demos_code_completion_vsc -ovms_demos_audio -ovms_demos_rerank -ovms_demos_embeddings -ovms_demos_continuous_batching_vlm -ovms_demos_image_generation -ovms_demo_clip_image_classification -ovms_demo_age_gender_guide -ovms_demo_face_detection -ovms_demo_capi_inference_demo -ovms_docs_demo_mediapipe_image_classification -ovms_docs_demo_mediapipe_multi_model -ovms_docs_demo_mediapipe_object_detection -ovms_docs_demo_mediapipe_holistic -ovms_docs_demo_mediapipe_iris -ovms_docs_image_classification -ovms_demo_using_onnx_model -ovms_demo_tf_classification -ovms_demo_person_vehicle_bike_detection -ovms_demo_real_time_stream_analysis -ovms_demo_using_paddlepaddle_model -ovms_demo_bert -ovms_demo_universal-sentence-encoder -ovms_string_output_model_demo -ovms_demos_gguf +Text generation +Image generation +Audio +Text Embeddings +Text Reranking +Classic models +MediaPipe +Python Node +Integrations ``` -OpenVINO Model Server demos have been created to showcase the usage of the model server as well as demonstrate it’s capabilities. -### Check Out New Generative AI Demos +## Text Generation | Demo | Description | |---|---| -|[AI Agents with MCP servers and serving language models](./continuous_batching/agentic_ai/README.md)|OpenAI agents with MCP servers and serving LLM models| -|[Integration with Open WebUI](integration_with_OpenWebUI/README.md)|Using OpenWeb UI with OVMS as inference provider. Shows text and image generation as well as usage with RAG and tools| -|[LLM Text Generation with continuous batching](continuous_batching/README.md)|Generate text with LLM models and continuous batching pipeline| -|[VLM Text Generation with continuous batching](continuous_batching/vlm/README.md)|Generate text with VLM models and continuous batching pipeline| -|[OpenAI API text embeddings ](embeddings/README.md)|Get text embeddings via endpoint compatible with OpenAI API| -|[Reranking with Cohere API](rerank/README.md)| Rerank documents via endpoint compatible with Cohere| -|[RAG with OpenAI API endpoint and langchain](https://github.com/openvinotoolkit/model_server/blob/releases/2026/3/demos/continuous_batching/rag/rag_demo.ipynb)| Example how to use RAG with model server endpoints| -|[LLM on NPU](./llm_npu/README.md)| Generate text with LLM models and NPU acceleration| -|[VLM on NPU](./vlm_npu/README.md)| Generate text with VLM models and NPU acceleration| -|[Long context LLMs](./continuous_batching/long_context/README.md)| Recommendations for handling very long context in LLM models| -|[Visual Studio Code assistant](./code_local_assistant/README.md)|Use Continue extension to Visual Studio Code with local OVMS serving| -|[Image Generation](image_generation/README.md)|Generate images| -|[GGUF models support](gguf/README.md)|Serve GGUF models with OVMS| +|[LLM Text Generation](continuous_batching/README.md)|Generate text with LLM models and continuous batching pipeline.| +|[VLM Text Generation](continuous_batching/vlm/README.md)|Generate text with VLM models and continuous batching pipeline.| +|[AI Agents with MCP servers](./continuous_batching/agentic_ai/README.md)|OpenAI agents with MCP servers and serving LLM models.| +|[RAG with OpenAI API endpoint and langchain](continuous_batching/rag/README.md)|Example how to use RAG with model server endpoints.| +|[Long context LLMs](./continuous_batching/long_context/README.md)|Recommendations for handling very long context in LLM models.| +|[Structured output](./continuous_batching/structured_output/README.md)|Generate structured (JSON) output from LLM models.| +|[Speculative decoding](./continuous_batching/speculative_decoding/README.md)|Speed up LLM inference with speculative decoding.| +|[LLM on NPU](./llm_npu/README.md)|Generate text with LLM models and NPU acceleration.| +|[Scaling on multi CPU and GPU](./continuous_batching/scaling/README.md)|Scale LLM serving across multiple CPUs and GPUs.| +|[Loading models in GGUF](gguf/README.md)|Serve GGUF models with OVMS.| -Check out the list below to see complete step-by-step examples of using OpenVINO Model Server with real world use cases: +## Image Generation +| Demo | Description | +|---|---| +|[Image Generation](image_generation/README.md)|Generate images with diffusion models.| + +## Audio +| Demo | Description | +|---|---| +|[Audio demos](audio/README.md)|Text-to-speech and automatic speech recognition demos.| + +## Text Embeddings +| Demo | Description | +|---|---| +|[OpenAI API text embeddings](embeddings/README.md)|Get text embeddings via endpoint compatible with OpenAI API.| + +## Text Reranking +| Demo | Description | +|---|---| +|[Reranking with Cohere API](rerank/README.md)|Rerank documents via endpoint compatible with Cohere.| -## With Traditional Models +## Classic Models | Demo | Description | |---|---| |[Image Classification](image_classification/python/README.md)|Run prediction on a JPEG image using image classification model via gRPC API.| -|[Using ONNX Model](using_onnx_model/python/README.md)|Run prediction on a JPEG image using image classification ONNX model via gRPC API in two preprocessing variants. This demo uses [pipeline](../docs/dag_scheduler.md) with [image_transformation custom node](https://github.com/openvinotoolkit/model_server/tree/releases/2026/3/src/custom_nodes/image_transformation). | -|[Using TensorFlow Model](image_classification_using_tf_model/python/README.md)|Run image classification using directly imported TensorFlow model. | -|[Age gender recognition](age_gender_recognition/python/README.md) | Run prediction on a JPEG image using age gender recognition model via gRPC API.| +|[Using ONNX Model](using_onnx_model/python/README.md)|Run prediction on a JPEG image using image classification ONNX model via gRPC API in two preprocessing variants. This demo uses [pipeline](../docs/dag_scheduler.md) with [image_transformation custom node](https://github.com/openvinotoolkit/model_server/tree/main/src/custom_nodes/image_transformation).| +|[Using TensorFlow Model](image_classification_using_tf_model/python/README.md)|Run image classification using directly imported TensorFlow model.| +|[Classification with PaddlePaddle](classification_using_paddlepaddle_model/python/README.md)|Perform classification on an image with a PaddlePaddle model.| +|[Age gender recognition](age_gender_recognition/python/README.md)|Run prediction on a JPEG image using age gender recognition model via gRPC API.| |[Face Detection](face_detection/python/README.md)|Run prediction on a JPEG image using face detection model via gRPC API.| -|[Classification with PaddlePaddle](classification_using_paddlepaddle_model/python/README.md)| Perform classification on an image with a PaddlePaddle model. | -|[Natural Language Processing with BERT](bert_question_answering/python/README.md)|Provide a knowledge source and a query and use BERT model for question answering use case via gRPC API. This demo uses dynamic shape feature. | -|[Using inputs data in string format with universal-sentence-encoder model](universal-sentence-encoder/README.md)| Handling AI model with text as the model input. | |[Person, Vehicle, Bike Detection](person_vehicle_bike_detection/python/README.md)|Run prediction on a video file or camera stream using person, vehicle, bike detection model via gRPC API.| +|[Using input strings](universal-sentence-encoder/README.md)|Handling AI model with text as the model input.| +|[Using output strings](image_classification_with_string_output/README.md)|Handling AI model with string output.| +|[Natural Language Processing with BERT](bert_question_answering/python/README.md)|Provide a knowledge source and a query and use BERT model for question answering via gRPC API. This demo uses dynamic shape feature.| |[Benchmark App](benchmark/python/README.md)|Generate traffic and measure performance of the model served in OpenVINO Model Server.| -## With Python Nodes +## MediaPipe | Demo | Description | |---|---| -|[CLIP image classification](python_demos/clip_image_classification/README.md) | Classify image according to provided labels using CLIP model embedded in a multi-node MediaPipe graph.| +|[Object Detection](./mediapipe/object_detection/README.md)|A pipeline implementing object detection.| +|[Iris](./mediapipe/iris_tracking/README.md)|A pipeline implementing iris detection.| +|[Holistic](./mediapipe/holistic_tracking/README.md)|A complex pipeline linking several image analytical models and image transformations.| +|[Realtime Stream Analysis](real_time_stream_analysis/python/README.md)|Analyze RTSP video stream in real time with generic application template for custom pre and post processing routines.| +|[Image classification](./mediapipe/image_classification/README.md)|Basic example with a single inference node.| +|[Chain of models](./mediapipe/multi_model_graph/README.md)|A chain of models in a graph.| +|[CLIP image classification](python_demos/clip_image_classification/README.md)|Classify image according to provided labels using CLIP model embedded in a multi-node MediaPipe graph.| -## With MediaPipe Graphs +## Python Node | Demo | Description | |---|---| -|[Real Time Stream Analysis](real_time_stream_analysis/python/README.md)| Analyze RTSP video stream in real time with generic application template for custom pre and post processing routines as well as simple results visualizer for displaying predictions in the browser. | -|[Image classification](./mediapipe/image_classification/README.md)| Basic example with a single inference node. | -|[Chain of models](./mediapipe/image_classification/README.md)| A chain of models in a graph. | -|[Object detection](./mediapipe/object_detection/README.md)| A pipeline implementing object detection | -|[Iris demo](./mediapipe/object_detection/README.md)| A pipeline implementing iris detection | -|[Holistic demo](./mediapipe/holistic_tracking/README.md)| A complex pipeline linking several image analytical models and image transformations | +|[OpenClip with python execution](./python_demos/clip_image_classification/README.md)|A pipeline implementing OpenClip classification in Python Node.| -## With C++ Client +## Integrations | Demo | Description | |---|---| -|[C API applications](c_api_minimal_app/README.md)|How to use C API from the OpenVINO Model Server to create C and C++ application.| - +|[Integration with Open WebUI](integration_with_OpenWebUI/README.md)|Using Open WebUI with OVMS as inference provider. Shows text and image generation as well as usage with RAG and tools.| +|[Visual Studio Code assistant](./code_local_assistant/README.md)|Use Continue or Cline extension to Visual Studio Code with local OVMS serving.| diff --git a/demos/classic_models_demos.md b/demos/classic_models_demos.md new file mode 100644 index 0000000000..9462a754d5 --- /dev/null +++ b/demos/classic_models_demos.md @@ -0,0 +1,17 @@ +# Classic models {#ovms_demos_classic_models} + +```{toctree} +--- +maxdepth: 1 +--- + +Image classification +ONNX +TensorFlow +PaddlePaddle +Age gender classification +Face detection +Person Detection +Using input strings +Using output strings +``` diff --git a/demos/continuous_batching/README.md b/demos/continuous_batching/README.md index af35fde575..793bf7949d 100644 --- a/demos/continuous_batching/README.md +++ b/demos/continuous_batching/README.md @@ -1,20 +1,5 @@ # LLM models via OpenAI API {#ovms_demos_continuous_batching} -```{toctree} ---- -maxdepth: 1 -hidden: ---- -ovms_demos_continuous_batching_agent -ovms_demos_continuous_batching_rag -ovms_demos_continuous_batching_scaling -ovms_demos_continuous_batching_speculative_decoding -ovms_structured_output -ovms_demo_long_context -ovms_demos_llm_npu -ovms_demos_continuous_batching_accuracy -``` - This demo shows how to deploy LLM models in the OpenVINO Model Server using continuous batching and paged attention algorithms. Text generation use case is exposed via OpenAI API `chat/completions`, `completions` and `responses` endpoints. That makes it easy to use and efficient especially on Intel® Xeon® processors and ARC GPUs. diff --git a/demos/continuous_batching/agentic_ai/README.md b/demos/continuous_batching/agentic_ai/README.md index ec01ff475f..f040c19df5 100644 --- a/demos/continuous_batching/agentic_ai/README.md +++ b/demos/continuous_batching/agentic_ai/README.md @@ -177,46 +177,6 @@ Let me know if you'd like forecast details or anything else! ::: :::: -### Deploying on Windows with NPU - -::::{tab-set} -:::{tab-item} Qwen3-8B -:sync: Qwen3-8B -Pull and start OVMS: -```bat -ovms.exe --rest_port 8000 --source_model OpenVINO/Qwen3-8B-int4-cw-ov --model_repository_path c:\models --tool_parser hermes3 --target_device NPU --task text_generation --cache_dir .cache --max_prompt_len 8000 -``` - -Use MCP server: -```bat -python openai_agent.py --query "What is the current weather in Tokyo?" --model OpenVINO/Qwen3-8B-int4-cw-ov --base-url http://localhost:8000/v3 --mcp-server-url http://localhost:8080/sse --mcp-server weather -``` - -Exemplary output: -```text -The current weather in Tokyo is overcast with a temperature of 9.4°C (feels like 6.4°C). The relative humidity is at 42%, and the dew point is at -2.9°C. The wind is blowing from the NE at 3.6 km/h, with gusts up to 24.8 km/h. The atmospheric pressure is 1018.9 hPa, and there is 84% cloud cover. Visibility is 24.1 km. -``` -::: -:::{tab-item} Qwen3-4B -:sync: Qwen3-4B -Pull and start OVMS: -```bat -ovms.exe --rest_port 8000 --source_model FluidInference/qwen3-4b-int4-ov-npu --model_repository_path c:\models --tool_parser hermes3 --target_device NPU --task text_generation --cache_dir .cache --max_prompt_len 8000 -``` - -Use MCP server: -```bat -python openai_agent.py --query "What is the current weather in Tokyo?" --model OpenVINO/Qwen3-8B-int4-cw-ov --base-url http://localhost:8000/v3 --mcp-server-url http://localhost:8080/sse --mcp-server weather -``` - -Exemplary output: -```text -The current weather in Tokyo is overcast with a temperature of 9.4°C (feels like 6.4°C). The relative humidity is at 42%, and the dew point is at -2.9°C. The wind is blowing from the NE at 3.6 km/h, with gusts up to 24.8 km/h. The atmospheric pressure is 1018.9 hPa, and there is 84% cloud cover. Visibility is 24.1 km. -``` -::: -:::: - -> **Note:** Setting the `--max_prompt_len` parameter too high may lead to performance degradation. It is recommended to use the smallest value that meets your requirements. ### Deploying in a docker container on CPU @@ -512,57 +472,6 @@ Let me know if you'd like forecast details or anything else! ::: :::: -### Deploying in a docker container on NPU - -The case of NPU is similar to GPU, but `--device` should be set to `/dev/accel`, `--group-add` parameter should be the same. -Running `docker run` command, use the image with GPU support. Export the models with precision matching the [NPU capacity](https://docs.openvino.ai/2026/openvino-workflow-generative/inference-with-genai/inference-with-genai-on-npu.html) and adjust pipeline configuration. -It can be applied using the commands below: - -::::{tab-set} -:::{tab-item} Qwen3-8B -:sync: Qwen3-8B -Pull and start OVMS: -```bash -mkdir -p ${HOME}/models -docker run -d --user $(id -u):$(id -g) --rm -p 8000:8000 -v ${HOME}/models:/models --device /dev/accel --group-add=$(stat -c "%g" /dev/dri/render* | head -1) openvino/model_server:weekly \ ---rest_port 8000 --model_repository_path /models --source_model OpenVINO/Qwen3-8B-int4-cw-ov --tool_parser hermes3 --target_device NPU --task text_generation --max_prompt_len 8000 -``` - -Use MCP server: -```bash -python openai_agent.py --query "What is the current weather in Tokyo?" --model OpenVINO/Qwen3-8B-int4-cw-ov --base-url http://localhost:8000/v3 --mcp-server-url http://localhost:8080/sse --mcp-server weather -``` - -Exemplary output: -```text -The current weather in Tokyo is overcast with a temperature of 9.4°C (feels like 6.4°C). The relative humidity is at 42%, and the dew point is at -2.9°C. The wind is blowing from the NE at 3.6 km/h with gusts up to 24.8 km/h. The atmospheric pressure is 1018.9 hPa with 84% cloud cover, and the visibility is 24.1 km. -``` -::: -:::{tab-item} Qwen3-4B -:sync: Qwen3-4B -Pull and start OVMS: -```bash -mkdir -p ${HOME}/models -docker run -d --user $(id -u):$(id -g) --rm -p 8000:8000 -v ${HOME}/models:/models --device /dev/accel --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) openvino/model_server:weekly \ ---rest_port 8000 --model_repository_path /models --source_model FluidInference/qwen3-4b-int4-ov-npu --tool_parser hermes3 --target_device NPU --task text_generation --max_prompt_len 8000 -``` - -Use MCP server: -```bash -python openai_agent.py --query "What is the current weather in Tokyo?" --model FluidInference/qwen3-4b-int4-ov-npu --base-url http://localhost:8000/v3 --mcp-server-url http://localhost:8080/sse --mcp-server weather --stream -``` - -Exemplary output: -```text -The current weather in Tokyo is overcast with a temperature of 9.4°C (feels like 6.4°C). The relative humidity is at 42%, and the dew point is at -2.9°C. The wind is blowing from the NE at 3.6 km/h with gusts up to 24.8 km/h. The atmospheric pressure is 1018.9 hPa with 84% cloud cover, and the visibility is 24.1 km. -``` -::: -:::: - -> **Note:** The tool checking the weather forecast in the demo is making a remote call to a REST API server. Make sure you have internet connection and proxy configured while running the agent. - -> **Note:** For more interactive mode you can run the application with streaming enabled by providing `--stream` parameter to the script. - ### Using Llama index agentic framework Pull and start OVMS: diff --git a/demos/continuous_batching/structured_output/README.md b/demos/continuous_batching/structured_output/README.md index 68e9db1049..ecc2ba3bee 100644 --- a/demos/continuous_batching/structured_output/README.md +++ b/demos/continuous_batching/structured_output/README.md @@ -19,14 +19,6 @@ mkdir models docker run --user $(id -u):$(id -g) -d --device /dev/dri --group-add=$(stat -c "%g" /dev/dri/render* | head -1) --rm -p 8000:8000 -v $(pwd)/models:/models:rw openvino/model_server:latest-gpu --source_model OpenVINO/Mistral-7B-Instruct-v0.3-int4-cw-ov --model_repository_path models --task text_generation --rest_port 8000 --target_device GPU ``` ::: -:::{tab-item} With Docker on NPU -**Required:** Docker Engine installed - -```bash -mkdir models -docker run --user $(id -u):$(id -g) -d --device /dev/accel --group-add=$(stat -c "%g" /dev/dri/render* | head -1) --rm -p 8000:8000 -v $(pwd)/models:/models:rw openvino/model_server:latest-gpu --source_model OpenVINO/Mistral-7B-Instruct-v0.3-int4-cw-ov --model_repository_path models --task text_generation --rest_port 8000 --target_device NPU -``` -::: :::{tab-item} With Docker on CPU **Required:** Docker Engine installed @@ -42,13 +34,6 @@ docker run --user $(id -u):$(id -g) -d --rm -p 8000:8000 -v $(pwd)/models:/model ovms.exe --source_model OpenVINO/Mistral-7B-Instruct-v0.3-int4-cw-ov --model_repository_path models --rest_port 8000 --target_device GPU --task text_generation ``` ::: -:::{tab-item} On Baremetal Host and NPU -**Required:** OpenVINO Model Server package - see [deployment instructions](../../../docs/deploying_server_baremetal.md) for details. - -```bat -ovms.exe --source_model OpenVINO/Mistral-7B-Instruct-v0.3-int4-cw-ov --model_repository_path models --rest_port 8000 --target_device NPU --task text_generation -``` -::: :::{tab-item} On Baremetal Host and CPU **Required:** OpenVINO Model Server package - see [deployment instructions](../../../docs/deploying_server_baremetal.md) for details. diff --git a/demos/continuous_batching/vlm/README.md b/demos/continuous_batching/vlm/README.md index c47f3d8751..322209fc25 100644 --- a/demos/continuous_batching/vlm/README.md +++ b/demos/continuous_batching/vlm/README.md @@ -1,13 +1,5 @@ # VLM models via OpenAI API {#ovms_demos_continuous_batching_vlm} -```{toctree} ---- -maxdepth: 1 -hidden: ---- -ovms_demos_vlm_npu -``` - This demo shows how to deploy Vision Language Models in the OpenVINO Model Server. Text generation use case is exposed via OpenAI API `chat/completions` and `responses` endpoints. diff --git a/demos/mediapipe_demos.md b/demos/mediapipe_demos.md new file mode 100644 index 0000000000..ca2b0b5b3b --- /dev/null +++ b/demos/mediapipe_demos.md @@ -0,0 +1,14 @@ +# MediaPipe demos {#ovms_demos_mediapipe} + +```{toctree} +--- +maxdepth: 1 +--- + +MediaPipe Object Detection Demo +Iris +Holistic +Realtime Stream Analysis +``` + + diff --git a/demos/text_generation.md b/demos/text_generation.md new file mode 100644 index 0000000000..c7587e5039 --- /dev/null +++ b/demos/text_generation.md @@ -0,0 +1,18 @@ +# Text generation demos {#ovms_text_generation} + +```{toctree} +--- +maxdepth: 1 +--- + +LLM demo +VLM demo +Agentic demo +RAG +Long Context +Structured output +Speculative decoding +LLM on NPU +Scaling on multi CPU and GPU +Loading models in GGUF +``` diff --git a/docs/home.md b/docs/home.md index 207273b7ae..0be71b83fe 100644 --- a/docs/home.md +++ b/docs/home.md @@ -17,53 +17,193 @@ ovms_docs_demos ovms_docs_troubleshooting ``` -Model Server hosts models and makes them accessible to software components over standard network protocols: a client sends a request to the model server, which performs model inference and sends a response back to the client. Model Server offers many advantages for efficient model deployment: -- Remote inference enables using lightweight clients with only the necessary functions to perform API calls to edge or cloud deployments. -- Applications are independent of the model framework, hardware device, and infrastructure. -- Client applications in any programming language that supports REST or gRPC calls can be used to run inference remotely on the model server. -- Clients require fewer updates since client libraries change very rarely. -- Model topology and weights are not exposed directly to client applications, making it easier to control access to the model. -- Ideal architecture for microservices-based applications and deployments in cloud environments – including Kubernetes and OpenShift clusters. -- Efficient resource utilization with horizontal and vertical inference scaling. +**High-performance model serving for Generative AI and classic deep learning — powered by [OpenVINO](https://github.com/openvinotoolkit/openvino) and optimized for Intel hardware.** + +--- + +## What is OVMS? + +OpenVINO Model Server (OVMS) is a production-grade, C++ inference server that exposes ML models over standard network APIs. It serves both **Generative AI**, **Agentic** workloads (LLMs, VLMs, image generation, audio) and **classic deep learning** models (object detection, classification, OCR, and more). + +- **OpenAI-compatible API** for text generation, embeddings, image generation, and audio +- **KServe** APIs for classic model inference +- **Runs anywhere** — Docker, bare metal, Kubernetes/OpenShift, Windows +- **Intel-optimized** — CPU, GPU, NPU acceleration via OpenVINO ![OVMS diagram](ovms_diagram.png) -## Serving with OpenVINO Model Server +--- + +## Quick Start -OpenVINO™ Model Server (OVMS) is a high-performance system for serving models. Implemented in C++ for scalability and optimized for deployment on Intel architectures. It uses the same API as [OpenAI](./genai.md), [Cohere](./model_server_rest_api_rerank.md) and [KServe](./model_server_grpc_api_kfs.md) while applying OpenVINO for inference execution. Inference service is provided via gRPC or REST API, making deploying new algorithms and AI experiments easy. +### Serve an LLM with OpenAI-compatible API -Check how to write the client applications using [generative endpoints](./clients_genai.md). +**On Linux (Docker):** +```bash +# Model is downloaded automatically from HuggingFace +docker run --rm -p 8000:8000 \ + openvino/model_server:latest \ + --source_model OpenVINO/Qwen3-4B-int4-ov \ + --model_repository_path /tmp/models \ + --rest_port 8000 +``` +> For GPU acceleration, use the `latest-gpu` image tag and pass `--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)` to expose the Intel GPU device. + +**On Windows (binary package):** +```bat +mkdir c:\models +ovms.exe --source_model OpenVINO/Qwen3-4B-int4-ov --model_repository_path c:\models --rest_port 8000 +``` -![OVMS picture](ovms_high_level.png) +**Query the model:** -The models used by the server need to be stored locally or hosted remotely by object storage services. For more details, refer to [Preparing Model Repository](./models_repository.md) documentation. Model server works inside [Docker containers](./deploying_server.md), on [Bare Metal](deploying_server.md), and in [Kubernetes environment](deploying_server.md). -Start using OpenVINO Model Server with a fast-forward serving example from the [QuickStart guide](ovms_quickstart.md) or [LLM QuickStart guide](./llm/quickstart.md). +```console +pip install openai +``` + +```python +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused") +stream = client.chat.completions.create( + model="OpenVINO/Qwen3-4B-int4-ov", + messages=[{"role": "user", "content": "What are the 3 main tourist attractions in Paris?"}], + stream=True, + extra_body={"chat_template_kwargs": {"enable_thinking": False}} +) +for chunk in stream: + if chunk.choices[0].delta.content: + print(chunk.choices[0].delta.content, end="", flush=True) +``` +> [LLM QuickStart](llm/quickstart.md) -### Key features: -- **[NEW]** [Speech Generation and Speech Recognition with OpenAI API](../demos/audio/README.md) -- **[NEW]** [Support for AI agents](../demos/continuous_batching/agentic_ai/README.md) -- **[NEW]** [Image generation and editing](../demos/image_generation/README.md) -- Native Windows support. Check updated [deployment guide](./deploying_server.md) -- [Embeddings endpoint compatible with OpenAI API](../demos/embeddings/README.md) -- [Reranking compatible with Cohere API](../demos/rerank/README.md) -- [Efficient Text Generation with OpenAI API](../demos/continuous_batching/README.md) -- [Python code execution](python_support/reference.md) + + +### Serve a Classic Model with KServe API + +**Download the model:** +```console +curl -L https://huggingface.co/OpenVINO/resnet50-int8-ov/resolve/main/resnet50.bin -O +curl -L https://huggingface.co/OpenVINO/resnet50-int8-ov/resolve/main/resnet50.xml -O +``` + +**On Linux (Docker):** +```bash +docker run --rm -d -u $(id -u) -v ${PWD}:/models -p 9000:9000 \ + openvino/model_server:latest \ + --model_name resnet --model_path /models/resnet50.xml \ + --mean "[123.675,116.28,103.53]" --scale "[58.395,57.12,57.375]" --layout "NHWC:NCHW" \ + --port 9000 +``` +> For GPU acceleration, use the `latest-gpu` image tag and pass `--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)` to expose the Intel GPU device. + +**Windows (binary package):** +```bat +ovms --model_name resnet --model_path resnet50.xml --mean "[123.675,116.28,103.53]" --scale "[58.395,57.12,57.375]" --layout "NHWC:NCHW" --port 9000 +``` + +Run inference with a sample python client + +```console +pip install numpy tritonclient[grpc] +curl -L -o image.jpeg https://github.com/openvinotoolkit/model_server/blob/main/demos/common/static/images/bee.jpeg?raw=true +``` + +```python +import numpy as np +import tritonclient.grpc as grpcclient +with open("image.jpeg", "rb") as f: + image_bytes = f.read() +client = grpcclient.InferenceServerClient(url="localhost:9000") +inputs = [grpcclient.InferInput("image", [1], "BYTES")] +inputs[0].set_data_from_numpy(np.array([image_bytes], dtype=object)) +outputs = [grpcclient.InferRequestedOutput("output")] +result = client.infer(model_name="resnet", inputs=inputs, outputs=outputs) +output = result.as_numpy("output") # (1, 1000) FP32 +print("Top-1 class index:", int(np.argmax(output[0]))) +``` + +> [Vision model QuickStart](ovms_quickstart.md) + +--- + +## Features + +### Generative AI +- [LLM text generation](../demos/continuous_batching/README.md) — continuous batching, streaming, structured output, speculative decoding +- [VLM (Vision Language Models)](../demos/continuous_batching/vlm/README.md) +- [AI Agents with MCP servers](../demos/continuous_batching/agentic_ai/README.md) +- [Text embeddings](../demos/embeddings/README.md) — OpenAI-compatible `/v1/embeddings` +- [Reranking](../demos/rerank/README.md) — Cohere-compatible API +- [Image generation](../demos/image_generation/README.md) — OpenAI-compatible `/v1/images/generations` +- [Speech recognition and TTS](../demos/audio/README.md) — OpenAI-compatible audio API +- [GGUF model support](../demos/gguf/README.md) + +### Classic Models & Pipelines +- TensorFlow, ONNX, PaddlePaddle, OpenVINO IR model formats +- [DAG pipelines](dag_scheduler.md) with [custom nodes](custom_node_development.md) +- [MediaPipe graphs](mediapipe.md) +- [Python execution nodes](python_support/reference.md) +- [Dynamic input shapes](shape_batch_size_and_layout.md) + +### Deployment & Integration +- Docker, bare metal (Linux & Windows), Kubernetes / OpenShift +- [Model repository](models_repository.md): local storage, S3, GCS, Azure Blob, HuggingFace Hub +- [Model versioning](model_version_policy.md) and [hot-reload](online_config_changes.md) +- [Prometheus-compatible metrics](metrics.md) - [gRPC streaming](streaming_endpoints.md) -- [MediaPipe graphs serving](mediapipe.md) -- Model management - including [model versioning](model_version_policy.md) and [model updates in runtime](online_config_changes.md) -- [Dynamic model inputs](shape_batch_size_and_layout.md) -- [Directed Acyclic Graph Scheduler](dag_scheduler.md) along with [custom nodes in DAG pipelines](custom_node_development.md) -- [Metrics](metrics.md) - metrics compatible with Prometheus standard -- Support for multiple frameworks, such as TensorFlow, PaddlePaddle and ONNX -- Support for [AI accelerators](./accelerators.md) +- [C API](model_server_c_api.md) for embedding OVMS in native applications + +### Hardware Acceleration +- CPU (x86, including Xeon), Intel integrated and discrete GPU, NPU +- See [supported accelerators](accelerators.md) + +[→ Full feature list](features.md) + +--- + +## Documentation + +| Topic | Link | +|---|---| +| Deployment | [Deploying the server](deploying_server.md) | +| Model repository | [Preparing models](models_repository.md) | +| Client libraries | [Writing client code](writing_app.md) | +| Demos & examples | [Demos](../demos/README.md) | +| Release notes | [GitHub Releases](https://github.com/openvinotoolkit/model_server/releases) | -## Additional Resources -* [ADVANCING GENAI WITH CPU OPTIMIZATION](https://cdrdv2-public.intel.com/864404/vFINAL_Intel%20SLM%20Whitepaper.pdf) +--- + +## Get the Server + +**Docker images** (recommended): +```text +docker pull openvino/model_server:latest # Intel CPU +docker pull openvino/model_server:latest-gpu # Intel CPU,GPU,NPU +``` -* [Manage deep learning models with OpenVINO Model Server](https://developers.redhat.com/articles/2024/07/03/manage-deep-learning-models-openvino-model-server#) +- [Docker Hub](https://hub.docker.com/r/openvino/model_server) +- [Red Hat Ecosystem Catalog](https://catalog.redhat.com/software/containers/intel/openvino-model-server/607833052937385fc98515de) + +**Binary packages** (Linux & Windows): [GitHub Releases](https://github.com/openvinotoolkit/model_server/releases) + +--- -* [RAG building blocks made easy and affordable with OpenVINO Model Server](https://medium.com/openvino-toolkit/rag-building-blocks-made-easy-and-affordable-with-openvino-model-server-e7b03da5012b) +## Contributing -* [Simple deployment with KServe API](https://blog.openvino.ai/blog-posts/kserve-api) +Contributions are welcome! Please open an issue or pull request on GitHub. +See [security policy](security.md) for responsible disclosure. + +--- + +## References + +- [OpenVINO Toolkit](https://software.intel.com/en-us/openvino-toolkit) +- [Performance benchmarks](https://docs.openvino.ai/2026/about-openvino/performance-benchmarks.html) +- [GenAI with CPU optimization — Intel whitepaper](https://cdrdv2-public.intel.com/864404/vFINAL_Intel%20SLM%20Whitepaper.pdf) +- [RAG with OpenVINO Model Server — blog post](https://medium.com/openvino-toolkit/rag-building-blocks-made-easy-and-affordable-with-openvino-model-server-e7b03da5012b) +- [AIPC turned into a mighty assistant](https://medium.com/openvino-toolkit/ai-pc-turned-into-a-mighty-ai-assistant-with-local-models-and-openvino-model-server-1f41913252c9) + +--- -* [Benchmarking results](https://docs.openvino.ai/2026/about-openvino/performance-benchmarks.html) +\* Other names and brands may be claimed as the property of others. diff --git a/docs/ovms_diagram.png b/docs/ovms_diagram.png index 2adfc99881..9cfa931647 100644 Binary files a/docs/ovms_diagram.png and b/docs/ovms_diagram.png differ diff --git a/tests/functional/object_model/ovms_instance.py b/tests/functional/object_model/ovms_instance.py index 3eee7096c1..f3c1d4b2b2 100644 --- a/tests/functional/object_model/ovms_instance.py +++ b/tests/functional/object_model/ovms_instance.py @@ -41,7 +41,7 @@ from tests.functional.utils.core import get_children_from_module from tests.functional.utils.inference.communication import GRPC, REST from tests.functional.utils.logger import get_logger -from tests.functional.constants.os_type import OsType +from tests.functional.constants.os_type import OsType, get_host_os from tests.functional.utils.port_manager import PortManager from tests.functional.utils.process import Process from tests.functional.utils.test_framework import change_dir_permissions, is_single_threaded @@ -66,7 +66,7 @@ from tests.functional.object_model.mediapipe_calculators import MediaPipeCalculator from tests.functional.object_model.ovms_config import OvmsConfig from tests.functional.object_model.package_manager import PackageManager -from tests.functional.object_model.resource_monitor import DockerResourceMonitor +from tests.functional.object_model.resource_monitor import DockerResourceMonitor, WindowsResourceMonitor from tests.functional.object_model.test_environment import TestEnvironment logger = get_logger(__name__) @@ -539,7 +539,13 @@ def attach_context(self, context): def attach_resource_monitor(self, context, start=True): if hasattr(self.ovms, "container"): self.resource_monitor = DockerResourceMonitor(self.ovms.container) - if start: - self.resource_monitor.start() - context.test_objects.append(self.resource_monitor) - return self.resource_monitor + elif get_host_os() == OsType.Windows or getattr(context, "base_os", None) == OsType.Windows: + ovms_pid = self.ovms._dmesg_log.ovms_pid + assert ovms_pid is not None, "Cannot attach Windows resource monitor: ovms_pid is not available" + self.resource_monitor = WindowsResourceMonitor(ovms_pid) + else: + return None + if start: + self.resource_monitor.start() + context.test_objects.append(self.resource_monitor) + return self.resource_monitor diff --git a/tests/functional/object_model/ovms_log_monitor.py b/tests/functional/object_model/ovms_log_monitor.py index bbd6e3791d..052152c808 100644 --- a/tests/functional/object_model/ovms_log_monitor.py +++ b/tests/functional/object_model/ovms_log_monitor.py @@ -15,6 +15,7 @@ # import datetime +import os import re import time @@ -362,6 +363,7 @@ class BinaryOvmsLogMonitor(OvmsLogMonitor): def __init__(self, ovms_process, **kwargs): super().__init__(**kwargs) self._proc = ovms_process + self._mirror_offset = 0 def is_ovms_running(self): status = get_pid_status(self._proc._proc.pid) @@ -371,13 +373,30 @@ def is_ovms_running(self): return True return True + def _read_mirror_tail(self, mirror_path): + with open(mirror_path, "rb") as fd: + fd.seek(self._mirror_offset) + chunk = fd.read() + # A trailing partial line would be torn in half between two reads. + complete, newline, _partial = chunk.rpartition(b"\n") + if not newline: + return [] + self._mirror_offset += len(complete) + len(newline) + return complete.decode("utf-8", "ignore").splitlines() + def get_all_logs(self): stdout, stderr = self._proc.get_output() if stderr: logger.error( f"Detect non-empty stderr! It is recommended to redirect stderr to stdout: 2>&1. STDERR: {stderr}" ) - self._read_lines += stdout.splitlines() + mirror_path = self._proc.get_stdout_mirror_path() + if mirror_path and os.path.exists(mirror_path): + # pop() drains the queue shared by every monitor of this process, so the lines one + # monitor takes would be missing from the others and from the saved log. + self._read_lines += self._read_mirror_tail(mirror_path) + else: + self._read_lines += stdout.splitlines() return self._read_lines diff --git a/tests/functional/object_model/resource_monitor.py b/tests/functional/object_model/resource_monitor.py index 5787b69b1f..ce0833785c 100644 --- a/tests/functional/object_model/resource_monitor.py +++ b/tests/functional/object_model/resource_monitor.py @@ -17,9 +17,11 @@ import csv import threading from abc import ABC, abstractmethod +from datetime import datetime from pathlib import Path import numpy as np +import psutil from dateutil import parser from tests.functional.utils.logger import get_logger @@ -32,6 +34,7 @@ class ResourceMonitor(threading.Thread, ABC): def __init__(self): threading.Thread.__init__(self) self._stop_event = threading.Event() + self.stop_reason = None def stop(self): self._stop_event.set() @@ -42,6 +45,7 @@ def run(self): try: self.check_resources() except StopIteration as e: + self.stop_reason = str(e) self._stop_event.set() break self.save_data() @@ -55,13 +59,33 @@ def save_data(self): pass +def _cgroup_cache_bytes(cgroup_memory_stats): + if "cache" in cgroup_memory_stats: + return float(cgroup_memory_stats.get("cache", 0)) + return float(cgroup_memory_stats.get("file", 0)) + + class DockerResourceMonitor(ResourceMonitor): MEMORY_USAGE = "MEMORY_USAGE" - FIELDS = ["DATE", "PIDS_COUNT", MEMORY_USAGE] # + ["CPU_USAGE"] # Enable in further releases + PRIVATE_MEMORY = "PRIVATE_MEMORY" + MEMORY_CACHE = "MEMORY_CACHE" + FIELDS = ["DATE", "PIDS_COUNT", MEMORY_USAGE, PRIVATE_MEMORY, MEMORY_CACHE] # + ["CPU_USAGE"] # Enable in further releases + VALIDATED_FIELDS = [MEMORY_USAGE, PRIVATE_MEMORY] + LOGGED_MEMORY_FIELDS = [MEMORY_CACHE] + COUNTER_FIELDS = ["PIDS_COUNT"] + LOGGED_FIELDS = LOGGED_MEMORY_FIELDS + COUNTER_FIELDS FIELDS_TO_STATS = { "DATE": lambda x: x["read"], "PIDS_COUNT": lambda x: int(x["pids_stats"].get("current", "0")), MEMORY_USAGE: lambda x: "{:.2f}M".format(float(x["memory_stats"].get("usage", "0.0")) / (2**20)), + PRIVATE_MEMORY: lambda x: "{:.2f}M".format( + float(x["memory_stats"].get("stats", {}).get( + "anon", x["memory_stats"].get("stats", {}).get("rss", 0) + )) / (2**20) + ), + MEMORY_CACHE: lambda x: "{:.2f}M".format( + _cgroup_cache_bytes(x["memory_stats"].get("stats", {})) / (2**20) + ), # Enable after debug & fixing # "CPU_USAGE": lambda x: # [cpu / x['cpu_stats']['cpu_usage']['total_usage'] for cpu in x['cpu_stats']['cpu_usage']['percpu_usage']], @@ -140,9 +164,142 @@ def _get_resource_data(self): 'networks' = {dict: 1} {'eth0': {'rx_bytes': 90, 'rx_packets': 1, 'rx_errors': 0, ... """ stats = self.container.stats(stream=False, decode=False) + # Prevent reading zeroes from stopped container + pids = int(stats.get("pids_stats", {}).get("current", 0) or 0) + usage = float(stats.get("memory_stats", {}).get("usage", 0) or 0) + if pids <= 0 or usage <= 0: + reason = ( + f"container {self.container.name} reports no live process " + f"(pids_stats.current={pids}, memory_stats.usage={usage} bytes) - the container " + f"stopped, crashed or was removed while the test was still sampling" + ) + logger.warning(f"Stopping Docker resource monitor: {reason}") + raise StopIteration(reason) return stats def get_stats_by_field(self, field): result = self._get_resource_data() self._docker_stats_data_raw.append(result) return self.get_field_data(field, result) + + def sample_all(self): + """Read one stats snapshot and return all tracked metrics as floats (MB / counts). + + A single snapshot keeps every metric in the returned sample mutually + consistent (same instant) and avoids one docker stats call per metric. + """ + stats = self._get_resource_data() + self._docker_stats_data_raw.append(stats) + return { + field: float(str(self.get_field_data(field, stats)).replace("M", "")) + for field in self.get_validated_metric_names() + self.get_logged_metric_names() + } + + @classmethod + def get_validated_metric_names(cls): + return cls.VALIDATED_FIELDS + + @classmethod + def get_logged_metric_names(cls): + return cls.LOGGED_FIELDS + + @classmethod + def get_memory_metric_names(cls): + return cls.VALIDATED_FIELDS + cls.LOGGED_MEMORY_FIELDS + + @classmethod + def get_counter_metric_names(cls): + return cls.COUNTER_FIELDS + + +class WindowsResourceMonitor(ResourceMonitor): + WORKING_SET_SIZE = "WORKING_SET_SIZE" + PRIVATE_BYTES = "PRIVATE_BYTES" + PAGE_FAULTS = "PAGE_FAULTS" + + MEMORY_USAGE = WORKING_SET_SIZE + + FIELDS = ["DATE", WORKING_SET_SIZE, PRIVATE_BYTES, PAGE_FAULTS] + VALIDATED_FIELDS = [PRIVATE_BYTES] + LOGGED_MEMORY_FIELDS = [WORKING_SET_SIZE] + COUNTER_FIELDS = [PAGE_FAULTS] + LOGGED_FIELDS = LOGGED_MEMORY_FIELDS + COUNTER_FIELDS + SAMPLE_INTERVAL_SEC = 1.0 + # Optional callback invoked after save_data with (log_path). + on_data_saved = None + + def __init__(self, ovms_pid): + super().__init__() + self.ovms_pid = ovms_pid + self._process = psutil.Process(int(ovms_pid)) + self._stats_data_raw = [] + + def cleanup(self): + if not self._stop_event.is_set(): + if self.is_alive(): + self.stop() + self.save_data() + + def _get_resource_data(self): + stats = {"DATE": datetime.now().isoformat()} + try: + info = self._process.memory_info() + except psutil.Error as error: + reason = (f"OVMS process pid={self.ovms_pid} is no longer readable ({error}) - it " + f"exited, crashed or was killed while the test was still sampling") + logger.warning(f"Stopping Windows resource monitor: {reason}") + raise StopIteration(reason) from error + # wset/private are the Win32 counters .NET exposes as WorkingSet64/PrivateMemorySize64. + stats[self.WORKING_SET_SIZE] = float(info.wset) / (1024 * 1024) + stats[self.PRIVATE_BYTES] = float(info.private) / (1024 * 1024) + stats[self.PAGE_FAULTS] = int(info.num_page_faults) + return stats + + def check_resources(self): + result = self._get_resource_data() + self._stats_data_raw.append(result) + self._stop_event.wait(self.SAMPLE_INTERVAL_SEC) + + def save_data(self): + self.rows = list(self._stats_data_raw) + log_path = Path(artifacts_dir, f"windows_stats_pid_{self.ovms_pid}.log") + with log_path.open("w") as csvfile: + writer = csv.DictWriter(csvfile, fieldnames=self.FIELDS) + writer.writeheader() + writer.writerows(self.rows) + if WindowsResourceMonitor.on_data_saved: + WindowsResourceMonitor.on_data_saved(log_path) + return log_path + + def get_stats_by_field(self, field): + result = self._get_resource_data() + self._stats_data_raw.append(result) + value = result[field] + if field in (self.WORKING_SET_SIZE, self.PRIVATE_BYTES): + return f"{value:.2f}M" + return str(value) + + def sample_all(self): + """Read one process snapshot and return all tracked metrics as floats (MB / counts).""" + stats = self._get_resource_data() + self._stats_data_raw.append(stats) + return { + field: float(stats[field]) + for field in self.get_validated_metric_names() + self.get_logged_metric_names() + } + + @classmethod + def get_validated_metric_names(cls): + return cls.VALIDATED_FIELDS + + @classmethod + def get_logged_metric_names(cls): + return cls.LOGGED_FIELDS + + @classmethod + def get_memory_metric_names(cls): + return cls.VALIDATED_FIELDS + cls.LOGGED_MEMORY_FIELDS + + @classmethod + def get_counter_metric_names(cls): + return cls.COUNTER_FIELDS diff --git a/tests/functional/object_model/test_helpers.py b/tests/functional/object_model/test_helpers.py index 00cd7938de..dc62fac700 100644 --- a/tests/functional/object_model/test_helpers.py +++ b/tests/functional/object_model/test_helpers.py @@ -81,7 +81,8 @@ class Endpoints(enum.Enum): RELOAD_CONFIG = "/v1/config/reload" -def send_request_to_endpoint(port, address=None, endpoint=None, expected_code=None, retry=1, timeout=60): +def send_request_to_endpoint(port, address=None, endpoint=None, expected_code=None, retry=1, timeout=60, + retry_delay=1, retry_backoff=1, retry_max_delay=None): address = TestEnvironment.get_server_address() if address is None else address url_with_endpoint = f"http://{address}:{port}{endpoint}" logger.info(f"Try to send request to endpoint: {url_with_endpoint}") @@ -93,7 +94,8 @@ def send_request_to_endpoint(port, address=None, endpoint=None, expected_code=No else: msg = f"Not supported endpoint: {endpoint}" raise ValueError(msg) - retry_setup = {"tries": int(retry), "delay": 1} + retry_setup = {"tries": int(retry), "delay": retry_delay, "backoff": retry_backoff, + "max_delay": retry_max_delay} kwargs = {"url": url_with_endpoint, "params": {}, "timeout": timeout} ret = retry_call(func, fkwargs=kwargs, **retry_setup) if expected_code is None: @@ -114,10 +116,13 @@ def send_request_to_endpoint(port, address=None, endpoint=None, expected_code=No return ret -def send_reload_request(port, address=None, expected_code=None, retry=1, timeout=60): +def send_reload_request(port, address=None, expected_code=None, retry=1, timeout=60, + retry_delay=1, retry_backoff=1, retry_max_delay=None): address = TestEnvironment.get_server_address() if address is None else address endpoint = Endpoints.RELOAD_CONFIG.value - return send_request_to_endpoint(port, address, endpoint, expected_code, retry, timeout=timeout) + return send_request_to_endpoint(port, address, endpoint, expected_code, retry, timeout=timeout, + retry_delay=retry_delay, retry_backoff=retry_backoff, + retry_max_delay=retry_max_delay) def get_config_request(port, address=None, expected_code=None, retry=1): diff --git a/tests/functional/utils/process.py b/tests/functional/utils/process.py index d10ee154e7..8e7a3c73c3 100644 --- a/tests/functional/utils/process.py +++ b/tests/functional/utils/process.py @@ -48,6 +48,9 @@ def __init__(self): def get_output(self): return self._std_stream.pop(), self._err_stream.pop() + def get_stdout_mirror_path(self): + return getattr(self._std_stream, "mirror_file_path", None) + def set_log_verbose(self): self.policy['log-run']['verbose'] = True self.policy['log-async-run']['verbose'] = True @@ -223,6 +226,11 @@ def wait(self, timeout): def is_alive(self): return self._proc.poll() is None + def poll_exitcode(self): + # get_exitcode() may kill the process, so post-mortem callers need a read-only variant. + self._proc.poll() + return self._proc.returncode + def timeout_detected(self): return self._std_stream.timeout_detected or self._err_stream.timeout_detected @@ -400,7 +408,7 @@ def __init__(self, stream, local_thread=True, mirror_file_path=None): self._stream = stream self.timeout_detected = False self._local_thread = local_thread - self._mirror_file_path = mirror_file_path + self.mirror_file_path = mirror_file_path def run(self): self._thread = Thread(target=self._enqueue_output, args=(self._stream, )) @@ -410,11 +418,13 @@ def run(self): def _enqueue_output(self, out): mirror_fd = None try: - if self._mirror_file_path: - mirror_dir = os.path.dirname(self._mirror_file_path) + if self.mirror_file_path: + mirror_dir = os.path.dirname(self.mirror_file_path) if mirror_dir: os.makedirs(mirror_dir, exist_ok=True) - mirror_fd = open(self._mirror_file_path, "a", encoding="utf-8", errors="ignore") + # Text mode would turn every CRLF the process wrote into CRCRLF, blank-lining the log. + mirror_fd = open(self.mirror_file_path, "a", encoding="utf-8", errors="ignore", + newline="") for line in iter(out.readline, b'' if self._local_thread else ''): line = line.decode('utf8', 'ignore') if self._local_thread else line self._queue.put(line)