Skip to content

deploying_server_docker.md

Latest commit

 

History

History
87 lines (60 loc) · 3.03 KB

deploying_server_docker.md

File metadata and controls

87 lines (60 loc) · 3.03 KB
Raw

Deploying Model Server in Docker Container {#ovms_docs_deploying_server_docker}

This is a step-by-step guide on how to deploy OpenVINO™ Model Server on Linux, using Docker.

Before you start, make sure you have:

  • Docker Engine installed
  • Intel® Core™ processor or Intel® Xeon® processor
  • Linux, macOS or Windows via WSL
  • (optional) AI accelerators supported by OpenVINO. Accelerators are tested only on bare-metal Linux hosts.

Launch Model Server Container

This example shows how to launch the model server with a ResNet50 image classification model from a cloud storage:

Step 1. Pull Model Server Image

Pull an image from Docker:

docker pull openvino/model_server:latest

or RedHat Ecosystem Catalog:

docker pull registry.connect.redhat.com/intel/openvino-model-server:latest

NOTE: You can also pull public image openvino/model_server:weekly with development version of the model server, which is built from the main branch. It allow you to evaluate the latest features ahead of official releases.

Step 2. Prepare Data for Serving

2.1 Start the container with the model
wget https://storage.openvinotoolkit.org/repositories/open_model_zoo/2022.1/models_bin/2/resnet50-binary-0001/FP32-INT1/resnet50-binary-0001.{xml,bin} -P models/resnet50/1
docker run -u $(id -u) -v $(pwd)/models:/models -p 9000:9000 openvino/model_server:latest \
--model_name resnet --model_path /models/resnet50 \
--layout NHWC:NCHW --port 9000
2.2 Download input files: an image and a label mapping file
wget https://raw.githubusercontent.com/openvinotoolkit/model_server/main/demos/common/static/images/zebra.jpeg
wget https://raw.githubusercontent.com/openvinotoolkit/model_server/main/demos/common/python/classes.py
2.3 Install the Python-based client package
pip3 install tritonclient[grpc] numpy

Step 3. Run Prediction

echo 'import numpy as np
from classes import imagenet_classes
import tritonclient.grpc as grpcclient

client = grpcclient.InferenceServerClient(url="localhost:9000")

# Get model metadata to discover input/output names
metadata = client.get_model_metadata("resnet")
input_name = metadata.inputs[0].name
output_name = metadata.outputs[0].name

# Load image as binary
with open("zebra.jpeg", "rb") as f:
    img_bytes = f.read()

# Run inference
infer_input = grpcclient.InferInput(input_name, [1], "BYTES")
infer_input.set_data_from_numpy(np.array([img_bytes], dtype=np.object_))
result = client.infer("resnet", [infer_input])
output = result.as_numpy(output_name)
result_index = np.argmax(output[0])
print(imagenet_classes[result_index])' >> predict.py

python predict.py
zebra

If everything is set up correctly, you will see 'zebra' prediction in the output.