Skip to content

Remove parsers cli flag from demos #4354

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dkalinowski wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Remove parsers cli flag from demos #4354

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
118bba1
save
dkalinowski Jul 2, 2026
1968032
Merge remote-tracking branch 'origin/main' into remove-parsers-from-d…
dkalinowski Jul 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions demos/code_local_assistant/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ With the rise of AI PC capabilities, hosting own Visual Studio code assistant is
:sync: OpenVINO/Qwen3-Coder-30B-A3B-Instruct-int4-ov
```bat
mkdir c:\models
ovms --model_repository_path c:\models --source_model OpenVINO/Qwen3-Coder-30B-A3B-Instruct-int4-ov --task text_generation --target_device GPU --tool_parser qwen3coder --rest_port 8000 --cache_dir .ovcache --model_name Qwen3-Coder-30B-A3B-Instruct
ovms --model_repository_path c:\models --source_model OpenVINO/Qwen3-Coder-30B-A3B-Instruct-int4-ov --task text_generation --target_device GPU --rest_port 8000 --cache_dir .ovcache --model_name Qwen3-Coder-30B-A3B-Instruct
```
> **Note:** For deployment, the model requires ~16GB disk space and recommended 19GB+ of VRAM on the GPU.

Expand All @@ -27,7 +27,7 @@ ovms --model_repository_path c:\models --source_model OpenVINO/Qwen3-Coder-30B-A
:sync: OpenVINO/gpt-oss-20b-int4-ov
```bat
mkdir c:\models
ovms --model_repository_path c:\models --source_model OpenVINO/gpt-oss-20b-int4-ov --task text_generation --target_device GPU --tool_parser gptoss --reasoning_parser gptoss --rest_port 8000 --cache_dir .ovcache --model_name gpt-oss-20b
ovms --model_repository_path c:\models --source_model OpenVINO/gpt-oss-20b-int4-ov --task text_generation --target_device GPU --rest_port 8000 --cache_dir .ovcache --model_name gpt-oss-20b
```
> **Note:** For deployment, the model requires ~12GB disk space and recommended 16GB+ of VRAM on the GPU.
:::
Expand All @@ -36,15 +36,15 @@ ovms --model_repository_path c:\models --source_model OpenVINO/gpt-oss-20b-int4-
:sync: OpenVINO/Qwen3-8B-int4-ov
```bat
mkdir c:\models
ovms --model_repository_path c:\models --source_model OpenVINO/Qwen3-8B-int4-ov --task text_generation --target_device GPU --tool_parser hermes3 --reasoning_parser qwen3 --rest_port 8000 --cache_dir .ovcache --model_name Qwen3-8B
ovms --model_repository_path c:\models --source_model OpenVINO/Qwen3-8B-int4-ov --task text_generation --target_device GPU --rest_port 8000 --cache_dir .ovcache --model_name Qwen3-8B
```
> **Note:** For deployment, the model requires ~4GB disk space and recommended 6GB+ of VRAM on the GPU.
:::
:::{tab-item} OpenVINO/Qwen3-8B-int4-cw-ov
:sync: OpenVINO/Qwen3-8B-int4-cw-ov
```bat
mkdir c:\models
ovms --model_repository_path c:\models --source_model OpenVINO/Qwen3-8B-int4-cw-ov --task text_generation --target_device NPU --tool_parser hermes3 --rest_port 8000 --max_prompt_len 16384 --plugin_config "{\"NPUW_LLM_PREFILL_ATTENTION_HINT\":\"PYRAMID\"}" --cache_dir .ovcache --model_name Qwen3-8B
ovms --model_repository_path c:\models --source_model OpenVINO/Qwen3-8B-int4-cw-ov --task text_generation --target_device NPU --rest_port 8000 --max_prompt_len 16384 --plugin_config "{\"NPUW_LLM_PREFILL_ATTENTION_HINT\":\"PYRAMID\"}" --cache_dir .ovcache --model_name Qwen3-8B
```
> **Note:** First model initialization might be long. With the compilation cache, sequential model loading will be fast.
:::
Expand All @@ -67,7 +67,7 @@ ovms --model_repository_path c:\models --source_model OpenVINO/Qwen3-VL-8B-Instr
mkdir -p models
docker run -d -p 8000:8000 --rm --user $(id -u):$(id -g) -v $(pwd)/models:/models/:rw --device /dev/dri --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) \
openvino/model_server:weekly \
--model_repository_path /models --source_model OpenVINO/Qwen3-Coder-30B-A3B-Instruct-int4-ov --task text_generation --target_device GPU --tool_parser qwen3coder --rest_port 8000 --model_name Qwen3-Coder-30B-A3B-Instruct
--model_repository_path /models --source_model OpenVINO/Qwen3-Coder-30B-A3B-Instruct-int4-ov --task text_generation --target_device GPU --rest_port 8000 --model_name Qwen3-Coder-30B-A3B-Instruct
```
> **Note:** For deployment, the model requires ~16GB disk space and recommended 19GB+ of VRAM on the GPU.

Expand All @@ -80,7 +80,7 @@ docker run -d -p 8000:8000 --rm --user $(id -u):$(id -g) -v $(pwd)/models:/model
mkdir -p models
docker run -d -p 8000:8000 --rm --user $(id -u):$(id -g) -v $(pwd)/models:/models/:rw --device /dev/dri --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) \
openvino/model_server:weekly \
--model_repository_path /models --source_model OpenVINO/gpt-oss-20b-int4-ov --task text_generation --target_device GPU --tool_parser gptoss --reasoning_parser gptoss --rest_port 8000 --model_name gpt-oss-20b
--model_repository_path /models --source_model OpenVINO/gpt-oss-20b-int4-ov --task text_generation --target_device GPU --rest_port 8000 --model_name gpt-oss-20b
```
> **Note:** For deployment, the model requires ~12GB disk space and recommended 16GB+ of VRAM on the GPU.
:::
Expand All @@ -91,7 +91,7 @@ docker run -d -p 8000:8000 --rm --user $(id -u):$(id -g) -v $(pwd)/models:/model
mkdir c:\models
docker run -d -p 8000:8000 --rm --user $(id -u):$(id -g) -v $(pwd)/models:/models/:rw --device /dev/dri --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) \
openvino/model_server:weekly \
--model_repository_path /models --source_model OpenVINO/Qwen3-8B-int4-ov --task text_generation --target_device GPU --tool_parser hermes3 --reasoning_parser qwen3 --rest_port 8000 --model_name Qwen3-8B
--model_repository_path /models --source_model OpenVINO/Qwen3-8B-int4-ov --task text_generation --target_device GPU --rest_port 8000 --model_name Qwen3-8B
```
> **Note:** For deployment, the model requires ~4GB disk space and recommended 6GB+ of VRAM on the GPU.
:::
Expand All @@ -101,7 +101,7 @@ docker run -d -p 8000:8000 --rm --user $(id -u):$(id -g) -v $(pwd)/models:/model
mkdir -p models
docker run -d -p 8000:8000 --rm --user $(id -u):$(id -g) -v $(pwd)/models:/models/:rw --device /dev/accel --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) \
openvino/model_server:weekly \
--model_repository_path /models --source_model OpenVINO/Qwen3-8B-int4-cw-ov --task text_generation --target_device NPU --tool_parser hermes3 --rest_port 8000 --max_prompt_len 16384 --plugin_config '{"NPUW_LLM_PREFILL_ATTENTION_HINT":"PYRAMID"}' --model_name Qwen3-8B
--model_repository_path /models --source_model OpenVINO/Qwen3-8B-int4-cw-ov --task text_generation --target_device NPU --rest_port 8000 --max_prompt_len 16384 --plugin_config '{"NPUW_LLM_PREFILL_ATTENTION_HINT":"PYRAMID"}' --model_name Qwen3-8B
```
> **Note:** First model initialization might be long. With the compilation cache, sequential model loading will be fast.
:::
Expand All @@ -124,10 +124,10 @@ Models which are not published in OpenVINO format can be exported and quantized

```
mkdir models
python export_model.py text_generation --source_model unsloth/Devstral-Small-2507 --weight-format int4 --config_file_path models/config_all.json --model_repository_path models --tool_parser devstral --target_device GPU
python export_model.py text_generation --source_model unsloth/Devstral-Small-2507 --weight-format int4 --config_file_path models/config_all.json --model_repository_path models --target_device GPU
curl -L -o models/unsloth/Devstral-Small-2507/chat_template.jinja https://raw.githubusercontent.com/openvinotoolkit/model_server/refs/heads/main/extras/chat_template_examples/chat_template_devstral.jinja

ovms --model_repository_path models --source_model unsloth/Devstral-Small-2507 --task text_generation --target_device GPU --tool_parser devstral --rest_port 8000 --cache_dir .ovcache
ovms --model_repository_path models --source_model unsloth/Devstral-Small-2507 --task text_generation --target_device GPU --rest_port 8000 --cache_dir .ovcache
```
> **Note:** Exporting models is a one time operation but might consume RAM at least of the model size and might take a lot of time depending on the model size.

Expand Down
4 changes: 2 additions & 2 deletions demos/continuous_batching/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ That makes it easy to use and efficient especially on on Intel® Xeon® processo
Running this command starts the container with CPU only target device:
```bash
mkdir -p ${HOME}/models
docker run -it -p 8000:8000 --rm --user $(id -u):$(id -g) -v ${HOME}/models:/models/:rw openvino/model_server:weekly --model_repository_path /models --source_model OpenVINO/Qwen3-30B-A3B-Instruct-2507-int4-ov --task text_generation --target_device CPU --tool_parser hermes3 --rest_port 8000 --model_name Qwen3-30B-A3B-Instruct-2507-int4-ov
docker run -it -p 8000:8000 --rm --user $(id -u):$(id -g) -v ${HOME}/models:/models/:rw openvino/model_server:weekly --model_repository_path /models --source_model OpenVINO/Qwen3-30B-A3B-Instruct-2507-int4-ov --task text_generation --target_device CPU --rest_port 8000 --model_name Qwen3-30B-A3B-Instruct-2507-int4-ov
```
> **Note:** In case you want to use GPU target device, add extra docker parameters `--device /dev/dri --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1)`
to `docker run` command. The parameter `--target_device` should be also updated to `GPU`.
Expand All @@ -46,7 +46,7 @@ to `docker run` command. The parameter `--target_device` should be also updated
After ovms is installed according to steps from [baremetal deployment guide](../../docs/deploying_server_baremetal.md), run the following command:

```bat
ovms.exe --model_repository_path c:\models --source_model OpenVINO/Qwen3-30B-A3B-Instruct-2507-int4-ov --task text_generation --target_device GPU --tool_parser hermes3 --rest_port 8000 --model_name Qwen3-30B-A3B-Instruct-2507-int4-ov
ovms.exe --model_repository_path c:\models --source_model OpenVINO/Qwen3-30B-A3B-Instruct-2507-int4-ov --task text_generation --target_device GPU --rest_port 8000 --model_name Qwen3-30B-A3B-Instruct-2507-int4-ov
```


Expand Down
Loading