diff --git a/demos/integration_with_OpenHands/.gitignore b/demos/integration_with_OpenHands/.gitignore new file mode 100644 index 0000000000..d48130e689 --- /dev/null +++ b/demos/integration_with_OpenHands/.gitignore @@ -0,0 +1,17 @@ +# Environment variables (local overrides) +.env + +# Log files +*.log + +#OpenHANDS session files +*.openhands/ + +openvino-env/ + +# Generated Docker Compose file (docker-compose.template.yml is the source) +docker-compose.yml + +# Deployment metadata (generated by deploy_model_ovms.sh) +.ovms-deployment + diff --git a/demos/integration_with_OpenHands/ADVANCED_DEPLOYMENT.md b/demos/integration_with_OpenHands/ADVANCED_DEPLOYMENT.md new file mode 100644 index 0000000000..822c5903df --- /dev/null +++ b/demos/integration_with_OpenHands/ADVANCED_DEPLOYMENT.md @@ -0,0 +1,842 @@ +# Advanced Deployment Guide + +This document contains detailed deployment and implementation reference material for the OpenHands + OVMS integration. For the Quick Start guide, see [README.md](README.md). + +--- + +## Request Flow + +1. User creates an agent task in the OpenHands web UI +2. OpenHands sends `POST /v3/chat/completions` requests to OVMS +3. OVMS routes requests through the MediaPipe LLM graph +4. OpenVINO inference engine processes the model +5. OVMS returns the completion (possibly with tool calls) +6. OpenHands parses the response and continues the agent loop + +--- + +## Why OpenHands Requires Additional Configuration + +Unlike simple chat UIs, OpenHands has specific requirements: + +- **Model prefix:** OpenHands expects `openai/` format in `LLM_MODEL` +- **API key placeholder:** A non-empty `LLM_API_KEY` is required even though OVMS doesn't authenticate +- **Stable networking:** Container-to-container communication on a shared Docker network +- **Docker socket access:** OpenHands creates runtime sandbox containers for code execution +- **Resource limits:** Sandbox memory limits prevent runaway agent processes + +--- + +## OVMS `--source_model` Workflow + +OVMS provides native model retrieval and preparation through the `--source_model` parameter: + +```bash +export GPU_ARGS=$(if ls /dev/dri/render* >/dev/null 2>&1; then echo "--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)"; fi) + +docker run --rm ${GPU_ARGS} \ + -v ${HOME}/ovms-openhands/models:/models \ + openvino/model_server:latest-gpu \ + --source_model OpenVINO/Qwen3-8b-int8-ov \ + --model_repository_path /models \ + --model_name qwen3-8b-int8-ov \ + --task text_generation +``` + +This command downloads the model from Hugging Face and stores the model artifacts in the specified model repository. When using the latest-py image, it converts the model to OpenVINO IR format if needed. + +--- + +## Model Workspace Layout + +After running the `--source_model` workflow, the model directory contains: + +```text +${HOME}/ovms-openhands/models/ +└── qwen3-8b-int8-ov/ + ├── openvino_model.xml # OpenVINO model structure + ├── openvino_model.bin # Model weights + ├── graph.pbtxt # MediaPipe LLM graph configuration + └── ....... +``` + +This external storage keeps the Git repository lightweight and allows model reuse across OVMS deployments. + +--- + +## Manual Deployment Workflow + +You can deploy using Docker commands directly without the helper scripts. This approach is useful for debugging and customization. + +**Repository not required:** These commands can be executed from any directory on a Linux system with Docker installed. The OpenHands state directory (`.openhands`) will be created relative to your current working directory. + +### Step 1: Set environment variables + +```bash +# Model configuration +export MODEL_ID="OpenVINO/Qwen3-8b-int8-ov" +export LOCAL_NAME="qwen3-8b-int8-ov" +export TARGET_DEVICE="CPU" +export REASONING_PARSER="" +export MODEL_CACHE_DIR="${HOME}/ovms-openhands/models" +export HF_TOKEN="${HF_TOKEN:-}" + +# Published ports (optional - defaults shown) +export OVMS_REST_PORT="${OVMS_REST_PORT:-8000}" +export OVMS_GRPC_PORT="${OVMS_GRPC_PORT:-9000}" +export OPENHANDS_PORT="${OPENHANDS_PORT:-3000}" + +# Proxy configuration (optional - forward to containers if set) +export http_proxy="${http_proxy:-}" +export https_proxy="${https_proxy:-}" +export HTTP_PROXY="${HTTP_PROXY:-}" +export HTTPS_PROXY="${HTTPS_PROXY:-}" +export no_proxy="${no_proxy:-}" +export NO_PROXY="${NO_PROXY:-}" +``` + +### Step 2: Create the model cache directory + +```bash +mkdir -p "$MODEL_CACHE_DIR" +``` + +> **Note:** OVMS runs as a non-root user inside the container. The mounted model cache directory must be writable by the OVMS container user. +> +> On some WSL2/Docker environments, the mounted directory permissions may prevent OVMS from creating model directories. If OVMS fails during startup with permission errors, you may see an error like: +> +> ``` +> Libgit2 clone error: failed to make directory '/models/...': Permission denied +> ``` +> +> Verify the directory permissions: +> +> ```bash +> ls -ld "$MODEL_CACHE_DIR" +> ``` +> +> Fix by making the directory writable by all users: +> +> ```bash +> chmod a+rwx "$MODEL_CACHE_DIR" +> ``` +> +> Then redeploy the OVMS container. + +### Step 3: Deploy OVMS + +```bash +# Create the Docker network +docker network create ovms-net 2>/dev/null || true + +# Run OVMS container +docker run -d \ + --name ovms-llm \ + --network ovms-net \ + --publish ${OVMS_REST_PORT}:8000 \ + --publish ${OVMS_GRPC_PORT}:9000 \ + --device /dev/dri:/dev/dri \ + --volume "$MODEL_CACHE_DIR:/models:rw" \ + --env HF_TOKEN="${HF_TOKEN:-}" \ + --restart unless-stopped \ + openvino/model_server:latest \ + --model_repository_path /models \ + --source_model "$MODEL_ID" \ + --model_name "$LOCAL_NAME" \ + --task text_generation \ + --target_device "$TARGET_DEVICE" \ + --port 9000 \ + --rest_port 8000 +``` + +This command downloads the OpenVINO model from Hugging Face (if not cached), generates the MediaPipe LLM graph, and starts the OVMS server with the OpenAI-compatible REST API. + +### Step 4: Deploy OpenHands + +```bash +# Run OpenHands container +docker run -d \ + --name openhands \ + --network ovms-net \ + --publish ${OPENHANDS_PORT}:3000 \ + --add-host host.docker.internal:host-gateway \ + --volume /var/run/docker.sock:/var/run/docker.sock \ + --volume "$(pwd)/.openhands:/.openhands" \ + --env LLM_BASE_URL="http://ovms-llm:8000/v3" \ + --env LLM_MODEL="openai/${LOCAL_NAME}" \ + --env LLM_API_KEY="unused" \ + --env LLM_TEMPERATURE="0.0" \ + --env LLM_MAX_OUTPUT_TOKENS="500" \ + --env LLM_MAX_INPUT_TOKENS="4096" \ + --env LLM_TIMEOUT="120000" \ + --env SANDBOX_DOCKER_ARGS="--memory=1536m --memory-swap=1536m" \ + --restart unless-stopped \ + ghcr.io/all-hands-ai/openhands:latest +``` + +The `--add-host` mapping allows the OpenHands container to reach host services if needed. It is optional for basic OVMS communication. + +### Step 5: Wait for OVMS to be ready + +OVMS needs time to download and initialize the model. Check the status: + +```bash +# Poll until model is AVAILABLE +curl -sf http://localhost:${OVMS_REST_PORT}/v1/config | grep AVAILABLE +``` + +Or check container logs: + +```bash +docker logs ovms-llm +``` + +--- + +## Compose Generation Workflow + +The deployment uses three files with distinct responsibilities: + +### docker-compose.template.yml + +Serves as the canonical template for deployment structure. It documents the service architecture and contains environment variable placeholders that the deployment script substitutes with runtime values. + +The template uses placeholders like `${MODEL_ID}`, `${LOCAL_NAME}`, `${TARGET_DEVICE}`, and others. The deployment script uses `envsubst` to replace these placeholders with actual values when generating the runtime compose file. + +Routine deployment customizations should be made through deployment script options, environment variables, or by editing the generated `docker-compose.yml`. Modify the template only when changing the deployment structure itself. + +### docker-compose.yml + +Generated from the template by the deployment script, this file represents the active deployment. It is gitignored and may be edited locally for customization. + +Standard Docker Compose commands operate on this file: +```bash +docker compose up -d +docker compose down +docker compose restart +docker compose restart openhands +docker compose restart ovms-llm +docker compose logs -f ovms-llm +``` + +Local modifications to the generated compose are preserved as long as the deployment configuration remains unchanged. See Deployment Lifecycle below. + +### .ovms-deployment + +Generated by the deployment script, this file stores deployment metadata (deployment fingerprint). The deployment script compares this metadata against the requested configuration to determine whether the existing `docker-compose.yml` can be reused or must be regenerated. + +The deployment fingerprint includes all parameters that affect compose generation: + +- Model identifier and local name +- Target device +- Reasoning parser +- OVMS image variant +- GPU device mapping +- Model cache directory +- Published ports +- Proxy configuration +- Metadata version and generation timestamp + +This file is for internal use by the deployment script and should not be edited manually. + +### Deployment Lifecycle + +The deployment script uses the deployment fingerprint to determine whether to preserve or regenerate the compose file. + +**First deployment:** If no generated compose exists, the script generates `docker-compose.yml` from the template, creates `.ovms-deployment`, and deploys with Docker Compose. + +**Redeploying an identical deployment:** If the stored deployment fingerprint matches the requested deployment (all parameters identical), the script preserves the existing `docker-compose.yml` and all local user modifications, then deploys using the existing compose. This allows users to make local compose customizations without having those changes discarded. + +**Deploying a different configuration:** If any deployment fingerprint parameter changes (model, device, reasoning parser, ports, proxy settings, cache directory, etc.), the script stops the existing deployment, removes the generated compose, regenerates it from the template, regenerates deployment metadata, and deploys the new configuration. This resets local compose modifications because the deployment itself has changed. + +--- + +## Template Architecture + +The `docker-compose.template.yml` file documents the service architecture. See the Compose Generation Workflow section above for details on how the deployment script generates the runtime compose file from this template. + +### Service: ovms-llm + +```yaml +ovms-llm: + image: openvino/model_server:latest-gpu + container_name: ovms-llm +``` + +The `container_name` provides a stable hostname for OpenHands to reach OVMS. + +**Device mapping:** +```yaml +devices: + - /dev/dri:/dev/dri +``` + +Provides GPU device access. For CPU-only deployments, this can be removed. + +**Port publishing:** +```yaml +ports: + - "${OVMS_REST_PORT}:8000" # REST API (default: 8000) + - "${OVMS_GRPC_PORT}:9000" # gRPC API (default: 9000) +``` + +Exposes the OpenAI-compatible REST API and gRPC API. The published host ports are configurable through environment variables (`OVMS_REST_PORT`, `OVMS_GRPC_PORT`), with defaults of 8000 and 9000 respectively. + +**Volume mount:** +```yaml +volumes: + - ${MODEL_CACHE_DIR:-./docker/models}:/models:rw +``` + +Mounts the model cache directory where OVMS materializes models via `--source_model`. The script sets `MODEL_CACHE_DIR` to `${HOME}/ovms-openhands/models` by default; the compose file fallback is `./docker/models` if the variable is unset. + +**Environment:** +```yaml +environment: + HF_TOKEN: ${HF_TOKEN:-} +``` + +Passes the Hugging Face token for gated models. + +**Command:** +```yaml +command: + - --model_repository_path /models + - --source_model ${MODEL_ID} + - --model_name ${LOCAL_NAME} + - --task text_generation + - --target_device ${TARGET_DEVICE} + - --port "9000" + - --rest_port "8000" +``` + +Configures OVMS to use `/models` as the repository, download from Hugging Face, serve under the local name, use the text-generation pipeline, and run on the specified device. Newer OVMS releases detect the appropriate tool parser automatically. + +### Service: openhands + +```yaml +openhands: + image: ghcr.io/all-hands-ai/openhands:latest + container_name: openhands + depends_on: + - ovms-llm +``` + +`depends_on` ensures OVMS starts first (though it does not wait for health). + +**Port publishing:** +```yaml +ports: + - "${OPENHANDS_PORT}:3000" # Web UI (default: 3000) +``` + +**Volume mounts:** +```yaml +volumes: + - /var/run/docker.sock:/var/run/docker.sock + - ./.openhands:/.openhands +``` + +Docker socket allows OpenHands to create runtime sandbox containers. The `.openhands` directory persists settings locally. + +**Environment variables:** +```yaml +environment: + LLM_BASE_URL: http://ovms-llm:8000/v3 + LLM_MODEL: openai/${LOCAL_NAME} + LLM_API_KEY: unused +``` + +Points OpenHands to the OVMS endpoint. Note the `openai/` prefix required by OpenHands. + +**Extra hosts:** +```yaml +extra_hosts: + - host.docker.internal:host-gateway +``` + +Allows the OpenHands container to reach the host machine. Optional for basic OVMS communication but may be needed for certain agent workflows. + +**Generation parameters:** +```yaml +LLM_TEMPERATURE: "0.0" +LLM_MAX_OUTPUT_TOKENS: "500" +LLM_MAX_INPUT_TOKENS: "4096" +LLM_TIMEOUT: "120000" +``` + +Temperature set to 0.0 for deterministic responses. Conservative output limits prevent runaway loops. Increased timeout accommodates CPU inference latency. + +**Sandbox limits:** +```yaml +SANDBOX_DOCKER_ARGS: --memory=1536m --memory-swap=1536m +``` + +Limits memory for OpenHands runtime sandboxes. Adjust based on available host RAM. + +### Network + +```yaml +networks: + ovms-net: + name: ovms-net + driver: bridge +``` + +Creates a shared Docker network for container-to-container communication. OpenHands reaches OVMS via `http://ovms-llm:8000`. + +--- + +## Understanding `deploy_model_ovms.sh` + +The `scripts/deploy_model_ovms.sh` script automates the deployment workflow using the template-based compose generation system. All steps it performs can be done manually using the documented Docker commands. + +### What the Script Does + +**1. Validates prerequisites** + +Checks for Docker and docker compose availability, warns if `HF_TOKEN` is not set for gated models, and validates the target device (`CPU` or `GPU`). + +**2. Normalizes the model name** + +```bash +# "OpenVINO/Qwen3-8b-int8-ov" → "qwen3-8b-int8-ov" +basename "$MODEL_ID" | tr '[:upper:]' '[:lower:]' | tr ' ' '-' +``` + +**3. Creates the model cache directory** + +```bash +mkdir -p "$MODEL_CACHE_DIR" # Defaults to ${HOME}/ovms-openhands/models +``` + +**4. Exports runtime configuration** + +```bash +export MODEL_ID LOCAL_NAME TARGET_DEVICE REASONING_PARSER MODEL_CACHE_DIR HF_TOKEN +export OVMS_IMAGE GPU_DEVICE WSL_LIBS +export OVMS_REST_PORT OVMS_GRPC_PORT OPENHANDS_PORT +export http_proxy https_proxy HTTP_PROXY HTTPS_PROXY no_proxy NO_PROXY +``` + +These variables are consumed by the generated docker-compose.yml via environment variable substitution. + +**6. Deploys OVMS and OpenHands** + +The script determines the deployment action based on existing state: + +- **No existing compose:** Generates `docker-compose.yml` from the template, writes `.ovms-deployment` metadata, and deploys. +- **Existing compose with matching fingerprint:** Preserves the existing compose and user modifications, updates the metadata timestamp, and deploys. +- **Existing compose with different fingerprint:** Stops the deployment, removes the old compose, regenerates from the template, writes new metadata, and deploys. + +Deployment proceeds with: +```bash +docker compose -f "$COMPOSE_FILE" up -d +``` + +**7. Waits for health** + +Polls `http://localhost:${OVMS_REST_PORT}/v1/config` until the model reports `AVAILABLE` (up to 5 minutes). + +**8. Prints the manual equivalent** + +Shows the manual Docker commands equivalent to what the script just performed. + +### Deployment Fingerprint + +The deployment fingerprint stored in `.ovms-deployment` includes all parameters that affect compose generation: + +- `MODEL_ID` — Hugging Face model identifier +- `TARGET_DEVICE` — CPU or GPU +- `LOCAL_NAME` — Normalized model name +- `REASONING_PARSER` — Reasoning parser for chain-of-thought +- `OVMS_IMAGE` — CPU or GPU variant +- `GPU_DEVICE` — Device mapping for GPU passthrough +- `MODEL_CACHE_DIR` — Model cache directory +- `OVMS_REST_PORT`, `OVMS_GRPC_PORT`, `OPENHANDS_PORT` — Published ports +- `http_proxy`, `https_proxy`, `HTTP_PROXY`, `HTTPS_PROXY`, `no_proxy`, `NO_PROXY` — Proxy configuration + +When any of these parameters change between deployments, the script regenerates the compose file. Otherwise, it preserves local modifications. + +### Script Usage + +```bash +./scripts/deploy_model_ovms.sh [OPTIONS] +``` + +**Arguments:** +- `model_id`: Hugging Face model ID (e.g., `OpenVINO/Qwen3-8b-int8-ov`) + +**Options:** +- `--device DEVICE`: Target device (`CPU` or `GPU`, default: `CPU`) +- `--cache-dir DIR`: Model cache directory (default: `${HOME}/ovms-openhands/models`) +- `--compose-file FILE`: Path to docker-compose.yml (default: generated from template) +- `--skip-wait`: Skip health check and return immediately after deploy + +**Environment variable overrides:** +- `HF_TOKEN`: Hugging Face token for gated models +- `LOCAL_NAME`: Override the auto-normalized model name +- `MODEL_CACHE_DIR`: Override model cache directory +- `TARGET_DEVICE`: Override target device +- `REASONING_PARSER`: Override reasoning parser + +--- + +## Debugging OVMS + +### Viewing OVMS Logs + +View OVMS logs from the Docker container: + +```bash +docker logs ovms-llm +``` + +Follow logs in real time while reproducing an issue: + +```bash +docker logs -f ovms-llm +``` + +### Running OVMS with TRACE Logging + +OVMS supports configurable logging levels. The following command demonstrates enabling TRACE logging for a standalone OVMS deployment: + +```bash +ovms \ + --rest_port 9001 \ + --model_repository_path ./models \ + --source_model OpenVINO/Qwen3-8b-int8-ov \ + --task text_generation \ + --target_device CPU \ + --model_name qwen3-8b-int8-ov \ + --log_level TRACE +``` + +TRACE logging provides detailed information helpful for diagnosing issues related to model loading, request processing, inference, and tool-calling behavior. + +### Enabling TRACE Logging with Docker Compose + +To enable TRACE logging for a compose-based deployment, edit the generated `docker-compose.yml` and modify the OVMS command to include the `--log_level` argument. + +The OVMS service in the generated compose uses a shell command to conditionally add reasoning parser arguments. Append the log level argument to the command array: + +```yaml +command: + - /bin/bash + - -c + - | + CMD_ARGS=( + --model_repository_path /models + --source_model "$${MODEL_ID}" + --model_name "$${LOCAL_NAME}" + --task text_generation + --target_device "$${TARGET_DEVICE}" + --port "9000" + --rest_port "8000" + ) + if [[ "$${REASONING_PARSER}" != "none" ]]; then + CMD_ARGS+=(--reasoning_parser "$${REASONING_PARSER}") + fi + CMD_ARGS+=(--log_level TRACE) + exec ovms "$${CMD_ARGS[@]}" +``` + +Append the following line before `exec ovms`: + +```yaml +CMD_ARGS+=(--log_level TRACE) +``` + +### Applying Configuration Changes + +After editing the generated compose file, apply the changes using standard Docker Compose commands: + +```bash +# Restart the OVMS service only +docker compose restart ovms-llm + +# Or recreate the service +docker compose up -d ovms-llm +``` + +### Viewing TRACE Logs + +View logs after restarting: + +```bash +docker logs -f ovms-llm +``` + +--- + +# GPU Acceleration + +## Overview + +OVMS supports GPU inference through OpenVINO. The deployment script already supports GPU deployment using the `--device GPU` flag: + +```bash +./scripts/deploy_model_ovms.sh OpenVINO/Qwen3-8b-int8-ov --device GPU +``` + +GPU enablement depends on the host environment rather than OVMS itself. The same OVMS container and model work on CPU or GPU—only the target device flag changes. Your host system must expose an OpenVINO-compatible Intel GPU runtime for GPU inference to function. + +> **Note:** Platform-specific GPU setup procedures may change with future OpenVINO, Intel GPU runtime, or OS releases. Always refer to the latest OpenVINO and Intel GPU documentation for your platform. + +## Supported Platforms + +GPU inference is supported on the following platforms: + +| Platform | Status | Notes | +| ----------------------- | -------------- | -------------------------------------------------- | +| Native Linux | ✅ Supported | Requires Intel GPU runtime and device access | +| WSL2 + Docker Desktop | ✅ Supported | Requires WSL2 GPU support and `/dev/dxg` (limited to 8b parameter models like the qwen3-8b family) | +| Native Windows | ❌ Not supported | Use WSL2 for Docker-based deployment | + + +## Verifying GPU Availability + +Before deploying with GPU, verify that your host system can access an Intel GPU through OpenVINO. + +### GPU Runtime Installation + +GPU inference requires an OpenVINO-compatible Intel GPU runtime on your host system. Installation procedures vary by platform and distribution. Refer to the official documentation: + +* **OpenVINO GPU documentation:** https://docs.openvino.ai/ +* **Intel GPU runtime documentation:** https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/documentation.html +* **WSL2 GPU setup:** https://learn.microsoft.com/en-us/windows/ai/directml/gpu-cuda-in-wsl + +Follow the official instructions for your platform before proceeding with verification. + +### 1. Verify OpenVINO Detects GPU + +Install OpenVINO on your host and verify GPU availability: + +```bash +python3 -c "from openvino import Core; print(Core().available_devices)" +``` + +Expected output: + +```text +['CPU', 'GPU'] +``` + +If only `['CPU']` is returned, the GPU runtime is not installed correctly or no compatible GPU is available. + +### 2. Verify GPU Device Access + +**Native Linux:** Check that GPU devices are accessible: + +```bash +ls -la /dev/dri/ +``` + +You should see `renderD*` and `card*` devices. + +**WSL2:** Verify the DirectX device exists: + +```bash +ls -la /dev/dxg +``` + +Expected output: + +```text +crw-rw-rw- 1 root root 247, 0 ... /dev/dxg +``` + +> **Note:** Under WSL2, `/dev/dri` is **not expected** to be present. GPU access is provided through `/dev/dxg` instead. Native Linux exposes GPUs through `/dev/dri`, while WSL2 exposes GPUs through `/dev/dxg`. + + +If only `['CPU']` is returned, ensure Docker Desktop WSL2 integration is enabled with GPU support. + +### 3. Deploy with GPU + +Run the deployment script with GPU device: + +```bash +./scripts/deploy_model_ovms.sh OpenVINO/Qwen3-8b-int8-ov --device GPU +``` + +### 5. Verify OVMS GPU Usage + +Check that OVMS detects the GPU: + +```bash +docker logs ovms-llm | grep -i "available devices" +``` + +Expected output: + +```text +Available devices for OpenVINO: CPU, GPU +``` + +Verify the model reaches `AVAILABLE` status: + +```bash +curl -s http://localhost:${OVMS_REST_PORT}/v1/config | grep AVAILABLE +``` + +Should return output containing `AVAILABLE`. + +# Troubleshooting + +### GPU Not Detected + +If `python3 -c "from openvino import Core; print(Core().available_devices)"` returns only `['CPU']`: + +1. Verify Intel GPU is physically present: `lspci | grep -Ei "vga|display"` +2. Refer to the GPU runtime installation documentation for your platform +3. Check OpenVINO installation: `pip3 show openvino` +4. Try restarting the WSL instance: `wsl --shutdown` (from Windows) + +### `/dev/dri` Missing Under WSL2 + +This is expected behavior under WSL2. GPU access is provided through `/dev/dxg` instead: + +```bash +ls -la /dev/dxg # Should exist +ls -la /dev/dri # May not exist—this is normal +``` + +Native Linux exposes GPUs through `/dev/dri`, while WSL2 exposes GPUs through `/dev/dxg`. The docker-compose.template.yml includes `/dev/dri` device mapping for native Linux deployments. + +### Docker Cannot Access GPU + +If OVMS logs show no GPU available: + +1. Ensure Docker Desktop (Windows) or Docker daemon (Linux) has GPU access enabled +2. On WSL2, verify Docker Desktop WSL2 integration is enabled +3. Check container logs: `docker logs ovms-llm | grep -i gpu` +4. Verify the `--device GPU` flag is being passed to OVMS + +### GPU Plugin Unavailable + +If OVMS starts but falls back to CPU despite `--device GPU`: + +```bash +docker logs ovms-llm | grep -i "target device" +``` + +Look for warnings about plugin loading. This may indicate: +- GPU runtime not available to the container +- Incompatible OpenVINO version + +### OpenVINO Python Not Installed + +If running verification commands produces: + +```text +ModuleNotFoundError: No module named 'openvino' +``` + +This means the OpenVINO Python package is not installed in the active Python environment, or the correct virtual environment is not activated. Install OpenVINO or activate the appropriate environment before running the verification commands. + +### Large-Model GPU Allocation Failure under WSL2 + +GPU inference under WSL2 works for smaller models, but larger OpenVINO models can fail during GPU initialization with a USM Host allocation error. In our testing: + +- `OpenVINO/Qwen3-8b-int8-ov` → worked with GPU under WSL2. +- `OpenVINO/Qwen3-14b-int8-ov` → failed with GPU under WSL2 with a USM allocation error. +- `OpenVINO/Qwen3-Coder-30B-A3B-Instruct-int4-ov` → failed with GPU under WSL2 with a USM allocation error. + +The representative error was: + +[CL ext] Can not allocate ... bytes for USM Host + +This is not simply a system-RAM exhaustion issue. During testing, WSL had approximately 27 GiB of available memory, and the same Qwen3-Coder-30B-A3B-Instruct-int4-ov model successfully loaded and served through OVMS on native Windows using `--target_device GPU`. + +CPU inference remains functional under WSL2. Native Windows OVMS was also successfully tested with Qwen3-Coder-30B-A3B-Instruct-int4-ov on GPU. For larger GPU models that encounter this WSL2 allocation failure, the recommended environment is a native Linux machine. WSL2 remains suitable for CPU inference and for GPU models that have been validated to work within the available memory and runtime constraints. + +Native Linux uses a Linux → Intel GPU runtime → OpenVINO GPU path, while WSL2 uses a Windows → WSL2 GPU layer → Linux/OpenVINO GPU runtime path. The issue occurs in the WSL2 GPU execution path during large-model initialization. + +## Performance Verification + +Verify GPU acceleration is working by comparing CPU and GPU performance with identical prompts. + +**1. Run a test prompt on CPU:** + +```bash +curl -X POST http://localhost:${OVMS_REST_PORT}/v3/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "qwen3-8b-int8-ov", + "messages": [{"role": "user", "content": "Explain recursion in programming with examples."}], + "max_tokens": 200 + }' +``` + +Record the total response time. + +**2. Redeploy with GPU:** + +```bash +docker compose -f docker-compose.yml down +./scripts/deploy_model_ovms.sh OpenVINO/Qwen3-8b-int8-ov --device GPU +``` + +**3. Run the same prompt on GPU:** + +Use the same curl command and compare response times. + +**4. Metrics to compare:** + +- **Total response time:** Compare end-to-end request duration +- **Tokens per second:** Calculate by dividing output tokens by generation time +- **CPU utilization:** GPU inference typically shows different CPU utilization patterns + +GPU acceleration is generally expected to improve throughput and reduce response time for sufficiently large inference workloads. Actual performance depends on the model, hardware, prompt length, and runtime configuration. + +> **Note:** The first inference on GPU includes model compilation overhead. Treat the first request as a warm-up; subsequent requests will reflect true GPU performance. + +**Example verification:** + +```bash +# Time the request +time curl -X POST http://localhost:${OVMS_REST_PORT}/v3/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "qwen3-8b-int8-ov", + "messages": [{"role": "user", "content": "Write a Python function to check if a number is prime."}], + "max_tokens": 150 + }' +``` + +Compare the `real` time between CPU and GPU runs. + +--- + +## GPT-OSS Temporary Workaround + +### Issue + +GPT-OSS models exposed through an OpenAI-compatible endpoint may not be recognized by OpenHands as a supported model. OpenHands may treat the model as an unrecognized OpenAI-compatible model, which can trigger a compatibility fallback for tool handling. This configuration may result in errors such as `empty content array` during agent execution. + +### Workaround + +As a temporary testing workaround, configure the deployment to present `gpt-4o` as the model identifier to OpenHands, regardless of the actual model being served. + +Set the local name when running the deployment script: + +```bash +LOCAL_NAME=gpt-4o ./scripts/deploy_model_ovms.sh +``` + +Then configure OpenHands to use: + +```text +openai/gpt-4o +``` + +### Important Notes + +This workaround **does not** change the actual model running inside OVMS. It only changes the identifier presented to OpenHands so that it follows the standard GPT-4o request flow. + +The backend model served by OVMS remains your GPT-OSS model. The `LOCAL_NAME` variable simply controls the model name that OVMS advertises and that OpenHands uses in its requests. + +This is intended as a **temporary testing workaround**, not the recommended long-term configuration. Once OpenHands properly supports GPT-OSS/OpenAI-compatible mappings, users should revert to using the actual model name. + +This workaround is intended only for current OpenHands compatibility and can be removed once GPT-OSS models are supported natively. diff --git a/demos/integration_with_OpenHands/README.md b/demos/integration_with_OpenHands/README.md new file mode 100644 index 0000000000..21e48b3240 --- /dev/null +++ b/demos/integration_with_OpenHands/README.md @@ -0,0 +1,407 @@ +# OpenHands Integration with OpenVINO Model Server {#ovms_demos_integration_with_openhands} + +## Description + +[OpenHands](https://github.com/All-Hands-AI/OpenHands) is an open-source software engineering agent that automates coding tasks through iterative LLM inference, tool execution, and runtime sandbox environments. Unlike simple chat interfaces, OpenHands maintains long-running conversations, creates code execution sandboxes, and performs multi-step problem solving. + +This demo integrates OpenHands with [OpenVINO Model Server](https://github.com/openvinotoolkit/model_server) using OVMS's OpenAI-compatible REST API. It demonstrates how to deploy OVMS as a backend for OpenHands, enabling agent workflows on local hardware with OpenVINO-optimized models. + +This README covers the recommended deployment workflow. For manual Docker deployment and implementation details, see [ADVANCED_DEPLOYMENT.md](ADVANCED_DEPLOYMENT.md). + +## Architecture + +```mermaid +flowchart TD + U[User] --> OH[OpenHands Container] + OH --> OVMS[OpenVINO Model Server] + OH --> S[Runtime Sandbox] + OVMS --> LLM[MediaPipe LLM Graph and Tool Parser] + LLM --> M[OpenVINO Model] +``` + +Ensure the required ports are available on your host. +OpenHands maintains conversation state and creates isolated Docker containers for code execution. It requires an OpenAI-compatible LLM endpoint with models that have sufficient context capacity and coding capability. + +OVMS serves generative models through an OpenAI-compatible REST API, handling model retrieval, OpenVINO conversion, and graph generation. Newer OVMS releases automatically detect the appropriate tool parser for supported models and run on CPU or GPU with OpenVINO optimization. + +For detailed request flow and configuration requirements, see [ADVANCED_DEPLOYMENT.md](ADVANCED_DEPLOYMENT.md). + +--- + +## Prerequisites + +- **Host architecture:** x86_64 +- **Operating system:** Linux (Docker-based deployment) +- **Docker Engine:** Installed and running +- **Docker Compose:** Plugin v2 or standalone +- **Memory:** Minimum 8GB RAM; 16GB+ recommended for agent workflows +- **Hugging Face account:** For model access (gated models may require token) + +### Network and Port Usage + +| Port | Component | Purpose | +|------|-----------|-----------------------------| +| 8000 | OVMS | OpenAI-compatible REST API (default) | +| 9000 | OVMS | gRPC API (not used here, default) | +| 3000 | OpenHands | Web UI (default) | + +The default published ports are 8000 (OVMS REST), 9000 (OVMS gRPC), and 3000 (OpenHands). You can override these defaults by setting environment variables before running the deployment script: + +```bash +export OVMS_REST_PORT=18000 +export OVMS_GRPC_PORT=19000 +export OPENHANDS_PORT=3300 + +./scripts/deploy_model_ovms.sh OpenVINO/Qwen3-8b-int8-ov +``` + +Ensure the required ports are available on your host. + +### Proxy Support + +In environments requiring HTTP/HTTPS proxies, export standard proxy environment variables before running the deployment script: + +```bash +export http_proxy=http://your-proxy:port +export https_proxy=http://your-proxy:port + +./scripts/deploy_model_ovms.sh OpenVINO/Qwen3-8b-int8-ov +``` + +The deployment script automatically forwards these variables to the Docker containers. + +--- + +## Preparing the Model + +### Choosing a Compatible Model + +OpenHands requires models with instruction-following capability, coding proficiency, sufficient context window (4096+ tokens), and tool calling support. + +> **Note:** Examples use `OpenVINO/Qwen3-8b-int8-ov`. Other compatible models may also be used. + +### Tool Parser Handling + +Newer OVMS releases automatically detect the appropriate tool parser for supported models, so you do not need to select or configure one in the deployment script or compose file. If tool calls are not being extracted correctly, make sure you are running a recent OVMS release and that the model is supported by OpenHands. + +For details on OVMS model retrieval and workspace layout, see [ADVANCED_DEPLOYMENT.md](ADVANCED_DEPLOYMENT.md). + +--- + +## Deployment + +This demo uses a deployment template and helper script to automate container orchestration. The deployment architecture separates the template (tracked in Git) from the generated compose file (used for runtime). + +### Compose Files and Responsibilities + +The repository contains three compose-related files with distinct purposes: + +| File | Purpose | Managed By | +|------|---------|------------| +| `docker-compose.template.yml` | Deployment template with placeholders | Git (source of truth for deployment structure) | +| `docker-compose.yml` | Generated compose file for active deployment | Deployment script (may be edited locally) | +| `.ovms-deployment` | Deployment metadata (deployment fingerprint) | Deployment script (internal use) | + +**Template (`docker-compose.template.yml`):** Committed to Git, serves as the canonical template for deployment structure. Contains environment variable placeholders that the deployment script substitutes with runtime values. Routine customizations should be made through deployment script options or by editing the generated compose file. Modify the template only when changing the deployment structure itself. + +**Generated compose (`docker-compose.yml`):** Created from the template on first deployment, gitignored. Represents the active deployment and may be edited locally for customization. Standard Docker Compose commands operate on this file. + +**Deployment metadata (`.ovms-deployment`):** Generated by the deployment script. Stores metadata describing the generated deployment. The deployment script compares this metadata against the requested configuration to determine whether the existing `docker-compose.yml` can be reused or must be regenerated. This deployment fingerprint is compared on subsequent runs. Not intended for manual editing. + +### Deployment Lifecycle + +The deployment script uses the deployment fingerprint to determine whether to preserve or regenerate the compose file: + +**First deployment:** If no generated compose exists, the script generates `docker-compose.yml` from the template, creates `.ovms-deployment`, and deploys with Docker Compose. + +**Redeploying an identical deployment:** If the stored deployment fingerprint matches the requested deployment, the script preserves the existing `docker-compose.yml` and all local user modifications, then deploys using the existing compose. + +**Deploying a different configuration:** If the deployment fingerprint changes (model, device, reasoning parser, ports, proxy settings, cache directory, etc.), the script stops the existing deployment, removes the generated compose, regenerates it from the template, regenerates deployment metadata, and deploys the new configuration. + +This design allows users to make local compose customizations (such as enabling TRACE logging) without having those changes discarded when redeploying the same configuration. + +### Running the Deployment + +Clone the repository and navigate to the demo directory before proceeding. + +**Prerequisites:** Docker Engine, Docker Compose, 8GB+ RAM, and `HF_TOKEN` for gated models. + +1. **Clone the repository:** + ```bash + git clone https://github.com/openvinotoolkit/model_server.git + cd model_server/demos/integration_with_OpenHands + ``` + +2. **Set your Hugging Face token** (required for gated models like Llama, Mistral): + ```bash + export HF_TOKEN="your_token_here" + ``` + +3. **Run the deployment script:** + ```bash + ./scripts/deploy_model_ovms.sh OpenVINO/Qwen3-8b-int8-ov + ``` + + The script validates your environment, prepares the model, generates the compose file if needed, and launches both containers. + +4. **Verify the deployment** (see next section) + +### Using Intel GPU (Optional) + +The deployment script supports GPU inference for improved performance: + +```bash +./scripts/deploy_model_ovms.sh OpenVINO/Qwen3-8b-int8-ov --device GPU +``` + +Before using GPU inference, ensure your host system exposes an OpenVINO-compatible Intel GPU runtime (see [GPU Acceleration](ADVANCED_DEPLOYMENT.md#gpu-acceleration) for platform-specific setup and verification). GPU support depends on your host environment—the same OVMS deployment works on CPU or GPU based on the `--device` flag. + +**Optional parameters:** +```bash +# Specify device, reasoning parser, or cache directory +./scripts/deploy_model_ovms.sh OpenVINO/Qwen3-8b-int8-ov \ + --device CPU \ + --reasoning-parser qwen3 \ + --cache-dir ~/custom-models + +# Skip health check for faster feedback +./scripts/deploy_model_ovms.sh OpenVINO/Qwen3-8b-int8-ov --skip-wait +``` + +### Container Management + +Use Docker Compose commands and the deployment script as complementary workflows: + +#### Standard Docker Compose Commands + +Use standard Docker Compose commands for routine container operations and local modifications to the generated compose file: + +```bash +# View status +docker compose ps + +# View logs +docker compose logs +docker compose logs -f ovms-llm + +# Restart all services +docker compose restart + +# Restart only OpenHands +docker compose restart openhands + +# Restart only OVMS +docker compose restart ovms-llm + +# Stop services +docker compose stop + +# Start services +docker compose start + +# Remove services (preserves compose file) +docker compose down +``` + +These commands operate on `docker-compose.yml` in the current directory. + +#### Deployment Script + +Run `deploy_model_ovms.sh` when changing deployment configuration: + +- Deploying a different model +- Switching between CPU and GPU +- Changing reasoning parser +- Changing published ports +- Changing proxy configuration +- Regenerating the deployment + +The script compares the requested configuration against the stored deployment metadata and regenerates the compose file only when the configuration has changed. + +For manual Docker deployment and implementation details, see [ADVANCED_DEPLOYMENT.md](ADVANCED_DEPLOYMENT.md). + +--- + +## Verifying the Deployment + +Verify the integration in two stages: first OVMS directly, then OpenHands. + +### Stage 1: Verify OVMS + +The deployment script sets the published OVMS REST port (default: 8000). Use this port for verification. + +**Check health:** +```bash +# Substitute your configured OVMS_REST_PORT if not using the default +curl -s http://localhost:${OVMS_REST_PORT:-8000}/v1/config | jq . +``` + +The response should include `"model_status": "AVAILABLE"`. + +**Test a completion request:** +```bash +curl -X POST http://localhost:${OVMS_REST_PORT:-8000}/v3/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "qwen3-8b-int8-ov", + "messages": [{"role": "user", "content": "Say hello"}], + "max_tokens": 10 + }' +``` + +If OVMS fails to respond, check `docker logs ovms-llm`, verify the model downloaded at `$MODEL_CACHE_DIR`, and ensure `HF_TOKEN` is set if needed. + +### Stage 2: Verify OpenHands + +The deployment script sets the published OpenHands port (default: 3000). + +1. **Open the web UI** at `http://localhost:${OPENHANDS_PORT:-3000}` + ![OpenHands UI]() + +2. **Configure the OVMS-backed model:** + - Click **Settings** → **LLM** + - Enable **Advanced** mode if needed + - Set **Custom Model:** `openai/qwen3-8b-int8-ov` + - Set **Base URL:** `http://ovms-llm:8000/v3` + - Set **API Key:** `unused` + - Click **Save** + + ![OpenHands LLM Configuration](screenshots/Pasted%20image.png) + +> **Known workaround for GPT-OSS models:** Some GPT-OSS models exposed through an OpenAI-compatible endpoint may not yet work correctly with OpenHands, resulting in errors such as `empty content array` during agent execution. As a temporary workaround, set `LOCAL_NAME=gpt-4o` when deploying OVMS and configure OpenHands to use `openai/gpt-4o`. This only changes the model identifier exposed to OpenHands; the backend model served by OVMS remains unchanged. See the Advanced Deployment guide for additional details. + +3. **Create an agent task:** + ``` + Create a Python function that calculates the factorial of a number. + ``` + +4. **Verify behavior:** + - OpenHands creates a runtime sandbox container + - The agent writes and tests code + - OVMS logs show incoming `/v3/chat/completions` requests + +**Common issues:** +- API errors: Verify `LLM_BASE_URL` and `LLM_MODEL` match OVMS configuration +- Slow responses: CPU inference is slower than GPU; consider `LLM_TIMEOUT` setting +- Task failures: The model may lack coding capability; try a larger model +- Tool-call failures: newer OVMS releases detect the tool parser automatically, so verify you are on a recent OVMS version and that the selected model is supported by OpenHands + +--- + +## Troubleshooting + +### OVMS Container Issues + +**OVMS exits immediately after starting** + +Check `docker logs ovms-llm`. Possible causes: +- Invalid `HF_TOKEN` for gated model +- Invalid model ID +- Device not available (change `TARGET_DEVICE` to `CPU`) +- Volume mount error (ensure `MODEL_CACHE_DIR` exists) +- Permission denied on `/models` (directory must be writable by OVMS container user) + +**Permission denied on `/models`** + +On some WSL2/Docker configurations, OVMS may fail to start with an error like: + +``` +Libgit2 clone error: failed to make directory '/models/OpenVINO': Permission denied +``` + +This occurs when the mounted model cache directory is not writable by the OVMS container user. The issue is specific to the permissions of the host directory, not OVMS itself. + +Verify the directory permissions: + +```bash +ls -ld "$MODEL_CACHE_DIR" +``` + +Fix by making the directory writable by all users: + +```bash +chmod a+rwx "$MODEL_CACHE_DIR" +``` + +Then redeploy: + +```bash +docker compose down +./scripts/deploy_model_ovms.sh +``` + +**Model status is not `AVAILABLE`** + +Check `curl -s http://localhost:8000/v1/config` (default port; override with `OVMS_REST_PORT`). Possible causes: +- Model still downloading (wait longer for large models) +- Out of memory (check host RAM; model may be too large) +- Older OVMS release in use; newer versions auto-detect the tool parser + +**Connection refused** + +Possible causes: +- OVMS container not running (`docker ps`) +- Wrong port mapping (verify the published OVMS REST port matches your configured `OVMS_REST_PORT` setting) +- Firewall blocking the configured port + +### OpenHands Container Issues + +**API errors** + +Check `docker logs openhands`. Possible causes: +- `LLM_BASE_URL` incorrect (should be `http://ovms-llm:8000/v3`) +- `LLM_MODEL` format wrong (should be `openai/`) +- OVMS not ready (verify model is `AVAILABLE`) + +**Fails to create runtime sandboxes** + +Check `docker logs openhands | grep -i sandbox`. Possible causes: +- Docker socket not mounted +- Permission denied on Docker socket +- Memory limit too low (increase `SANDBOX_DOCKER_ARGS`) + +### Performance Issues + +**Slow responses** + +CPU inference is inherently slower than GPU. First-token latency is higher for CPU-optimized models. Smaller models are faster. Check resource usage with `docker stats`. + +**Agent tasks fail or produce poor results** + +Possible causes: +- Model lacks coding capability (try a model optimized for code) +- Context window too small (increase `LLM_MAX_INPUT_TOKENS`) +- Output limit too low (increase `LLM_MAX_OUTPUT_TOKENS`) +- Temperature too low (try `0.1` or `0.2`) + +### Network Issues + +**Containers cannot communicate** + +Check `docker network inspect ovms-net`. Verify both containers use `ovms-net` and that OpenHands expects the `ovms-llm` hostname. + +### Getting Help + +- [OpenHands documentation](https://docs.all-hands.dev/) +- [OVMS documentation](https://github.com/openvinotoolkit/model_server) + + +## References + +- [OpenHands Project](https://github.com/All-Hands-AI/OpenHands) +- [OpenHands Documentation](https://docs.all-hands.dev/) +- [OpenVINO Model Server](https://github.com/openvinotoolkit/model_server) +- [OVMS Documentation](https://github.com/openvinotoolkit/model_server/tree/main/docs) +- [Hugging Face Models](https://huggingface.co/models) +- [OpenAI API Specification](https://platform.openai.com/docs/api-reference) + +### Related OVMS Demos + +- [integration_with_OpenWebUI](../integration_with_OpenWebUI/) — General model interface integration +- [llm_standalone_flow](../llm_standalone_flow/) — Standalone LLM deployment + +### Model Documentation + +- [Qwen Models](https://huggingface.co/Qwen) +- [Llama Models](https://huggingface.co/meta-llama) +- [Mistral Models](https://huggingface.co/mistralai) diff --git a/demos/integration_with_OpenHands/docker-compose.template.yml b/demos/integration_with_OpenHands/docker-compose.template.yml new file mode 100644 index 0000000000..1d8b32b56d --- /dev/null +++ b/demos/integration_with_OpenHands/docker-compose.template.yml @@ -0,0 +1,105 @@ +version: "3.8" + +services: + ovms-llm: + image: ${OVMS_IMAGE} + container_name: ovms-llm + entrypoint: [] + user: "${HOST_UID}:${HOST_GID}" +${OVMS_GPU_CONFIG} + ports: + - "${OVMS_REST_PORT:-8000}:8000" # REST API + - "${OVMS_GRPC_PORT:-9000}:9000" # gRPC API + volumes: + # Model cache directory - OVMS will materialize models here on --source_model pull + # Defaults to ${HOME}/ovms-openhands/models if not set + - ${MODEL_CACHE_DIR:-./docker/models}:/models:rw + # WSL library dependencies for GPU passthrough (harmless on native Linux) + - ${WSL_LIBS:-/.nonexistent:/.nonexistent:ro} + environment: + # Hugging Face token for gated models. + HF_TOKEN: ${HF_TOKEN:-} + # Model configuration variables + MODEL_ID: ${MODEL_ID} + LOCAL_NAME: ${LOCAL_NAME} + TARGET_DEVICE: ${TARGET_DEVICE} + REASONING_PARSER: ${REASONING_PARSER} + # Proxy configuration (forwarded from host if set) + http_proxy: ${http_proxy:-} + https_proxy: ${https_proxy:-} + HTTP_PROXY: ${HTTP_PROXY:-} + HTTPS_PROXY: ${HTTPS_PROXY:-} + no_proxy: ${no_proxy:-} + NO_PROXY: ${NO_PROXY:-} + command: + - /bin/bash + - -c + - | + CMD_ARGS=( + --model_repository_path /models + --source_model "${MODEL_ID}" + --model_name "${LOCAL_NAME}" + --task text_generation + --target_device "${TARGET_DEVICE}" + --port "9000" + --rest_port "8000" + ) + if [[ "${REASONING_PARSER}" != "none" ]]; then + CMD_ARGS+=(--reasoning_parser "${REASONING_PARSER}") + fi + exec ovms "$${CMD_ARGS[@]}" + networks: + - ovms-net + restart: unless-stopped + + openhands: + image: ghcr.io/all-hands-ai/openhands:latest + container_name: openhands + depends_on: + - ovms-llm + ports: + - "${OPENHANDS_PORT:-3000}:3000" # Web UI + extra_hosts: + # Allows OpenHands container to reach host services if needed + - host.docker.internal:host-gateway + environment: + # OVMS OpenAI-compatible endpoint + LLM_BASE_URL: http://ovms-llm:8000/v3 + # Model identifier - must include 'openai/' prefix for OpenHands. + LLM_MODEL: openai/${LOCAL_NAME} + # OpenHands requires a non-empty API key even though OVMS doesn't authenticate + LLM_API_KEY: unused + # Generation parameters + LLM_TEMPERATURE: "0.0" + # Generation circuit breaker - prevents runaway agent loops. + # Increase for complex tasks, but keep conservative for stability. + LLM_MAX_OUTPUT_TOKENS: "5000" + # Context window limit - adjust based on model's actual capacity. + # Some modern models support 32k+ tokens. + LLM_MAX_INPUT_TOKENS: "4096" + # Request timeout in milliseconds. Increased from default to accommodate + # larger models, CPU inference, and first-token latency. + LLM_TIMEOUT: "120000" + # Memory limit for OpenHands runtime sandboxes (agent code execution). + # Adjust based on available host RAM. + SANDBOX_DOCKER_ARGS: --memory=1536m --memory-swap=1536m + # Proxy configuration (forwarded from host if set) + http_proxy: ${http_proxy:-} + https_proxy: ${https_proxy:-} + HTTP_PROXY: ${HTTP_PROXY:-} + HTTPS_PROXY: ${HTTPS_PROXY:-} + no_proxy: ${no_proxy:-} + NO_PROXY: ${NO_PROXY:-} + volumes: + # Docker socket for OpenHands to create runtime sandbox containers + - /var/run/docker.sock:/var/run/docker.sock + # Persistent OpenHands settings - repo-local for transparency + - ./.openhands:/.openhands + networks: + - ovms-net + restart: unless-stopped + +networks: + ovms-net: + name: ovms-net + driver: bridge diff --git a/demos/integration_with_OpenHands/screenshots/.gitkeep b/demos/integration_with_OpenHands/screenshots/.gitkeep new file mode 100644 index 0000000000..e69de29bb2 diff --git a/demos/integration_with_OpenHands/screenshots/LLM-settings.png b/demos/integration_with_OpenHands/screenshots/LLM-settings.png new file mode 100644 index 0000000000..182e27b7f8 Binary files /dev/null and b/demos/integration_with_OpenHands/screenshots/LLM-settings.png differ diff --git a/demos/integration_with_OpenHands/screenshots/OpenHands-UI.png b/demos/integration_with_OpenHands/screenshots/OpenHands-UI.png new file mode 100644 index 0000000000..e13a30555a Binary files /dev/null and b/demos/integration_with_OpenHands/screenshots/OpenHands-UI.png differ diff --git a/demos/integration_with_OpenHands/scripts/deploy_model_ovms.sh b/demos/integration_with_OpenHands/scripts/deploy_model_ovms.sh new file mode 100755 index 0000000000..ceef5827dc --- /dev/null +++ b/demos/integration_with_OpenHands/scripts/deploy_model_ovms.sh @@ -0,0 +1,694 @@ +#!/usr/bin/env bash +# +# deploy_model_ovms.sh +# +# Convenience helper for deploying OpenVINO Model Server with OpenHands configuration. +# +# This script automates the runtime environment setup and OVMS deployment documented +# in the README.md. It is optional - users can achieve the same result by following +# the manual workflow documented in the README. +# +# Compose file workflow: +# - docker-compose.template.yml is the immutable template (tracked in Git) +# - docker-compose.yml is generated from the template (not tracked) +# - .ovms-deployment stores the complete deployment fingerprint +# - The script compares the deployment fingerprint: +# * Identical fingerprint: preserves existing compose (user edits kept) +# * Different fingerprint: regenerates compose (discards old edits) +# - Users may freely edit the generated docker-compose.yml after deployment +# +# Usage: +# ./scripts/deploy_model_ovms.sh [OPTIONS] +# +# Arguments: +# model_id Hugging Face model ID (e.g., "OpenVINO/qwen3-8b-int8-ov") +# +# Options: +# --device DEVICE Target device: CPU or GPU (default: CPU) +# --reasoning-parser PARSER Reasoning parser: gemma4, gptoss, or none (default: auto-resolved) +# --cache-dir DIR Model cache directory (default: ${HOME}/ovms-openhands/models) +# --compose-file FILE Path to docker-compose.yml (default: /docker-compose.yml, generated from template) +# --skip-wait Skip health check and return immediately after deploy +# +# Example: +# ./scripts/deploy_model_ovms.sh OpenVINO/qwen3-0.6b-int8-ov --device CPU +# +# Environment Variables: +# HF_TOKEN Hugging Face token for gated models (required for some models) +# LOCAL_NAME Override the local model name (default: auto-normalized from model_id) +# MODEL_CACHE_DIR Override model cache directory +# TARGET_DEVICE Override target device +# REASONING_PARSER Override reasoning parser +# OVMS_REST_PORT OVMS REST API published port (default: 8000) +# OVMS_GRPC_PORT OVMS gRPC API published port (default: 9000) +# OPENHANDS_PORT OpenHands Web UI published port (default: 3000) +# http_proxy HTTP proxy for container network access +# https_proxy HTTPS proxy for container network access +# HTTP_PROXY HTTP proxy (uppercase variant) +# HTTPS_PROXY HTTPS proxy (uppercase variant) +# no_proxy No-proxy list for container network access +# NO_PROXY No-proxy list (uppercase variant) +# +# The script exports environment variables consumed by docker-compose.yml: +# MODEL_ID, LOCAL_NAME, TARGET_DEVICE, REASONING_PARSER, MODEL_CACHE_DIR, GPU_DEVICE, WSL_LIBS +# HOST_UID, HOST_GID, RENDER_GID +# OVMS_REST_PORT, OVMS_GRPC_PORT, OPENHANDS_PORT +# http_proxy, https_proxy, HTTP_PROXY, HTTPS_PROXY, no_proxy, NO_PROXY + +set -euo pipefail + +################################################################################ +# Constants and Directory Resolution +################################################################################ + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +DEMO_ROOT="$(dirname "$SCRIPT_DIR")" +COMPOSE_TEMPLATE="${DEMO_ROOT}/docker-compose.template.yml" +DEFAULT_COMPOSE_FILE="${DEMO_ROOT}/docker-compose.yml" +DEPLOYMENT_METADATA="${DEMO_ROOT}/.ovms-deployment" +DEFAULT_MODEL_CACHE_DIR="${HOME}/ovms-openhands/models" +OVMS_CONTAINER_NAME="ovms-llm" +OPENHANDS_CONTAINER_NAME="openhands" +DOCKER_NETWORK="ovms-net" + +# Configurable ports with defaults (can be overridden via environment variables) +OVMS_REST_PORT="${OVMS_REST_PORT:-8000}" +OVMS_GRPC_PORT="${OVMS_GRPC_PORT:-9000}" +OPENHANDS_PORT="${OPENHANDS_PORT:-3000}" + +# Reasoning parser mapping: model family patterns to parser names +declare -A REASONING_PARSERS=( + ["Gemma4"]="gemma4" + ["gemma4"]="gemma4" + ["Gemma-4"]="gemma4" + ["gemma-4"]="gemma4" + ["GPT-OSS"]="gptoss" + ["gpt-oss"]="gptoss" +) + +################################################################################ +# GPU Device Detection +################################################################################ + +detect_gpu_device() { + # Check for WSL2 first (GPU passthrough device) + if [[ -e /dev/dxg ]]; then + echo "/dev/dxg:/dev/dxg" + # Fallback: check /proc/version for WSL signature + elif grep -qi microsoft /proc/version 2>/dev/null; then + echo "/dev/dxg:/dev/dxg" + # Native Linux with GPU device + elif [[ -e /dev/dri ]]; then + echo "/dev/dri:/dev/dri" + # No GPU device detected + else + echo "" + fi +} + +################################################################################ +# Argument Parsing +################################################################################ + +print_usage() { + grep '^#' "${BASH_SOURCE[0]}" | grep -v '^#!/usr/bin/env' | sed 's/^# //' | sed 's/^#//' + exit 0 +} + +parse_args() { + if [[ $# -eq 0 ]] || [[ "$1" == "-h" ]] || [[ "$1" == "--help" ]]; then + print_usage + fi + + MODEL_ID="$1" + shift + + # Initialize from environment or defaults + TARGET_DEVICE="${TARGET_DEVICE:-CPU}" + REASONING_PARSER="${REASONING_PARSER:-}" + MODEL_CACHE_DIR="${MODEL_CACHE_DIR:-${DEFAULT_MODEL_CACHE_DIR}}" + COMPOSE_FILE="${DEFAULT_COMPOSE_FILE}" + SKIP_WAIT=false + + while [[ $# -gt 0 ]]; do + case "$1" in + --device) + TARGET_DEVICE="$2" + shift 2 + ;; + --reasoning-parser) + REASONING_PARSER="$2" + shift 2 + ;; + --cache-dir) + MODEL_CACHE_DIR="$2" + shift 2 + ;; + --compose-file) + COMPOSE_FILE="$2" + shift 2 + ;; + --skip-wait) + SKIP_WAIT=true + shift + ;; + *) + echo "ERROR: Unknown option: $1" >&2 + echo "Use --help for usage information." >&2 + exit 1 + ;; + esac + done +} + +################################################################################ +# Validation Functions +################################################################################ + +validate_prerequisites() { + local errors=0 + + # Check Docker + if ! command -v docker &>/dev/null; then + echo "ERROR: Docker is not installed or not in PATH" >&2 + errors=$((errors + 1)) + fi + + # Check Docker Compose plugin + if ! docker compose version &>/dev/null; then + echo "ERROR: docker compose plugin is not available" >&2 + echo "Install Docker Compose v2 or use 'docker-compose' standalone" >&2 + errors=$((errors + 1)) + fi + + # Check HF_TOKEN for gated models (warning only) + if [[ -z "${HF_TOKEN:-}" ]]; then + if [[ "$MODEL_ID" =~ meta-llama|Llama|mistralai ]]; then + echo "WARNING: HF_TOKEN is not set. This model may require authentication." >&2 + echo "Set HF_TOKEN environment variable for gated models." >&2 + fi + fi + + # Validate compose template exists + if [[ ! -f "$COMPOSE_TEMPLATE" ]]; then + echo "ERROR: docker-compose.template.yml not found: $COMPOSE_TEMPLATE" >&2 + errors=$((errors + 1)) + fi + + return $errors +} + +validate_device() { + local device="$1" + + case "$device" in + CPU|GPU) + # Valid device types + ;; + *) + echo "ERROR: Invalid device: $device" >&2 + echo "Supported devices: CPU, GPU" >&2 + exit 1 + ;; + esac +} + +################################################################################ +# Model Name Normalization +################################################################################ + +normalize_model_name() { + local model_id="$1" + + # Convert Hugging Face model ID to filesystem-safe local name + # e.g., "OpenVINO/qwen3-0.6b-int8-ov" → "qwen3-0.6b-int8-ov" + basename "$model_id" | tr '[:upper:]' '[:lower:]' | tr ' ' '-' +} + +resolve_reasoning_parser() { + local model_id="$1" + local override="${2:-}" + + # If override provided, use it + if [[ -n "$override" ]]; then + echo "$override" + return + fi + + # Try to match against known model families + for pattern in "${!REASONING_PARSERS[@]}"; do + if [[ "$model_id" == *"$pattern"* ]]; then + echo "${REASONING_PARSERS[$pattern]}" + return + fi + done + + # Default: no reasoning parser + echo "none" +} + +################################################################################ +# Workspace Preparation +################################################################################ + +prepare_model_workspace() { + local cache_dir="$1" + + # Create cache directory if it doesn't exist + if [[ ! -d "$cache_dir" ]]; then + echo "Creating model cache directory: $cache_dir" + mkdir -p "$cache_dir" + fi + + # OVMS will handle model download and graph generation via --source_model + echo "Model cache ready: $cache_dir" +} + +################################################################################ +# Runtime Configuration Export +################################################################################ + +build_ovms_gpu_config() { + local gpu_device="$1" + local render_gid="$2" + + if [[ -z "$gpu_device" ]]; then + echo "" + return + fi + + if [[ -n "$render_gid" ]]; then + cat <}" + echo " OVMS_REST_PORT: $OVMS_REST_PORT" + echo " OVMS_GRPC_PORT: $OVMS_GRPC_PORT" + echo " OPENHANDS_PORT: $OPENHANDS_PORT" +} + +################################################################################ +# Compose File Generation and Management +################################################################################ + +# Metadata file management +# .ovms-deployment stores the complete deployment fingerprint for comparison + +write_deployment_metadata() { + local metadata_file="$1" + local timestamp="$2" + + cat > "$metadata_file" << EOF +METADATA_VERSION=1 +MODEL_ID=${MODEL_ID} +TARGET_DEVICE=${TARGET_DEVICE} +LOCAL_NAME=${LOCAL_NAME} +REASONING_PARSER=${REASONING_PARSER} +OVMS_IMAGE=${OVMS_IMAGE} +GPU_DEVICE=${GPU_DEVICE} +MODEL_CACHE_DIR=${MODEL_CACHE_DIR} +HOST_UID=${HOST_UID} +HOST_GID=${HOST_GID} +RENDER_GID=${RENDER_GID} +OVMS_REST_PORT=${OVMS_REST_PORT} +OVMS_GRPC_PORT=${OVMS_GRPC_PORT} +OPENHANDS_PORT=${OPENHANDS_PORT} +http_proxy=${http_proxy:-} +https_proxy=${https_proxy:-} +HTTP_PROXY=${HTTP_PROXY:-} +HTTPS_PROXY=${HTTPS_PROXY:-} +no_proxy=${no_proxy:-} +NO_PROXY=${NO_PROXY:-} +GENERATION_TIMESTAMP=${timestamp} +EOF +} + +# Compare current deployment fingerprint with stored metadata +# Returns 0 if all deployment parameters match, 1 if any differ +# Sets METADATA_OUTDATED to 1 if metadata version is incompatible +compare_deployment_fingerprint() { + local metadata_file="$1" + + if [[ ! -f "$metadata_file" ]]; then + return 1 + fi + + # Export current runtime values with CURRENT_ prefix for comparison + # Source metadata in a subshell to avoid overwriting global variables + # The subshell outputs a status code and outdated flag + local result + result=$( + # Current runtime values (from caller's scope) + export CURRENT_MODEL_ID="$MODEL_ID" + export CURRENT_TARGET_DEVICE="$TARGET_DEVICE" + export CURRENT_LOCAL_NAME="$LOCAL_NAME" + export CURRENT_REASONING_PARSER="$REASONING_PARSER" + export CURRENT_OVMS_IMAGE="$OVMS_IMAGE" + export CURRENT_GPU_DEVICE="$GPU_DEVICE" + export CURRENT_MODEL_CACHE_DIR="$MODEL_CACHE_DIR" + export CURRENT_HOST_UID="$HOST_UID" + export CURRENT_HOST_GID="$HOST_GID" + export CURRENT_RENDER_GID="$RENDER_GID" + export CURRENT_OVMS_REST_PORT="$OVMS_REST_PORT" + export CURRENT_OVMS_GRPC_PORT="$OVMS_GRPC_PORT" + export CURRENT_OPENHANDS_PORT="$OPENHANDS_PORT" + export CURRENT_http_proxy="${http_proxy:-}" + export CURRENT_https_proxy="${https_proxy:-}" + export CURRENT_HTTP_PROXY="${HTTP_PROXY:-}" + export CURRENT_HTTPS_PROXY="${HTTPS_PROXY:-}" + export CURRENT_no_proxy="${no_proxy:-}" + export CURRENT_NO_PROXY="${NO_PROXY:-}" + + # Source metadata file (overwrites variables with stored values) + source "$metadata_file" + + # Check metadata version compatibility + if [[ "${METADATA_VERSION:-}" != "1" ]]; then + echo "outdated" + exit 1 + fi + + # Compare stored values (from source) against current values (CURRENT_ prefix) + [[ "$MODEL_ID" == "$CURRENT_MODEL_ID" ]] || exit 1 + [[ "$TARGET_DEVICE" == "$CURRENT_TARGET_DEVICE" ]] || exit 1 + [[ "$LOCAL_NAME" == "$CURRENT_LOCAL_NAME" ]] || exit 1 + [[ "$REASONING_PARSER" == "$CURRENT_REASONING_PARSER" ]] || exit 1 + [[ "$OVMS_IMAGE" == "$CURRENT_OVMS_IMAGE" ]] || exit 1 + [[ "$GPU_DEVICE" == "$CURRENT_GPU_DEVICE" ]] || exit 1 + [[ "$MODEL_CACHE_DIR" == "$CURRENT_MODEL_CACHE_DIR" ]] || exit 1 + [[ "$HOST_UID" == "$CURRENT_HOST_UID" ]] || exit 1 + [[ "$HOST_GID" == "$CURRENT_HOST_GID" ]] || exit 1 + [[ "$RENDER_GID" == "$CURRENT_RENDER_GID" ]] || exit 1 + [[ "$OVMS_REST_PORT" == "$CURRENT_OVMS_REST_PORT" ]] || exit 1 + [[ "$OVMS_GRPC_PORT" == "$CURRENT_OVMS_GRPC_PORT" ]] || exit 1 + [[ "$OPENHANDS_PORT" == "$CURRENT_OPENHANDS_PORT" ]] || exit 1 + [[ "${http_proxy:-}" == "$CURRENT_http_proxy" ]] || exit 1 + [[ "${https_proxy:-}" == "$CURRENT_https_proxy" ]] || exit 1 + [[ "${HTTP_PROXY:-}" == "$CURRENT_HTTP_PROXY" ]] || exit 1 + [[ "${HTTPS_PROXY:-}" == "$CURRENT_HTTPS_PROXY" ]] || exit 1 + [[ "${no_proxy:-}" == "$CURRENT_no_proxy" ]] || exit 1 + [[ "${NO_PROXY:-}" == "$CURRENT_NO_PROXY" ]] || exit 1 + + echo "ok" + exit 0 + ) + + # Check if metadata is outdated + if [[ "$result" == "outdated" ]]; then + METADATA_OUTDATED=1 + return 1 + fi + + # Check if all comparisons succeeded (result is "ok") + if [[ "$result" != "ok" ]]; then + return 1 + fi + + return 0 +} + +# Generate docker-compose.yml from the template +# Uses explicit variable allowlist to avoid unintended substitutions +generate_compose_from_template() { + local template_file="$1" + local output_file="$2" + + echo "Generating docker-compose.yml from template..." + + # envsubst requires space-separated variable names, not comma-separated + # Substitute ALL deployment-managed variables to generate a complete, self-contained + # runtime compose file. The generated docker-compose.yml contains concrete values, + # not placeholders. + envsubst '$MODEL_ID $LOCAL_NAME $TARGET_DEVICE $REASONING_PARSER $MODEL_CACHE_DIR $HF_TOKEN $OVMS_IMAGE $GPU_DEVICE $WSL_LIBS $HOST_UID $HOST_GID $RENDER_GID $OVMS_GPU_CONFIG $OVMS_REST_PORT $OVMS_GRPC_PORT $OPENHANDS_PORT $http_proxy $https_proxy $HTTP_PROXY $HTTPS_PROXY $no_proxy $NO_PROXY' < "$template_file" > "$output_file" + echo " Generated: $output_file" +} + +################################################################################ +# Docker Compose Deployment +################################################################################ + +deploy_ovms() { + local compose_file="$1" + local template_file="$COMPOSE_TEMPLATE" + local metadata_file="$DEPLOYMENT_METADATA" + local generation_timestamp + generation_timestamp=$(date -u +"%Y-%m-%dT%H:%M:%SZ") + + echo "Deploying OVMS and OpenHands via Docker Compose..." + + # Determine deployment action based on existing state + if [[ -f "$compose_file" ]]; then + # Compose file exists - compare deployment fingerprint + METADATA_OUTDATED=0 + if compare_deployment_fingerprint "$metadata_file"; then + # Identical deployment fingerprint - preserve existing compose and user edits + echo "Deployment fingerprint matches - reusing existing docker-compose.yml" + echo " (User modifications preserved)" + # Update timestamp in metadata + sed -i.bak "s/^GENERATION_TIMESTAMP=.*/GENERATION_TIMESTAMP=${generation_timestamp}/" "$metadata_file" + rm -f "${metadata_file}.bak" + else + # Deployment fingerprint differs - regenerate compose + if [[ "$METADATA_OUTDATED" -eq 1 ]]; then + echo "Deployment metadata is outdated or incompatible. Regenerating deployment configuration..." + else + echo "Deployment configuration changed - regenerating compose..." + fi + docker compose -f "$compose_file" down 2>/dev/null || true + rm -f "$compose_file" + generate_compose_from_template "$template_file" "$compose_file" + write_deployment_metadata "$metadata_file" "$generation_timestamp" + fi + else + # No existing compose - fresh deployment + echo "No existing docker-compose.yml found." + echo "Generating from: $template_file" + generate_compose_from_template "$template_file" "$compose_file" + write_deployment_metadata "$metadata_file" "$generation_timestamp" + fi + + # Deploy via docker compose + docker compose -f "$compose_file" up -d +} + +################################################################################ +# Health Check Polling +################################################################################ + +wait_for_health() { + local max_retries=60 # 60 * 5s = 5 minutes total polling + local retry_interval=30 + + echo "Waiting for OVMS LLM graph to initialize" + + # Initial sleep to let container start + sleep 10 + + for i in $(seq 1 $max_retries); do + local status + status=$(curl -sf "http://localhost:${OVMS_REST_PORT}/v1/config" 2>/dev/null || true) + + if echo "$status" | grep -q '"AVAILABLE"'; then + echo "✓ OVMS is ready. Model status: AVAILABLE" + return 0 + fi + + echo " Attempt $i/$max_retries: model not available yet..." + sleep "$retry_interval" + done + + echo "ERROR: OVMS failed to become ready within expected time." >&2 + echo "Check container logs: docker logs $OVMS_CONTAINER_NAME" >&2 + return 1 +} + +################################################################################ +# Diagnostics and Manual Equivalent +################################################################################ + +print_manual_equivalent() { + echo "" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "Manual Equivalent (README documents this workflow)" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + echo "The same deployment can be achieved manually with:" + echo "" + echo " # Set environment variables" + echo " export MODEL_ID=\"$MODEL_ID\"" + echo " export LOCAL_NAME=\"$LOCAL_NAME\"" + echo " export TARGET_DEVICE=\"$TARGET_DEVICE\"" + echo " export REASONING_PARSER=\"$REASONING_PARSER\"" + echo " export MODEL_CACHE_DIR=\"$MODEL_CACHE_DIR" + echo " export HF_TOKEN=\"\${HF_TOKEN:-}\"" + echo "" + echo " # Deploy via Docker Compose" + echo " docker compose -f $COMPOSE_FILE up -d" + echo "" + echo " # Wait for OVMS to become ready" + echo " curl -sf http://localhost:\${OVMS_REST_PORT:-8000}/v1/config | grep AVAILABLE" + echo "" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" +} + +cleanup_on_error() { + local exit_code="$1" + + if [[ $exit_code -ne 0 ]]; then + echo "" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "Deployment failed. For troubleshooting, see:" + echo " - Container logs: docker logs $OVMS_CONTAINER_NAME" + echo " - README.md troubleshooting section" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "" + fi +} + +################################################################################ +# Main Orchestration +################################################################################ + +main() { + parse_args "$@" + validate_prerequisites + validate_device "$TARGET_DEVICE" + + # Normalize model name if not overridden + LOCAL_NAME="${LOCAL_NAME:-$(normalize_model_name "$MODEL_ID")}" + + # Resolve reasoning parser if not overridden + REASONING_PARSER="$(resolve_reasoning_parser "$MODEL_ID" "$REASONING_PARSER")" + + # Prepare workspace + prepare_model_workspace "$MODEL_CACHE_DIR" + + # Export runtime configuration + export_runtime_configuration + + # Deploy + deploy_ovms "$COMPOSE_FILE" + + # Health check (unless skipped) + if [[ "$SKIP_WAIT" == "false" ]]; then + if ! wait_for_health; then + cleanup_on_error 1 + exit 1 + fi + else + echo "Skipping health check (--skip-wait specified)" + fi + + # Print manual equivalent + print_manual_equivalent + + # Success summary + echo "✓ Deployment complete!" + echo "" + echo "Services running:" + echo " - OVMS: http://localhost:${OVMS_REST_PORT}/v3" + echo " - OpenHands: http://localhost:${OPENHANDS_PORT}" + echo "" + echo "Next steps (from README.md):" + echo " 1. Verify OVMS: curl http://localhost:${OVMS_REST_PORT}/v3/models" + echo " 2. Open OpenHands: http://localhost:${OPENHANDS_PORT}" + echo " 3. Create an agent task to test the integration" + echo "" +} + +# Execute main function with all arguments +main "$@"