Qwen3-vl and Qwen2.5-vl Video Mode

[ericcurtin]

ollama/ollama#12962

[ashish159357]

hey @ericcurtin , can i work on this ?

Thanks

[ericcurtin]

Sure first step would be to try and push these somewhere on Docker Hub, test, etc.

[ashish159357]

@ericcurtin Both models are already available on Docker — Qwen2.5-VL and Qwen3-VL.

Should I proceed with testing them using the model runner only, or is there anything else I need to do?

[ericcurtin]

Sure... Can we test if things like:

curl http://localhost:12434/v1/chat/completions -d '{
  "model": "ai/qwen3-vl",
  "messages": [{
    "role": "user",
    "content": [
      {"type": "text", "text": "What is in this video?"},
      {"type": "video_url", "video_url": {"url": "https://example.com/video.mp4"}}
    ]
  }]
}'

work?

[ashish159357]

@ericcurtin I have tested :

curl http://localhost:12434/v1/chat/completions -d '{
"model": "ai/qwen3-vl",
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": "What is in this video?"},
{"type": "video_url", "video_url": {"url": "https://example.com/video.mp4"}}
]
}]
}'

but it not working, give below response :

{
"error": {
"code": 500,
"message": "unsupported content[].type",
"type": "server_error"
}
}

[ericcurtin]

We might need a llama.cpp bump, but unsure, can you debug?

[ashish159357]

@ericcurtin in code of llamacpp they have provided only support for image_url, input_audio and text.

as shown in below SS, 689 line will throw error "unsupported content[].type" if i use video_url.

[ashish159357]

Hii @ericcurtin ,

I think we have to do something like below :

1. Video Processing Logic (llamacpp/native/src/server/utils.hpp)

• Adding extract_video_frames() function that uses ffmpeg to extract 8 frames from any video
• Adding video_url content type handler supporting both HTTP URLs and base64-encoded videos
• Each frame is processed as an image through the existing multimodal pipeline

2. Dependency (scripts/apt-install.sh)

• Added ffmpeg to Docker image installation

How It Works

• User sends video_url in API request
• Server downloads/decodes the video
• ffmpeg extracts 8 evenly-distributed frames
• Frames are processed as images through the mtmd pipeline
• Model generates response based on video content

Yet, its not tested

[doringeman]

Hey @ashish159357, are you willing to work on this? 😀

[ashish159357]

@doringeman yes