Support add_generation_prompt request parameter for chat completions (#3877)

[exzile]

Summary

Closes #3877.

The chat template was always rendered with add_generation_prompt=true, hardcoded in every servable. This exposes an optional add_generation_prompt field (bool, default true) on /v3/chat/completions, matching HF transformers and vLLM. When false, the trailing generation prompt is omitted — the building block for assistant prefill.

Changes

• Parse and validate add_generation_prompt in the request handler; store it on the request struct.
• Honor it at every chat-template application site: the MINJA path (LLM and VLM continuous batching, legacy) and the Python-Jinja path (read from the request body and passed into the template render).
• Replaces the previously hardcoded add_generation_prompt = true.

Testing

Added tests covering default (generation prompt added) and false (generation prompt omitted, assistant message preserved).

Verified end-to-end on the MINJA path with HuggingFaceTB/SmolLM2-360M-Instruct:

• default → prompt ends with <|im_start|>assistant
• add_generation_prompt: false → that trailing generation prompt is omitted

Scope note

This implements add_generation_prompt only. Full assistant prefill — continuing from the final assistant message without closing it (continue_final_message in transformers/vLLM) — is a separate control that the genai C++ apply_chat_template does not currently expose, and is left as a follow-up.

🤖 Generated with Claude Code

[mzegla]

Looks like a rebase with conflict resolution is needed.

[mzegla]

Please have a look at those few comments. Once we have it resolved I think we will be good to run it through the CI and merge.

[exzile]

@mzegla tended to all those comments in the latest commit.
Removed the leftover top-level add_generation_prompt field/parsing (now lives solely in chat_template_kwargs).
Fixed a real bug: the MINJA path used get_bool() outside its try/catch, which would throw an unhandled ov::Exception on a non-boolean value instead of a clean error — now uses as_bool() and returns InvalidArgumentError.
Trimmed the leftover comment line in input_request.hpp per the inline suggestion.
Updated both REST docs pages to document add_generation_prompt under chat_template_kwargs instead of as a separate parameter.
Added two unit tests (false-case and invalid-type rejection).

[Copilot]

Pull request overview

Adds support for controlling whether the chat template appends the trailing “generation prompt” marker via an add_generation_prompt boolean (default true). This aligns OVMS chat-template rendering behavior with common serving stacks and enables assistant prefill by omitting the final assistant-start marker when requested.

Changes:

• Plumbs add_generation_prompt through both chat-template rendering paths: Python-Jinja (PyJinjaTemplateProcessor) and MINJA/native tokenizer (ChatTemplateProcessor), including type validation.
• Updates REST API docs to describe add_generation_prompt under chat_template_kwargs.
• Adds unit tests covering default behavior, false behavior, and non-boolean rejection.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/llm/py_jinja_template_processor.cpp	Extracts add_generation_prompt from chat_template_kwargs and passes it explicitly to Jinja render (avoids duplicate-kwarg issues).
src/llm/io_processing/input_processors/chat_template_processor.cpp	Extracts/validates add_generation_prompt from kwargs for the native tokenizer path and removes it from the kwargs map to avoid double-supplying.
src/llm/apis/openai_api_handler.cpp	Clarifies that add_generation_prompt is consumed via chat_template_kwargs on the copied ChatHistory.
src/llm/io_processing/input_request.hpp	Notes that add_generation_prompt is carried via ChatHistory extra context.
src/test/llm/llmtemplate_test.cpp	Adds Python-Jinja path tests for default (true) and explicit false.
src/test/llm/input_processing/chat_template_processor_test.cpp	Adds native tokenizer path tests for false and non-boolean invalid argument handling.
docs/model_server_rest_api_chat.md	Documents add_generation_prompt under chat_template_kwargs for /v3/chat/completions.
docs/model_server_rest_api_responses.md	Documents add_generation_prompt under chat_template_kwargs for /v3/responses.

Comments suppressed due to low confidence (1)

src/llm/io_processing/input_processors/chat_template_processor.cpp:102

• kwargs is copied into optKwargs via std::make_optional(kwargs). Since kwargs is not used afterwards, this introduces an avoidable JsonContainer copy on the request hot path (potentially expensive for large chat_template_kwargs). Move kwargs into the optional instead.

        const std::optional<ov::genai::JsonContainer> optKwargs =
            kwargs.empty() ? std::nullopt : std::make_optional(kwargs);

[dkalinowski]

@exzile Do you know about any existing app that uses this feature (add_generation_prompt set to false)? If so, it would be worth aligning to the API - it could point us if this param should be in chat_template_extra_kwargs field or level above.

[exzile]

Checked the two serving frameworks that actually implement this (checked their current source, not just docs):

vLLM (vllm/entrypoints/openai/chat_completion/protocol.py) declares both as top-level ChatCompletionRequest fields, siblings to chat_template_kwargs, not nested inside it:

add_generation_prompt: bool = Field(default=True, ...)
continue_final_message: bool = Field(default=False, ...)  # the actual prefill toggle; mutually exclusive with add_generation_prompt
...
chat_template_kwargs: dict[str, Any] | None = Field(default=None, ...)

It only merges them into the kwargs dict internally, right before calling the Jinja renderer — that's an implementation detail of how vLLM talks to the template, not the client-facing contract:

chat_template_kwargs=merge_kwargs(
    self.chat_template_kwargs,
    dict(add_generation_prompt=self.add_generation_prompt, continue_final_message=self.continue_final_message, ...),
)

SGLang (sglang/srt/entrypoints/openai/protocol.py) follows the same shape: continue_final_message: bool = False is a top-level field, separate from chat_template_kwargs: Optional[Dict] = None. SGLang doesn't expose add_generation_prompt at all — continue_final_message is its only prefill control.

llama.cpp server doesn't expose add_generation_prompt (or an equivalent) as a chat-completions request parameter at all — only the generic chat_template_kwargs passthrough exists there, so no data point either way.

So the two real implementations of this feature both keep it as a top-level sibling to chat_template_kwargs, and only fold it in internally at render time. That's actually what this PR had before this round moved it under chat_template_kwargs — worth reconsidering going back to a top-level field to match the existing convention, if that's the direction you'd like. Happy to make that change if so.