Feature/assistant prefill - exzile contribution#4364
Closed
mzegla wants to merge 10 commits into
Closed
Conversation
The chat template was always rendered with add_generation_prompt=true, hardcoded in every servable. This exposes an optional add_generation_prompt field (bool, default true) on the /v3/chat/completions request, matching HF transformers and vLLM. When false, the trailing generation prompt is omitted, which is the building block for assistant prefill. - Parse add_generation_prompt in the request (openai_api_handler.cpp) and store it on the request struct (openai_request.hpp). - Honor it in all chat-template application sites: the MINJA path (LLM and VLM continuous batching, legacy) and the Python-Jinja path (read from the request body and pass into the template render). - Add tests covering default (generation prompt added) and false (generation prompt omitted, assistant message preserved). Verified end-to-end on the MINJA path with HuggingFaceTB/SmolLM2-360M-Instruct: default renders a trailing "<|im_start|>assistant", add_generation_prompt=false omits it. Note: true assistant prefill (continue_final_message - continuing from the final assistant message without closing it) is a separate control and is left as a follow-up. Implements #3877 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per review: add_generation_prompt is valid and functional for the Responses API - the Endpoint::RESPONSES path in servable.cpp reads request.addGenerationPrompt and passes it to apply_chat_template (and the Jinja path reads it from the request JSON), same as chat completions. Document it in the responses REST API parameters table. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per review feedback: add_generation_prompt is only ever consumed by chat template rendering, so route it through chat_template_kwargs instead of a dedicated InputRequest field. The MINJA path extracts it back out for genai's dedicated apply_chat_template argument; the Python-Jinja path pops it from the kwargs dict before splatting into render() to avoid a duplicate-keyword collision. Also fixes the Python-Jinja path never having wired add_generation_prompt through at all, and two llmtemplate_test.cpp tests that referenced a stale 4-arg applyChatTemplate signature and never compiled. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ature/assistant-prefill
…validate kwargs type - Remove the now-redundant top-level add_generation_prompt request field/parsing and the OpenAIRequest.addGenerationPrompt member: the parameter lives entirely in chat_template_kwargs now, and both template paths already default it to true when absent. - Fix MINJA path to use JsonContainer::as_bool() instead of get_bool(), which threw an unhandled ov::Exception (outside the surrounding try/catch) when add_generation_prompt was present but not a boolean; now returns a clean InvalidArgumentError, matching the Python-Jinja path's existing validation. - Trim a leftover comment line in input_request.hpp per review suggestion. - Update chat/responses REST docs to document add_generation_prompt as part of chat_template_kwargs instead of a separate top-level parameter. - Add unit tests for add_generation_prompt=false and the invalid-type rejection on the MINJA path.
…ature/assistant-prefill
…tion_prompt parsing - Remove the comments in openai_api_handler.cpp and input_request.hpp that called out add_generation_prompt specifically among chat_template_kwargs entries (mzegla) -- the field-level docs already cover it. - Extract the add_generation_prompt lookup/validation in ChatTemplateProcessor::process into its own extractAddGenerationPrompt method (dkalinowski).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Creating this PR as a WA to trigger Ci for #4331