Datagen inputs are too big for model context length

[changminbark]

What happened: The ShareGPT dataset seems to have 2 inputs that are longer the max input size for Qwen2.5 0.5B. This results in 2 failures when running the following config.

What you expected to happen: I am not sure what is the intended behavior. Is it supposed to just fail or should we truncate the datagen input to fit within the model's context length

How to reproduce it (as minimally and precisely as possible): I used Qwen/Qwen2.5-0.5B-Instruct along with the shareGPT dataset. The following is the config.yml:

load:
  type: constant
  stages:
  - rate: 1
    duration: 30
api: 
  type: completion
  streaming: true
server:
  type: vllm
  model_name: Qwen/Qwen2.5-0.5B-Instruct
  base_url: http://0.0.0.0:8000
  ignore_eos: true
tokenizer:
  pretrained_model_name_or_path: Qwen/Qwen2.5-0.5B-Instruct
data:
  type: shareGPT
metrics:
  type: prometheus
  prometheus:
    url: http://localhost:9090
    scrape_interval: 15
report:
  request_lifecycle:
    summary: true
    per_stage: true
    per_request: false
    per_adapter: true
    per_adapter_stage: true
  prometheus:
    summary: true
    per_stage: false

Anything else we need to know?:

Environment:

• inference-perf version: Latest (v0.3.0)
• config.yml (entire one printed by the benchmark run): See above
• cloud provider or hardware configuration: local
• others:

[jjk-g]

#290 may be related to behavior observed.

The model selected may fail to report the max length in a way that our tokenizer initialization can utilize it.

[jjk-g]

@changminbark

For ShareGPT datagen, if you specify the input_distribution.max you will not accept ShareGPT prompts that are beyond that max.

It seems that the issue here is that the model server you are testing against has a default or set max-model-len that is less than some of the prompt lengths in ShareGPT and is causing the error.

[changminbark]

@jjk-g where can we specify the input_distribution.max value?

I was using vLLM as my model server. It seems that it automatically detected the max model length via the model config.

[jjk-g]

@changminbark input_distribution.max is configured on inference-perf to cap the max length when using shareGPT. This is how it is usually configured to stay under the max-model-len configured.

[changminbark]

@jjk-g So, does that mean the only way to avoid this behavior would be to find the max-model-len in vLLM by running inference-perf or feeding a long enough input to find the model server's max content length and then set the config's input_distribution.max to that value?

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale