[DO NOT MERGE] Refactor/aiter integration by vllmellm · Pull Request #76 · EmbeddedLLM/vllm

vllmellm · 2025-11-17T08:52:41Z

Purpose

Test Plan

Test Result

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

github-actions · 2025-11-17T08:52:52Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

vllmellm · 2025-11-17T09:01:29Z

...model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_fp8.py

@@ -189,6 +189,8 @@ def process_weights_after_loading(self, layer) -> None:
        if self.strategy == QuantizationStrategy.BLOCK:
            maybe_post_process_fp8_weight_block(layer)



Check if fp8_linear is initialised.

vllmellm · 2025-11-17T09:01:34Z

vllm/model_executor/layers/quantization/kernels/scaled_mm/aiter.py

+        N = w_q.shape[1]
+        K = w_q.shape[0]
+
+        if N % 16 == 0 and K % 16 == 0:


Add https://github.com/ROCm/vllm/blob/c88d6d2ec7299605bb2ed8a4aee9260d90ef0631/vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_fp8.py#L153 to the rocm_aiter_ops and use that to replace this if conditions.

…9189) Signed-off-by: nandan2003 <nandan.vallamdasu@outlook.com> Signed-off-by: Nandan Vallamdasu <nandan.vallamdasu@outlook.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

…llm-project#28832) Signed-off-by: Bram Wasti <bwasti@meta.com> Signed-off-by: Bram Wasti <bwasti@fb.com> Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>

vllm-project#29084) Signed-off-by: NickLucche <nlucches@redhat.com>

…llm-project#29216) Signed-off-by: Nick Hill <nhill@redhat.com>

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

…oject#29232) Signed-off-by: zitian.zhao <zitian.zhao@tencentmusic.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

…h NEON (vllm-project#29193) Signed-off-by: Fadi Arafeh <fadi.arafeh@arm.com>

…vllm-project#29239) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

…t#26966) Signed-off-by: bbartels <benjamin@bartels.dev>

Signed-off-by: Yizhou Liu <liu_yizhou@outlook.com>

Signed-off-by: yewentao256 <zhyanwentao@126.com>

) Signed-off-by: Qidong Su <soodoshll@gmail.com>

Signed-off-by: jiahanc <173873397+jiahanc@users.noreply.github.com> Signed-off-by: mgoin <mgoin64@gmail.com> Co-authored-by: mgoin <mgoin64@gmail.com>

…llm-project#29173) Signed-off-by: Michael Act <michael.a.c.tulenan@gdplabs.id> Co-authored-by: Michael Goin <mgoin64@gmail.com>

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

…in test_multi_connector.py due to hipErrorLaunchFailure when calling .cpu() (vllm-project#29253) Signed-off-by: Randall Smith <ransmith@amd.com> Co-authored-by: Randall Smith <ransmith@amd.com>

…ntion.py (vllm-project#29252) Signed-off-by: Randall Smith <ransmith@amd.com> Co-authored-by: Randall Smith <ransmith@amd.com>

…st_pynccl.py (vllm-project#29119) Signed-off-by: Micah Williamson <micah.williamson@amd.com>

…istry (vllm-project#28958) Signed-off-by: Luke <yq0536@gmail.com> Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>

…coding (vllm-project#29194) Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

…-project#29276) Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

… message history format (vllm-project#29249) Signed-off-by: joshiemoore <joshiemoore98@gmail.com>

…ct#29724) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

…9727) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

…ct#24722) Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com> Signed-off-by: Michael Goin <mgoin64@gmail.com> Signed-off-by: Jinzhen Lin <linjinzhen@hotmail.com> Co-authored-by: Michael Goin <mgoin64@gmail.com> Co-authored-by: Michael Goin <mgoin@redhat.com>

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

) Signed-off-by: Xin Yang <xyangx@amazon.com> Signed-off-by: Xin Yang <105740670+xyang16@users.noreply.github.com> Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>

…ject#29732) Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

Signed-off-by: Huamin Li <3ericli@gmail.com>

…gibberish output (vllm-project#28783) Signed-off-by: vensen <vensenmu@gmail.com> Co-authored-by: TJian <tunjian.tan@embeddedllm.com>

Signed-off-by: BowTen <bowten@qq.com>

…r` (vllm-project#29730) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

…project#29741) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

…end` (vllm-project#29234) Signed-off-by: ganyi <ygan@amd.com>

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

…ject#29004)

…ect#29749) Signed-off-by: Xingyu Liu <charlotteliu12x@gmail.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

…#29756) Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

Signed-off-by: Shu Wang <shuw@nvidia.com> Signed-off-by: Shu Wang. <shuw@nvidia.com> Signed-off-by: Michael Goin <mgoin64@gmail.com> Co-authored-by: root <root@umbriel-b200-017.ipp4a1.colossus.nvidia.com> Co-authored-by: Michael Goin <mgoin64@gmail.com>

…ect#29568) Signed-off-by: Yifei Zhang <yifei.zhang1992@outlook.com>

Signed-off-by: Huamin Li <3ericli@gmail.com>

Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>

Signed-off-by: Daniel Salib <danielsalib@meta.com> Co-authored-by: Chauncey <chaunceyjiang@gmail.com>

…t#29750) Signed-off-by: Mickael Seznec <mickael@mistral.ai> Co-authored-by: Roger Wang <hey@rogerw.io>

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

…ect#29774) Signed-off-by: Fanli Lin <fanli.lin@intel.com>

… OOM (vllm-project#29504) Signed-off-by: zhxchen17 <zhxchen17@fb.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

…kMask building (vllm-project#26015) Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Signed-off-by: baonudesifeizhai <baonudesifeizhai@gmail.com> Co-authored-by: baonudesifeizhai <baonudesifeizhai@gmail.com>

…9414) Signed-off-by: Marcin Ostrowski <marcinx.ostrowski@intel.com>

Signed-off-by: Shengqi Chen <harry-chen@outlook.com>

Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>

vllmellm commented Nov 17, 2025

View reviewed changes

nandan2003 and others added 28 commits November 22, 2025 04:58

fix: clean up function never use in setup.py (vllm-project#29061)

2d4978a

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

[tiny] Remove unsupported TRITON_MLA backend from batch invariance (v…

5f7209a

…llm-project#28832) Signed-off-by: Bram Wasti <bwasti@meta.com> Signed-off-by: Bram Wasti <bwasti@fb.com> Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>

[Attention] Refactor FA block_size limitations to hybrid models only (

066209a

vllm-project#29084) Signed-off-by: NickLucche <nlucches@redhat.com>

[BugFix] Fix returned logprobs with spec decode + prefill chunking (v…

d44a63c

…llm-project#29216) Signed-off-by: Nick Hill <nhill@redhat.com>

[Misc] Fix pre-commit (vllm-project#29238)

ae66818

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Fix EVS crash when using video_embeds inputs in Qwen2.5-VL (vllm-pr…

d84d8f4

…oject#29232) Signed-off-by: zitian.zhao <zitian.zhao@tencentmusic.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

chore: add RTX_PRO_6000 GLM4.6-FP8 kernel tuning (vllm-project#29240)

f55c76c

[perf][cpu] Accelerate paged attention GEMMs (QK, PV) on Arm CPUs wit…

730bd35

…h NEON (vllm-project#29193) Signed-off-by: Fadi Arafeh <fadi.arafeh@arm.com>

[Bugfix] Use HF config fields as fallback when loading Mistral config (…

d1cf821

…vllm-project#29239) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

[CI/build] Removes source compilation from runtime image (vllm-projec…

eb5352a

…t#26966) Signed-off-by: bbartels <benjamin@bartels.dev>

[BugFix] Fix chunked prompt logprobs + preemption (vllm-project#29071)

7df331c

Refactor: Move CUDA graph dispatch logic earlier (vllm-project#27382)

df78aee

Signed-off-by: Yizhou Liu <liu_yizhou@outlook.com>

[Chore] Update batch invariant code owner (vllm-project#29246)

472fdee

Signed-off-by: yewentao256 <zhyanwentao@126.com>

Patch DeepEP when building docker image with CUDA 13 (vllm-project#29154

4587063

) Signed-off-by: Qidong Su <soodoshll@gmail.com>

[Fix] Add SM check to flashinfer MOE backend (vllm-project#29144)

5f96c00

Signed-off-by: jiahanc <173873397+jiahanc@users.noreply.github.com> Signed-off-by: mgoin <mgoin64@gmail.com> Co-authored-by: mgoin <mgoin64@gmail.com>

docs: fixes distributed executor backend config for multi-node vllm (v…

3ed767e

…llm-project#29173) Signed-off-by: Michael Act <michael.a.c.tulenan@gdplabs.id> Co-authored-by: Michael Goin <mgoin64@gmail.com>

[Doc] Update more docs with respect to V1 (vllm-project#29188)

389aa1b

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

[Model Runner V2] Minor fix for cudagraph_utils (vllm-project#29256)

20ee418

[CI/Build][AMD] Skip test_multi_shared_storage_connector_consistency …

71362ff

…in test_multi_connector.py due to hipErrorLaunchFailure when calling .cpu() (vllm-project#29253) Signed-off-by: Randall Smith <ransmith@amd.com> Co-authored-by: Randall Smith <ransmith@amd.com>

[CI/Build][AMD] Add check for flash_att_varlen_func to test_tree_atte…

3999442

…ntion.py (vllm-project#29252) Signed-off-by: Randall Smith <ransmith@amd.com> Co-authored-by: Randall Smith <ransmith@amd.com>

[ROCm][CI] Fix "Cannot re-initialize CUDA in forked subprocess" in te…

55c21c8

…st_pynccl.py (vllm-project#29119) Signed-off-by: Micah Williamson <micah.williamson@amd.com>

[Bugfix] Use lazy string reference for DeepseekV3Config in config reg…

6fb0215

…istry (vllm-project#28958) Signed-off-by: Luke <yq0536@gmail.com> Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>

[Model Runner V2] Change bookkeeping logic in preparation for spec de…

7f12c82

…coding (vllm-project#29194) Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

[Model Runner V2] Support spec decoding [1/N] (vllm-project#29274)

b004c00

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

[Model Runner V2] Optimize CUDA graph capture time (vllm-project#29275)

62d54ba

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

[Model Runner V2] Add apply_temperature option to gumbel_sample (vllm…

3e1ad40

…-project#29276) Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

[Bugfix] Update Gradio OpenAI Chatbot Webserver example to new Gradio…

c309bb5

… message history format (vllm-project#29249) Signed-off-by: joshiemoore <joshiemoore98@gmail.com>

DarkLight1337 and others added 30 commits November 29, 2025 06:25

[Chore] Enable passing tokenizer=None into MM processor (vllm-proje…

fe3398f

…ct#29724) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

[Chore] Move detokenizer_utils to vllm/tokenizers (vllm-project#2…

fa59fe4

…9727) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

[Bugfix] Revert test_tokenization.py (vllm-project#29729)

b9d0504

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

[LoRA] Support FusedMoE LoRA Triton kernel for mxfp4 (vllm-project#29708

a491b09

) Signed-off-by: Xin Yang <xyangx@amazon.com> Signed-off-by: Xin Yang <105740670+xyang16@users.noreply.github.com> Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>

[Quantization] Enable compressed-tensors AWQ for Turing GPU (vllm-pro…

e1464c3

…ject#29732) Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

Fix AttributeError about _use_fi_prefill (vllm-project#29734)

82c795d

Signed-off-by: Huamin Li <3ericli@gmail.com>

[Bugfix][sleepmode][fp8 kv cache]: Fix FP8 KV cache + sleep(level=2) …

66b5840

…gibberish output (vllm-project#28783) Signed-off-by: vensen <vensenmu@gmail.com> Co-authored-by: TJian <tunjian.tan@embeddedllm.com>

[Doc]: Fix typo in fused_moe layer (vllm-project#29731)

9381b5c

Signed-off-by: BowTen <bowten@qq.com>

[Misc] Update TokenizerLike interface and move `get_cached_tokenize…

2afcec4

…r` (vllm-project#29730) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

[Bugfix] Fix mismatched nvfp4 gemm output shape (vllm-project#29742)

47539cf

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

[Core] Enable inputs_embeds_size separate from hidden_size (vllm-…

64bc09b

…project#29741) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

[ROCm][Attention] Sliding window support for `AiterFlashAttentionBack…

8c363ed

…end` (vllm-project#29234) Signed-off-by: ganyi <ygan@amd.com>

Fix RoPE failures in Transformers nightly (vllm-project#29700)

cd719de

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

[Feat] Support non-gated activations in NVFP4 modelopt path (vllm-pro…

39d2810

…ject#29004)

[Misc]Remove redundant hidden_size property in ModelConfig (vllm-proj…

21c2627

…ect#29749) Signed-off-by: Xingyu Liu <charlotteliu12x@gmail.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

[Model Runner V2] Use packed mask for prompt bin counts (vllm-project…

ec38a73

…#29756) Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

Make PyTorch profiler gzip and CUDA time dump configurable (vllm-proj…

1ab8fc8

…ect#29568) Signed-off-by: Yifei Zhang <yifei.zhang1992@outlook.com>

[CI] Skip paddleocr_vl for transformer 4.57.3 (vllm-project#29758)

83805a6

Signed-off-by: Huamin Li <3ericli@gmail.com>

[Frontend] Resettle pooling entrypoints (vllm-project#29634)

62de4f4

Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>

[Frontend] Add tool filtering support to ToolServer (vllm-project#29224)

014ece9

Signed-off-by: Daniel Salib <danielsalib@meta.com> Co-authored-by: Chauncey <chaunceyjiang@gmail.com>

[crashfix] Eagle + multimodal can crash on mm cache miss (vllm-projec…

86e178f

…t#29750) Signed-off-by: Mickael Seznec <mickael@mistral.ai> Co-authored-by: Roger Wang <hey@rogerw.io>

[Misc] Unify tokenizer registration (vllm-project#29767)

f0a28bf

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

[XPU] Fix AWQ skipped layer detection in IPEX quantization (vllm-proj…

f37e893

…ect#29774) Signed-off-by: Fanli Lin <fanli.lin@intel.com>

[multimodal][test] Reduce memory utilization for test_siglip to avoid…

ad9d656

… OOM (vllm-project#29504) Signed-off-by: zhxchen17 <zhxchen17@fb.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

[v1] Add real sliding window calculation to FlexAttention direct Bloc…

b95db24

…kMask building (vllm-project#26015) Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Signed-off-by: baonudesifeizhai <baonudesifeizhai@gmail.com> Co-authored-by: baonudesifeizhai <baonudesifeizhai@gmail.com>

[Bugfix] TypeError: 'NoneType' object is not callable (vllm-project#2…

5cfa967

…9414) Signed-off-by: Marcin Ostrowski <marcinx.ostrowski@intel.com>

[CI] Renovation of nightly wheel build & generation (vllm-project#29690)

36db0a3

Signed-off-by: Shengqi Chen <harry-chen@outlook.com>

sync upstream

30624ea

Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DO NOT MERGE] Refactor/aiter integration#76

[DO NOT MERGE] Refactor/aiter integration#76
vllmellm wants to merge 581 commits intorefactor-fp8-linearfrom
refactor/aiter_integration

vllmellm commented Nov 17, 2025 •

edited by github-actions bot

Loading

Uh oh!

github-actions bot commented Nov 17, 2025

Uh oh!

vllmellm Nov 17, 2025

Uh oh!

vllmellm Nov 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

		@@ -189,6 +189,8 @@ def process_weights_after_loading(self, layer) -> None:
		if self.strategy == QuantizationStrategy.BLOCK:
		maybe_post_process_fp8_weight_block(layer)

Conversation

vllmellm commented Nov 17, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

github-actions bot commented Nov 17, 2025

Uh oh!

vllmellm Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

vllmellm Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

vllmellm commented Nov 17, 2025 •

edited by github-actions bot

Loading