Release full TextEncoder

[vladmandic]

Right now Lens ships with LensGptOssEncoder at https://huggingface.co/microsoft/Lens/tree/main/text_encoder which is pre-quantized using mxfp4 method - which is great for Hopper architecture, but not-so-great for any other GPU due to lack of native FP4 support.

Ask is to release text_encoder in non-quantized BFloat16 so it can be quantized as needed by different apps.

For example, SD.Next prefers its own SDNQ method which among others, has efficient kernels for UINT dtypes and is fully cross-platform compatible (nVidia, AMD, IPEX, etc.)

[youchenghuanxian]

Thanks for the feedback!

The CLI already provides a --disable_mxfp4 flag that dequantizes the text encoder to bf16/fp16 at load time, which works on A100/V100 and other non-Hopper GPUs:

python inference.py --disable_mxfp4 --offload --prompt "a cat"
If you need a standalone bf16 checkpoint for custom quantization (e.g. SDNQ), you can export it with:

from lens import LensGptOssEncoder
from transformers import Mxfp4Config
import torch

encoder = LensGptOssEncoder.from_pretrained(
"microsoft/Lens",
subfolder="text_encoder",
quantization_config=Mxfp4Config(dequantize=True),
dtype=torch.bfloat16,
)
encoder.save_pretrained("./text_encoder_bf16")

This gives you full bf16 weights that you can then re-quantize with your preferred method.

[czkoko]

@youchenghuanxian
Has the text encoder been fine-tuned? I am currently implementing Lens using MLX. Since the expert modules require special handling, I opted for the pre-converted mlx-community/gpt-oss-20b-MXFP4-Q4 model, and the generated images appear to look normal.

However, I noticed that the gpt-oss files used by Lens seem to differ from those in openai/gpt-oss-20b. Do I need to perform the conversion again?

[vladmandic]

@youchenghuanxian
thanks, that works (although i'd assume there is a slight loss going from bf16->mxfp4->bf16->sdnq)

not related and just fyi, slightly modified lens pipeline: https://github.com/vladmandic/sdnext/tree/dev/pipelines/lens

• patched transformers to avoid some dtype mismatch issues
• added img2img and inpaint pipelines, no reason why lens would be txt2img only

anyhow, thanks for the quick reply, feel free to close this issue afaic
(i'm leaving it open for the moment due to second question from @czkoko)

[youchenghuanxian]

@youchenghuanxian Has the text encoder been fine-tuned? I am currently implementing Lens using MLX. Since the expert modules require special handling, I opted for the pre-converted mlx-community/gpt-oss-20b-MXFP4-Q4 model, and the generated images appear to look normal.

However, I noticed that the gpt-oss files used by Lens seem to differ from those in openai/gpt-oss-20b. Do I need to perform the conversion again?

Thanks for trying Lens with MLX!

No, the text encoder has not been fine-tuned. We use the original OpenAI GPT-OSS-20B model weights without any modification. The files in our repo are simply repacked for compatibility with our pipeline's loading logic (e.g., config/naming conventions expected by transformers + diffusers), but the underlying weights are identical to openai/gpt-oss-20b.

Regarding mlx-community/gpt-oss-20b-MXFP4-Q4, if your generated images look normal, it should be fine. We haven't verified third-party quantized variants ourselves, but as long as the text embeddings are reasonably close to the original, it should work. The key thing is to make sure you're extracting the correct multi-layer features (see selected_layer_index in our transformer config) rather than just the final hidden state.

[czkoko]

@youchenghuanxian Thank you for your answer. I have submitted a new issue.