MOSS-Image FP16-Fixed for Older GPUs

Introduction

If you tried to run MOSS-Image in plain fp16 on an older card and felt one bad prompt away from NaNs, that was not paranoia. KBlueLeaf (

View Post on X

) just published MOSS-Image-FP16-fixed on Hugging Face — Apache-2.0 remaps of the official Base and SFT transformers that keep the residual stream in a comfortable fp16 range, with ComfyUI single-file weights ready today.

This is not another fine-tune. It is a precision fold: same outputs as the official model up to rounding, closer to an fp32 reference than stock bf16 in KBlueLeaf's measurements, and aimed at cards that only do reliable fp16 (think V100 / 2080 Ti).

We already covered the MOSS-Image open-weights drop, native ComfyUI support, and today's Kromoss fine-tune. This one is for local runners who want the official recipe without living on a 1.05× fp16 cliff.

What Shipped

Repo: KBlueLeaf/MOSS-Image-FP16-fixed (created October 19, 2026, about 09:40 UTC)
Vendor / author: KBlueLeaf (KohakuLab)
Modality: text-to-image transformers for MOSS-Image Base and SFT
Open vs closed: open weights, Apache-2.0 (same as official MOSS-Image)
Where to run today: ComfyUI (--fp16-unet + Load Diffusion Model), diffusers, or KohakuMoss
Concrete fact: residual stream scaled by c = 1/16, giving about 16.9× headroom at the hottest block versus ~1.05× on official weights in fp16

Only the transformer changes. Keep the official Qwen3.5 text encoder, tokenizer, VAE, scheduler, and pipeline code.

Why FP16 Needed a Fix

MOSS-Image dumps every block's output into one residual stream. On the official weights, KBlueLeaf measured peak values around 62k in fp32 over a sweep of prompts, resolutions, and sigmas — and fp16 tops out at 65,504. The images can look fine… until a longer prompt, a LoRA, or a finetune pushes that stream up another 5% and you get inf / NaN.

The fix folds the stream as c · x with c = 2⁻⁴. Norm math is scale-invariant when you adjust eps correctly, so attention and FFN behavior stay the same while the stored stream sits lower. The card ships stock norm eps so stock ComfyUI / diffusers loaders work unmodified.

Numbers from the card

Against an fp32 reference on ComfyUI (1024², 50 steps, CFG 3):

Base fp16-fixed: LPIPS 0.0062 / PSNR 44.8 dB
Base official bf16: LPIPS 0.0501 / PSNR 29.3 dB
SFT fp16-fixed: LPIPS 0.0042 / PSNR 43.2 dB
SFT official bf16: LPIPS 0.0413 / PSNR 29.3 dB

No non-finite latents or images showed up in those runs. Treat the side-by-sides on the model card as the visual proof.

Files to Grab

PathUse

base/transformer/ or sft/transformer/

diffusers MOSSImageTransformer

comfyui/moss_image_base_fp16_fixed.safetensors

ComfyUI single file (Base)

comfyui/moss_image_sft_fp16_fixed.safetensors

ComfyUI single file (SFT)

reports/*_fp16_fix.json

Per-image measurement dumps

How to Run It in ComfyUI

Drop moss_image_base_fp16_fixed.safetensors (or the _sft_ file) into models/diffusion_models/.
Keep the usual Comfy-Org text encoder (qwen3.5_9b_bf16.safetensors) and flux2-vae_small.safetensors VAE.
Launch ComfyUI with --fp16-unet. Without that flag, ComfyUI's MOSS entry still prefers bf16/fp32 and will not give you the fp16 path you want.
Open the official MOSS-Image workflow, load the fixed file with Load Diffusion Model, and sample with euler / simple, 50 steps, CFG 3.

LoRAs and adapters

If your LoRA or PEFT adapter only touches Q / K / V / O and MLP layers, KBlueLeaf says you can apply it as-is — the fold leaves those weights alone. Adapters that touch video_embedder, text_embedder, or shared_block_modulation need their deltas scaled by the same c; KohakuMoss's export/merge helpers can do that when baking into these files.

Prefer Diffusers?

import importlib
import torch
from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
    "OpenMOSS-Team/MOSS-Image-Base",  # or MOSS-Image-SFT
    custom_pipeline="pipeline_moss_image",
    trust_remote_code=True,
    transformer=None,
    torch_dtype=torch.float16,
)
pipe.scheduler = type(pipe.scheduler).from_config(pipe.scheduler.config)
MOSSImageTransformer = importlib.import_module(type(pipe).__module__).MOSSImageTransformer
pipe.transformer = MOSSImageTransformer.from_pretrained(
    "KBlueLeaf/MOSS-Image-FP16-fixed",
    subfolder="base/transformer",  # or sft/transformer
    torch_dtype=torch.float16,
)
pipe.to("cuda")
image = pipe(
    "A photograph of a tabby cat sitting on a sunlit windowsill, shallow depth of field.",
    height=1024,
    width=1024,
    num_inference_steps=50,
    guidance_scale=3.0,
).images[0]

Original Source

Model card: huggingface.co/KBlueLeaf/MOSS-Image-FP16-fixed
Announcement: