Feat: fp8 compute raw

docs/src/content/docs/configuration/fp8-storage.mdx

@@ -1,5 +1,5 @@
 ---
-title: FP8 Storage
+title: FP8 Storage & Compute
 sidebar:
   order: 3
 ---
@@ -29,9 +29,9 @@ InvokeAI's FP8 path stores weights in FP8 and casts them back to BF16/FP16 on ea
 
 The toggle works as advertised: the UNet / transformer drops by roughly 50% on the GPU. Per-step latency is the same or marginally slower because every forward pass adds an FP8 → BF16 cast on entry and a BF16 → FP8 cast on exit. This is the **largest target group**: 3090 owners squeezing FLUX into 24 GB benefit the most.
 
-### RTX 40-series, RTX 50-series, and Hopper — VRAM win today, compute win possible later
+### RTX 40-series, RTX 50-series, and Hopper — VRAM win, plus a compute win via a separate setting
 
-These GPUs have native FP8 tensor cores. The toggle still buys you the same ~50% VRAM reduction today, because the forward pass still runs in BF16 — the hook casts weights back up to compute precision before each layer. If InvokeAI later wires up a true FP8 matmul path (e.g. via `torchao`), the same toggle will *also* unlock compute speedups on this hardware. Until then, treat the benefit as "VRAM only, same as Ampere".
+These GPUs have native FP8 tensor cores. FP8 *Storage* on its own still only buys the ~50% VRAM reduction, because the forward pass runs in BF16 — the hook casts weights back up to compute precision before each layer. To actually use the tensor cores you need the separate [FP8 Compute](#fp8-compute) setting, which applies to checkpoints that ship pre-quantized ("scaled fp8") rather than to full-precision ones.
 
 ### Older CUDA cards — still a VRAM win
 
@@ -90,10 +90,40 @@ If you see unexpected quality regressions, disable FP8 Storage on the affected m
 
 ## Combining with Low-VRAM mode
 
-**FP8 + partial loading**: fully supported. FP8 Storage shrinks the layers; partial loading streams them between RAM and VRAM as needed. Use both on tight VRAM budgets.
+**FP8 Storage + partial loading**: fully supported. FP8 Storage shrinks the layers; partial loading streams them between RAM and VRAM as needed. Use both on tight VRAM budgets.
+
+**FP8 Compute + partial loading** is a different story — it still works, but it costs both speed and reproducibility. See [FP8 Compute](#fp8-compute) below.
 
 (For why FP8 Storage doesn't stack on top of GGUF / NF4 / int8 checkpoints, see the callout at the top of this page.)
 
+## FP8 Compute
+
+Everything above describes FP8 *Storage*, which changes how weights are **stored** while the math still runs in BF16. `fp8_compute` is a separate, global setting in `invokeai.yaml` that also does the **math** in FP8, on GPUs that have hardware for it. That makes generation faster, not just smaller.
+
+It is for models that were **already saved in FP8** by whoever published them — you'll often see these labelled "fp8" or "fp8_scaled" in the filename. Normally InvokeAI unpacks them back to BF16 while loading; with `fp8_compute` on, they stay as they are and run directly on the GPU's FP8 hardware. FP8 Storage is the toggle for the other case: a full-precision model you want to shrink yourself.
+
+Not every part of a model stays in FP8 — the small, precision-sensitive pieces are always unpacked. Those are a tiny share of the weights, so you still get nearly the full VRAM saving.
+
+You need an FP8-capable GPU: RTX 40-series or newer, the datacenter cards of those generations, or AMD MI300 and newer. InvokeAI checks your card the first time it needs to — by actually trying a small FP8 operation rather than going by the model name — so a card that can't do it quietly falls back to the normal path instead of failing partway through a generation.
+
+:::danger[Reproducibility requires the model to be fully in VRAM]
+With `fp8_compute` enabled, **the same seed only reproduces the same image if the model is 100% resident in VRAM.**
+
+FP8 math only happens for the parts of the model that are actually on the GPU. Anything still sitting in system RAM takes the normal path instead, which gives slightly different numbers — and *which* parts that affects depends on how much of the model happened to fit at that moment, which changes from run to run.
+
+Measured on a 24 GB card with a ~12 GB model at 88–95% loaded: two runs with an identical seed and identical settings differed in **98.7% of all pixels**. With the model fully loaded, repeated runs came out **identical**.
+
+To get repeatable output, check the model's load line in the log for `VRAM: … (100.0%)`. If it is below that, free up VRAM (lower resolution, fewer models loaded at once, a smaller text encoder) or set `enable_partial_loading: false`.
+:::
+
+Keeping the whole model on the GPU is worth it for speed as well: on the same 24 GB card, having to stream the last 5–12% of the model over PCIe cost **+47% per step** (1.03 → 1.51 s/it at 1024², 8 steps).
+
+### When the model asks for full precision
+
+Some FP8 models come with a note from whoever made them, marking certain layers as ones that should not use FP8 math. InvokeAI follows those notes by default, so those layers run the slower way. On a model that marks a lot of layers, this can eat much of the FP8 Compute speedup.
+
+Setting `fp8_compute_full_precision_hints: false` ignores the notes and runs everything on the FP8 hardware. It is faster, but you are overriding the model author's judgement about which layers are sensitive — so compare a few images before sticking with it.
+
 ## Troubleshooting
 
 ### "I toggled FP8 Storage but VRAM usage didn't change"

docs/src/generated/settings.json

@@ -457,6 +457,28 @@
       "type": "<class 'bool'>",
       "validation": {}
     },
+    {
+      "category": "CACHE",
+      "default": false,
+      "description": "Keep ComfyUI 'scaled fp8' checkpoints quantized instead of dequantizing them at load, and run their matmuls on the fp8 tensor cores (requires an Ada/SM 8.9 or newer NVIDIA GPU; falls back automatically otherwise). Roughly halves the transformer's VRAM and speeds up denoising, but quantizes activations as well, so images will differ from previous versions at the same seed. Reproducibility also requires the model to be FULLY resident in VRAM: a layer whose weights are still in RAM falls back to the dequantized path, and since which layers are resident shifts from run to run, the same seed then yields visibly different images. For repeatable output, ensure the model loads at 100% (e.g. enable_partial_loading=false with enough free VRAM).",
+      "env_var": "INVOKEAI_FP8_COMPUTE",
+      "literal_values": [],
+      "name": "fp8_compute",
+      "required": false,
+      "type": "<class 'bool'>",
+      "validation": {}
+    },
+    {
+      "category": "CACHE",
+      "default": true,
+      "description": "Honor the per-layer 'full_precision_matrix_mult' flags that some scaled-fp8 checkpoints ship. Those layers then dequantize on every forward instead of using the fp8 tensor cores, which can cost a large part of the fp8_compute speedup - on checkpoints that mark many layers, most of it. Set to false to run every quantized layer on the fp8 tensor cores, ignoring the producer's instruction; faster, but the marked layers were flagged as numerically sensitive, so quality may suffer. Only has an effect when fp8_compute is enabled.",
+      "env_var": "INVOKEAI_FP8_COMPUTE_FULL_PRECISION_HINTS",
+      "literal_values": [],
+      "name": "fp8_compute_full_precision_hints",
+      "required": false,
+      "type": "<class 'bool'>",
+      "validation": {}
+    },
     {
       "category": "CACHE",
       "default": null,

invokeai/app/invocations/qwen_image_image_to_latents.py

@@ -26,14 +26,18 @@
     title="Image to Latents - Qwen Image",
     tags=["image", "latents", "vae", "i2l", "qwen_image"],
     category="image",
-    version="1.0.0",
+    version="1.1.0",
     classification=Classification.Prototype,
 )
 class QwenImageImageToLatentsInvocation(BaseInvocation, WithMetadata, WithBoard):
     """Generates latents from an image using the Qwen Image VAE."""
 
     image: ImageField = InputField(description="The image to encode.")
     vae: VAEField = InputField(description=FieldDescriptions.vae, input=Input.Connection)
+    tiled: bool = InputField(default=False, description=FieldDescriptions.tiled)
+    # NOTE: tile_size = 0 is a special value meaning "use the model's default", matching the
+    # SD/SDXL i2l node. `int | None` is avoided because the workflow UI does not handle it well.
+    tile_size: int = InputField(default=0, multiple_of=8, description=FieldDescriptions.vae_tile_size)
     width: int | None = InputField(
         default=None,
         description="Resize the image to this width before encoding. If not set, encodes at the image's original size.",
@@ -44,21 +48,39 @@ class QwenImageImageToLatentsInvocation(BaseInvocation, WithMetadata, WithBoard)
     )
 
     @staticmethod
-    def vae_encode(vae_info: LoadedModel, image_tensor: torch.Tensor) -> torch.Tensor:
+    def vae_encode(
+        vae_info: LoadedModel, image_tensor: torch.Tensor, tiled: bool = False, tile_size: int = 0
+    ) -> torch.Tensor:
         # NOTE: vae_info.model may be an AutoencoderKLWan (a native-layout qwen_image_vae single file is
         # classified with the Anima base); it is reinterpreted as AutoencoderKLQwenImage inside the
         # model_on_device context below. The working-memory estimate only reads tensor shape + element
         # size, so it is safe to run on either class here.
+        # Resolve tile_size=0 ("model default") before estimating, so the reserved working memory
+        # matches the tiles the VAE will actually use.
+        effective_tile_size = None
+        if tiled:
+            effective_tile_size = tile_size if tile_size > 0 else getattr(vae_info.model, "tile_sample_min_height", 256)
+
         estimated_working_memory = estimate_vae_working_memory_qwen_image(
             operation="encode",
             image_tensor=image_tensor,
             vae=vae_info.model,
+            tile_size=effective_tile_size,
         )
         with vae_info.model_on_device(working_mem_bytes=estimated_working_memory) as (_, vae):
             # Reinterpret an Anima-classified Wan VAE as AutoencoderKLQwenImage (identical weights).
             vae = as_qwen_image_vae(vae)
 
-            vae.disable_tiling()
+            # Tiling bounds the encode's peak memory to a single tile, which is what makes large
+            # inputs (e.g. a 2560x1440 upscale round-trip) encodable while a multi-GB transformer
+            # is still resident. Off by default: full-frame is faster and avoids tile blending.
+            if tiled:
+                if tile_size > 0:
+                    vae.enable_tiling(tile_sample_min_height=tile_size, tile_sample_min_width=tile_size)
+                else:
+                    vae.enable_tiling()
+            else:
+                vae.disable_tiling()
 
             image_tensor = image_tensor.to(device=TorchDevice.choose_torch_device(), dtype=vae.dtype)
             with torch.inference_mode():
@@ -91,7 +113,12 @@ def invoke(self, context: InvocationContext) -> LatentsOutput:
 
         # If target dimensions are specified, resize the image BEFORE encoding
         # (matching the diffusers pipeline which resizes in pixel space, not latent space).
-        if self.width is not None and self.height is not None:
+        #
+        # `width`/`height` are `int | None`, but the workflow UI cannot represent None in a number
+        # input and sends 0 for "unset" — which `is not None`, so a naive check reached
+        # `resize((0, 0))` and raised "height and width must be > 0". Treat any non-positive value
+        # as unset, which is also how `tile_size` uses 0.
+        if self.width and self.height and self.width > 0 and self.height > 0:
             image = image.convert("RGB").resize((self.width, self.height), resample=PILImage.LANCZOS)
 
         # multiple_of=16 ensures the post-VAE latents (vae_scale_factor=8) have even
@@ -102,7 +129,12 @@ def invoke(self, context: InvocationContext) -> LatentsOutput:
 
         vae_info = context.models.load(self.vae.vae)
 
-        latents = self.vae_encode(vae_info=vae_info, image_tensor=image_tensor)
+        latents = self.vae_encode(
+            vae_info=vae_info,
+            image_tensor=image_tensor,
+            tiled=self.tiled or context.config.get().force_tiled_decode,
+            tile_size=self.tile_size,
+        )
 
         latents = latents.to("cpu")
         name = context.tensors.save(tensor=latents)

invokeai/app/invocations/qwen_image_latents_to_image.py

@@ -27,20 +27,34 @@
     title="Latents to Image - Qwen Image",
     tags=["latents", "image", "vae", "l2i", "qwen_image"],
     category="latents",
-    version="1.0.0",
+    version="1.1.0",
     classification=Classification.Prototype,
 )
 class QwenImageLatentsToImageInvocation(BaseInvocation, WithMetadata, WithBoard):
     """Generates an image from latents using the Qwen Image VAE."""
 
     latents: LatentsField = InputField(description=FieldDescriptions.latents, input=Input.Connection)
     vae: VAEField = InputField(description=FieldDescriptions.vae, input=Input.Connection)
+    tiled: bool = InputField(default=False, description=FieldDescriptions.tiled)
+    # NOTE: tile_size = 0 is a special value meaning "use the model's default", matching the
+    # SD/SDXL l2i node. `int | None` is avoided because the workflow UI does not handle it well.
+    tile_size: int = InputField(default=0, multiple_of=8, description=FieldDescriptions.vae_tile_size)
 
     @torch.no_grad()
     def invoke(self, context: InvocationContext) -> ImageOutput:
         latents = context.tensors.load(self.latents.latents_name)
 
         vae_info = context.models.load(self.vae.vae)
+        tiled = self.tiled or context.config.get().force_tiled_decode
+        # Resolve tile_size=0 ("model default") before estimating, so the memory the cache reserves
+        # matches the tiles the VAE will actually use. Without this the estimate stays at the
+        # full-frame figure (~21 GB at 2560x1440 on CUDA) and tiling frees nothing: the VAE is
+        # bounded, but the cache still evicts other models to honour the reservation.
+        effective_tile_size = None
+        if tiled:
+            effective_tile_size = (
+                self.tile_size if self.tile_size > 0 else getattr(vae_info.model, "tile_sample_min_height", 256)
+            )
         # NOTE: vae_info.model may be an AutoencoderKLWan (a native-layout qwen_image_vae single file is
         # classified with the Anima base); it is reinterpreted as AutoencoderKLQwenImage inside the
         # model_on_device context below. The working-memory estimate only reads tensor shape + element
@@ -49,6 +63,7 @@ def invoke(self, context: InvocationContext) -> ImageOutput:
             operation="decode",
             image_tensor=latents,
             vae=vae_info.model,
+            tile_size=effective_tile_size,
         )
         with vae_info.model_on_device(working_mem_bytes=estimated_working_memory) as (_, vae):
             context.util.signal_progress("Running VAE")
@@ -62,13 +77,15 @@ def invoke(self, context: InvocationContext) -> ImageOutput:
                 # which would wrongly place the latents (and thus the whole decode) on the CPU (see #9373).
                 latents = latents.to(device=vae_info.compute_device, dtype=vae.dtype)
 
-                # Honor the global force_tiled_decode setting, like the SD/SDXL l2i node. Tiling bounds the
-                # VAE's per-tile memory, which is the scalable way to decode very large outputs that would
-                # exceed VRAM even after offloading the transformer/text encoder. For normal sizes, leave
-                # it off (faster, no tile blending) — the reserved working memory offloads other models so
-                # the full-frame decode fits.
-                if context.config.get().force_tiled_decode:
-                    vae.enable_tiling()
+                # Tiling bounds the VAE's per-tile memory, which is the scalable way to decode very
+                # large outputs that would exceed VRAM even after offloading the transformer/text
+                # encoder. For normal sizes, leave it off (faster, no tile blending) — the reserved
+                # working memory offloads other models so the full-frame decode fits.
+                if tiled:
+                    if self.tile_size > 0:
+                        vae.enable_tiling(tile_sample_min_height=self.tile_size, tile_sample_min_width=self.tile_size)
+                    else:
+                        vae.enable_tiling()
                 else:
                     vae.disable_tiling()