Stable Diffusion Web UI Forge

Stable Diffusion Web UI Forge is a platform on top of Stable Diffusion WebUI to make development easier, optimize resource management, and speed up inference.

The name "Forge" is inspired from "Minecraft Forge". This project is aimed at becoming SD WebUI's Forge.

Compared to original WebUI (for SDXL inference at 1024px), you can expect the below speed-ups:

1. If you use common GPU like 8GB vram, you are expected to get about 30~45% speed up in inference speed (it/s), the GPU memory peak (in task manager) will drop about 700MB to 1.3GB, the maximum diffusion resolution (that will not OOM) will increase about 2x to 3x, and the maximum diffusion batch size (that will not OOM) will increase about 4x to 6x.

2. If you use less powerful GPU like 6GB vram, you are expected to get about 60~75% speed up in inference speed (it/s), the GPU memory peak (in task manager) will drop about 800MB to 1.5GB, the maximum diffusion resolution (that will not OOM) will increase about 3x, and the maximum diffusion batch size (that will not OOM) will increase about 4x.

3. If you use powerful GPU like 4090 with 24GB vram, you are expected to get about 3~6% speed up in inference speed (it/s), the GPU memory peak (in task manager) will drop about 1GB to 1.4GB, the maximum diffusion resolution (that will not OOM) will increase about 1.6x, and the maximum diffusion batch size (that will not OOM) will increase about 2x.

4. If you use ControlNet for SDXL, the maximum ControlNet count (that will not OOM) will increase about 2x, the speed with SDXL+ControlNet will speed up about 30~45%.

Another very important change that Forge brings is Unet Patcher. Using Unet Patcher, methods like Self-Attention Guidance, Kohya High Res Fix, FreeU, StyleAlign, Hypertile can all be implemented in about 100 lines of codes.

Thanks to Unet Patcher, many new things are possible now and supported in Forge, including SVD, Z123, masked Ip-adapter, masked controlnet, photomaker, etc.

No need to monkey patch UNet and conflict other extensions anymore!

Installing Forge

You can install Forge using same method as SD-WebUI. (Install Git, Python, Git Clone this repo and then run webui-user.bat).

Or you can just use this one-click installation package (with git and python included).

>>> Click Here to Download One-Click Package<<<

After you download, you can use update.bat to update and use run.bat to run.

Screenshots of Comparison

I tested with several devices, and this is a typical result from 8GB VRAM (3070ti laptop) with SDXL.

This is original WebUI:

(average about 7.4GB/8GB, peak at about 7.9GB/8GB)

This is WebUI Forge:

(average and peak are all 6.3GB/8GB)

You can see that Forge does not change WebUI results. Installing Forge is not a seed breaking change.

Forge can perfectly keep WebUI unchanged even for most complicated prompts like fantasy landscape with a [mountain:lake:0.25] and [an oak:a christmas tree:0.75][ in foreground::0.6][ in background:0.25] [shoddy:masterful:0.5].

All your previous works still work in Forge!

Also, Forge promise that we will only do our jobs. We will not add unnecessary opinioned changes to UI. You are still using 100% Automatic1111 WebUI.

Forge Backend

Forge backend removes all WebUI's codes related to resource management and reworked everything. All previous CMD flags like medvram, lowvram, medvram-sdxl, precision full, no half, no half vae, attention_xxx, upcast unet, ... are all REMOVED. Adding these flags will not cause error but they will not do anything now. We highly encourage Forge users to remove all cmd flags and let Forge to decide how to load models.

Without any cmd flag, Forge can run SDXL with 4GB vram and SD1.5 with 2GB vram.

The only one flag that you may still need is --always-offload-from-vram (This flag will make things slower). This option will let Forge always unload models from VRAM. This can be useful if you use multiple software together and want Forge to use less VRAM and give some vram to other software, or when you are using some old extensions that will compete vram with Forge, or (very rarely) when you get OOM.

If you really want to play with cmd flags, you can additionally control the GPU with:

(extreme VRAM cases)

--always-gpu
--always-cpu

(rare attention cases)

--attention-split
--attention-quad
--attention-pytorch
--disable-xformers
--disable-attention-upcast

(float point type)

--all-in-fp32
--all-in-fp16
--unet-in-bf16
--unet-in-fp16
--unet-in-fp8-e4m3fn
--unet-in-fp8-e5m2
--vae-in-fp16
--vae-in-fp32
--vae-in-bf16
--clip-in-fp8-e4m3fn
--clip-in-fp8-e5m2
--clip-in-fp16
--clip-in-fp32

(rare platforms)

--directml
--disable-ipex-hijack
--pytorch-deterministic

Again, Forge do not recommend users to use any cmd flags unless you are very sure that you really need these.

UNet Patcher

Now developing an extension is super simple. We finally have a patchable UNet.

Below is using one single file with 80 lines of codes to support FreeU:

extensions-builtin/sd_forge_freeu/scripts/forge_freeu.py

import torch
import gradio as gr
from modules import scripts


def Fourier_filter(x, threshold, scale):
    x_freq = torch.fft.fftn(x.float(), dim=(-2, -1))
    x_freq = torch.fft.fftshift(x_freq, dim=(-2, -1))
    B, C, H, W = x_freq.shape
    mask = torch.ones((B, C, H, W), device=x.device)
    crow, ccol = H // 2, W //2
    mask[..., crow - threshold:crow + threshold, ccol - threshold:ccol + threshold] = scale
    x_freq = x_freq * mask
    x_freq = torch.fft.ifftshift(x_freq, dim=(-2, -1))
    x_filtered = torch.fft.ifftn(x_freq, dim=(-2, -1)).real
    return x_filtered.to(x.dtype)


def set_freeu_v2_patch(model, b1, b2, s1, s2):
    model_channels = model.model.model_config.unet_config["model_channels"]
    scale_dict = {model_channels * 4: (b1, s1), model_channels * 2: (b2, s2)}

    def output_block_patch(h, hsp, *args, **kwargs):
        scale = scale_dict.get(h.shape[1], None)
        if scale is not None:
            hidden_mean = h.mean(1).unsqueeze(1)
            B = hidden_mean.shape[0]
            hidden_max, _ = torch.max(hidden_mean.view(B, -1), dim=-1, keepdim=True)
            hidden_min, _ = torch.min(hidden_mean.view(B, -1), dim=-1, keepdim=True)
            hidden_mean = (hidden_mean - hidden_min.unsqueeze(2).unsqueeze(3)) / \
                          (hidden_max - hidden_min).unsqueeze(2).unsqueeze(3)
            h[:, :h.shape[1] // 2] = h[:, :h.shape[1] // 2] * ((scale[0] - 1) * hidden_mean + 1)
            hsp = Fourier_filter(hsp, threshold=1, scale=scale[1])
        return h, hsp

    m = model.clone()
    m.set_model_output_block_patch(output_block_patch)
    return m


class FreeUForForge(scripts.Script):
    def title(self):
        return "FreeU Integrated"

    def show(self, is_img2img):
        # make this extension visible in both txt2img and img2img tab.
        return scripts.AlwaysVisible

    def ui(self, *args, **kwargs):
        with gr.Accordion(open=False, label=self.title()):
            freeu_enabled = gr.Checkbox(label='Enabled', value=False)
            freeu_b1 = gr.Slider(label='B1', minimum=0, maximum=2, step=0.01, value=1.01)
            freeu_b2 = gr.Slider(label='B2', minimum=0, maximum=2, step=0.01, value=1.02)
            freeu_s1 = gr.Slider(label='S1', minimum=0, maximum=4, step=0.01, value=0.99)
            freeu_s2 = gr.Slider(label='S2', minimum=0, maximum=4, step=0.01, value=0.95)

        return freeu_enabled, freeu_b1, freeu_b2, freeu_s1, freeu_s2

    def process_before_every_sampling(self, p, *script_args, **kwargs):
        # This will be called before every sampling.
        # If you use highres fix, this will be called twice.

        freeu_enabled, freeu_b1, freeu_b2, freeu_s1, freeu_s2 = script_args

        if not freeu_enabled:
            return

        unet = p.sd_model.forge_objects.unet

        unet = set_freeu_v2_patch(unet, freeu_b1, freeu_b2, freeu_s1, freeu_s2)

        p.sd_model.forge_objects.unet = unet

        # Below codes will add some logs to the texts below the image outputs on UI.
        # The extra_generation_params does not influence results.
        p.extra_generation_params.update(dict(
            freeu_enabled=freeu_enabled,
            freeu_b1=freeu_b1,
            freeu_b2=freeu_b2,
            freeu_s1=freeu_s1,
            freeu_s2=freeu_s2,
        ))

        return

It looks like this:

Similar components like HyperTile, KohyaHighResFix, SAG, can all be implemented within 100 lines of codes (see also the codes).

ControlNets can finally be called by different extensions.

Implementing Stable Video Diffusion and Zero123 are also super simple now (see also the codes).

Stable Video Diffusion:

extensions-builtin/sd_forge_svd/scripts/forge_svd.py

```python import torch import gradio as gr import os import pathlib

from modules import script_callbacks from modules.paths import models_path from modules.ui_common import ToolButton, refresh_symbol from modules import shared

from modules_forge.forge_util import numpy_to_pytorch, pytorch_to_numpy from ldm_patched.modules.sd import load_checkpoint_guess_config from ldm_patched.contrib.external_video_model import VideoLinearCFGGuidance, SVD_img2vid_Conditioning from ldm_patched.contrib.external import KSampler, VAEDecode

opVideoLinearCFGGuidance = VideoLinearCFGGuidance() opSVD_img2vid_Conditioning = SVD_img2vid_Conditioning() opKSampler = KSampler() opVAEDecode = VAEDecode()

svd_root = os.path.join(models_path, 'svd') os.makedirs(svd_root, exist_ok=True) svd_filenames = []

def update_svd_filenames(): global svd_filenames svd_filenames = [ pathlib.Path(x).name for x in shared.walk_files(svd_root, allowed_extensions=[".pt", ".ckpt", ".safetensors"]) ] return svd_filenames

@torch.inference_mode() @torch.no_grad() def predict(filename, width, height, video_frames, motion_bucket_id, fps, augmentation_level, sampling_seed, sampling_steps, sampling_cfg, sampling_sampler_name, sampling_scheduler, sampling_denoise, guidance_min_cfg, input_image): filename = os.path.join(svd_root, filename) model_raw, _, vae, clip_vision = \ load_checkpoint_guess_config(filename, output_vae=True, output_clip=False, output_clipvision=True) model = opVideoLinearCFGGuidance.patch(model_raw, guidance_min_cfg)[0] init_image = numpy_to_pytorch(input_image) positive, negative, latent_image = opSVD_img2vid_Conditioning.encode( clip_vision, init_image, vae, width, height, video_frames, motion_bucket_id, fps, augmentation_level) output_latent = opKSampler.sample(model, sampling_seed, sampling_steps, sampling_cfg, sampling_sampler_name, sampling_scheduler, positive, negative, latent_image, sampling_denoise)[0] output_pixels = opVAEDecode.decode(vae, output_latent)[0] outputs = pytorch_to_numpy(output_pixels) return outputs

def on_ui_tabs(): with gr.Blocks() as svd_block: with gr.Row(): with gr.Column(): input_image = gr.Image(label='Input Image', source='upload', type='numpy', height=400)

            with gr.Row():
                filename = gr.Dropdown(label="SVD Checkpoint Filename",
                                       choices=svd_filenames,
                                       value=svd_filenames[0] if len(svd_filenames) > 0 else None)
                refresh_button = ToolButton(value=refresh_symbol, tooltip="Refresh")
                refresh_button.click(
                    fn=lambda: gr.update(choices=update_svd_filenames),
                    inputs=[], outputs=filename)

            width = gr.Slider(label='Width', minimum=16, maximum=8192, step=8, value=1024)
            height = gr.Slider(label='Height', minimum=16, maximum=8192, step=8, value=576)
            video_frames = gr.Slider(label='Video Frames', minimum=1, maximum=4096, step=1, value=14)
            motion_bucket_id = gr.Slider(label='Motion Bucket Id', minimum=1, maximum=1023, step=1, value=127)
            fps = gr.Slider(label='Fps', minimum=1, maximum=1024, step=1, value=6)
            augmentation_level = gr.Slider(label='Augmentation Level', minimum=0.0, maximum=10.0, step=0.01,