DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps

The official code for the paper DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps (Neurips 2022 Oral) and DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models by Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li and Jun Zhu.

DPM-Solver (and the improved version DPM-Solver++) is a fast dedicated high-order solver for diffusion ODEs with the convergence order guarantee. DPM-Solver is suitable for both discrete-time and continuous-time diffusion models without any further training. Experimental results show that DPM-Solver can generate high-quality samples in only 10 to 20 function evaluations on various datasets.

Guided-Diffusion with DPM-Solver:

Stable-Diffusion with DPM-Solver++:

DiffEdit with DPM-Solver++:

Use the SOTA Multistep DPM-Solver++ with Diffusers

🤗 Diffusers is a fantastic library for diffusion models. It supports both DPM-Solver and DPM-Solver++. The multistep DPM-Solver++ is the fastest solver currently.

Stable-Diffusion

The second-order multistep DPM-Solver++ is the default solver for Stable-Diffusion online demos (e.g., see example) and can also be used in LoRA (e.g., see example). Here is an example:

import torch
from diffusers import StableDiffusionPipeline, DPMSolverMultistepScheduler

model_id = "stabilityai/stable-diffusion-2-1"

# Use the DPMSolverMultistepScheduler (DPM-Solver++) scheduler here
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
pipe = pipe.to("cuda")

prompt = "a photo of an astronaut riding a horse on mars"
image = pipe(prompt).images[0]

image.save("astronaut_rides_horse.png")

DeepFloyd-IF

We recommend the SDE version DPM-Solver++ for the stage-1, and the ODE version DPM-Solver++ for the upscaling stages (both stage-2 and 3).

from diffusers import DiffusionPipeline, DPMSolverMultistepScheduler
from diffusers.utils import pt_to_pil
import torch

# stage 1
stage_1 = DiffusionPipeline.from_pretrained("DeepFloyd/IF-I-XL-v1.0", variant="fp16", torch_dtype=torch.float16)
stage_1.enable_xformers_memory_efficient_attention()  # remove line if torch.__version__ >= 2.0.0
stage_1.enable_model_cpu_offload()

# stage 2
stage_2 = DiffusionPipeline.from_pretrained(
    "DeepFloyd/IF-II-L-v1.0", text_encoder=None, variant="fp16", torch_dtype=torch.float16
)
stage_2.enable_xformers_memory_efficient_attention()  # remove line if torch.__version__ >= 2.0.0
stage_2.enable_model_cpu_offload()

# stage 3
safety_modules = {"feature_extractor": stage_1.feature_extractor, "safety_checker": stage_1.safety_checker, "watermarker": stage_1.watermarker}
stage_3 = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-x4-upscaler", **safety_modules, torch_dtype=torch.float16)
stage_3.enable_xformers_memory_efficient_attention()  # remove line if torch.__version__ >= 2.0.0
stage_3.enable_model_cpu_offload()


def set_scheduler(stage):
    if scheduler_name == 'dpm++':
        scheduler = DPMSolverMultistepScheduler.from_config(stage.scheduler.config)
        scheduler.config.algorithm_type = 'dpmsolver++'
    elif scheduler_name == 'sde-dpm++':
        scheduler = DPMSolverMultistepScheduler.from_config(stage.scheduler.config)
        scheduler.config.algorithm_type = 'sde-dpmsolver++'
    stage.scheduler = scheduler
    return stage


upscale_steps = 25
stage_1 = set_scheduler(stage_1, 'sde-dpm++')
stage_2 = set_scheduler(stage_2, 'dpm++')
stage_3 = set_scheduler(stage_3, 'dpm++')

prompt = "casual photo of a leaf maple syrup glass container sitting on a wooden table in a log cabin, high depth of field during golden hour as the sunlight shines through the windows, dusty air"

# text embeds
prompt_embeds, negative_embeds = stage_1.encode_prompt(prompt)

generator = torch.manual_seed(0)

# stage 1
image = stage_1(prompt_embeds=prompt_embeds, negative_prompt_embeds=negative_embeds, generator=generator, output_type="pt").images
pt_to_pil(image)[0].save("./if_stage_I.png")

# stage 2
image = stage_2(
    image=image, prompt_embeds=prompt_embeds, negative_prompt_embeds=negative_embeds, generator=generator, output_type="pt", num_inference_steps=upscale_steps
).images
pt_to_pil(image)[0].save("./if_stage_II.png")

# stage 3
image = stage_3(prompt=prompt, image=image, generator=generator, noise_level=100, num_inference_steps=upscale_steps).images
image[0].save("./if_stage_III.png")

• News
• Supported Models and Algorithms
• Code Examples
• Stable-Diffusion with DPM-Solver
• ScoreSDE with DPM-Solver
• Other Examples
• Use DPM-Solver in your own code
• Suggestions for Choosing the Hyperparameters
• Suggestions for the Detailed Settings
  • Example: Unconditional Sampling by DPM-Solver
  • Example: Classifier Guidance Sampling by DPM-Solver
  • Example: Classifier-Free Guidance Sampling by DPM-Solver
• Documentation
• 1. Define the noise schedule.
• 2. Wrap your model to a continuous-time noise predicition model.
• 3. Define DPM-Solver
• TODO List
• References

News

• DPM-Solver has been used in:

  • DreamStudio and StableBoost (thanks for the implementations by Katherine Crowson's k-diffusion repo).
  • Stable-Diffusion-WebUI, which supports both DPM-Solver and DPM-Solver++. DPM-Solver++2M is the fastest solver currently. Also many Thanks to Katherine Crowson's k-diffusion repo.
  • Diffusers, a widely-used library for diffusion models.
  • Stable-Diffusion v2 Official Code and Stable-Diffusion v1 Official Code. Check this pull request.
  • Stable-Diffusion v2.0 Online Demo and Stable-Diffusion v1.5 Online Demo on HuggingFace, which uses DPM-Solver in Diffusers.
  • Core ML Stable Diffusion, by Apple, and Swift Core ML Diffusers by Hugging Face.

• 2022-12-15. Swift port of DPM-Solver++ (restricted to order 2, no dynamic thresholding), used in Apple's Core ML Stable Diffusion library and 🤗 Hugging Face Swift Core ML Diffusers demo app: Source code, App Store link. See this PR for details.

• 2022-11-11. The official demo of stable-diffusion in HuggingFace Spaces 🤗 uses DPM-Solver and runs twice as fast! (From 50 steps to 25 steps.) It can generate 8 images within only 4 seconds using JAX on TPUv2-8. Check this twitter.

• 2022-11-08. We provide an online demo for DPM-Solver with stable-diffusion. Many thanks for the help and harware resource support by HuggingFace 🤗!

• 2022-11-07. Happy to announce that the multistep DPM-Solver has been supported by diffusers! Thanks for all the efforts of huggingface team (and me ^_^). Check this PR for details.

• 2022-10-26. We have updated the DPM-Solver v2.0, a more stable version for high-resolutional image synthesis tasks. We have the following upgrades:

  • We support the discrete-time DPMs by implementing a picewise linear interpolation of $\log\alpha_t$ for the NoiseScheduleVP.

    We strongly recommend to use the new implementation for discrete-time DPMs, especially for high-resolutional image synthesis. You can set schedule='discrete' to use the corresponding noise schedule. We also change the mapping between discrete-time inputs and continuous-time inputs in the model_wrapper, which has a consistent converged results with the other solvers. - We change the API for model_wrapper: - We support four types of diffusion models: noise prediction model, data prediction model, velocity prediction model, score function. - We support unconditional sampling, classifier guidance sampling and classifier-free guidance sampling. - We support new algorithms for DPM-Solver, which greatly improve the high-resolutional image sample quality by guided sampling. - We support both DPM-Solver and DPM-Solver++. For DPM-Solver++, we further support the dynamic thresholding introduced by Imagen. - We support both singlestep solver (i.e. Runge-Kutta-like solver) and multistep solver (i.e. Adams-Bashforth-like solver) for DPM-Solver, including order 1, 2, 3.

Supported Models and Algorithms

Models

We support the following four types of diffusion models. You can set the model type by the argument model_type in the function model_wrapper.

Sampling Types

We support the following three types of sampling by diffusion models. You can set the argument guidance_type in the function model_wrapper.

Algorithms in DPM-Solver

We support the following four algorithms. The algorithms are DPM-Solver and DPM-Solver++.

We also support the dynamic thresholding introduced by Imagen for algorithms with data-prediction. T