Skip to content

overview.md

Latest commit

 

History

History
95 lines (69 loc) · 8.57 KB

overview.md

File metadata and controls

95 lines (69 loc) · 8.57 KB

Schedulers

🤗 Diffusers provides many scheduler functions for the diffusion process. A scheduler takes a model's output (the sample which the diffusion process is iterating on) and a timestep to return a denoised sample. The timestep is important because it dictates where in the diffusion process the step is; data is generated by iterating forward n timesteps and inference occurs by propagating backward through the timesteps. Based on the timestep, a scheduler may be discrete in which case the timestep is an int or continuous in which case the timestep is a float.

Depending on the context, a scheduler defines how to iteratively add noise to an image or how to update a sample based on a model's output:

  • during training, a scheduler adds noise (there are different algorithms for how to add noise) to a sample to train a diffusion model
  • during inference, a scheduler defines how to update a sample based on a pretrained model's output

Many schedulers are implemented from the k-diffusion library by Katherine Crowson, and they're also widely used in A1111. To help you map the schedulers from k-diffusion and A1111 to the schedulers in 🤗 Diffusers, take a look at the table below:

A1111/k-diffusion 🤗 Diffusers Usage
DPM++ 2M [DPMSolverMultistepScheduler]
DPM++ 2M Karras [DPMSolverMultistepScheduler] init with use_karras_sigmas=True
DPM++ 2M SDE [DPMSolverMultistepScheduler] init with algorithm_type="sde-dpmsolver++"
DPM++ 2M SDE Karras [DPMSolverMultistepScheduler] init with use_karras_sigmas=True and algorithm_type="sde-dpmsolver++"
DPM++ 2S a N/A very similar to DPMSolverSinglestepScheduler
DPM++ 2S a Karras N/A very similar to DPMSolverSinglestepScheduler(use_karras_sigmas=True, ...)
DPM++ SDE [DPMSolverSinglestepScheduler]
DPM++ SDE Karras [DPMSolverSinglestepScheduler] init with use_karras_sigmas=True
DPM2 [KDPM2DiscreteScheduler]
DPM2 Karras [KDPM2DiscreteScheduler] init with use_karras_sigmas=True
DPM2 a [KDPM2AncestralDiscreteScheduler]
DPM2 a Karras [KDPM2AncestralDiscreteScheduler] init with use_karras_sigmas=True
DPM adaptive N/A
DPM fast N/A
Euler [EulerDiscreteScheduler]
Euler a [EulerAncestralDiscreteScheduler]
Heun [HeunDiscreteScheduler]
LMS [LMSDiscreteScheduler]
LMS Karras [LMSDiscreteScheduler] init with use_karras_sigmas=True
N/A [DEISMultistepScheduler]
N/A [UniPCMultistepScheduler]

Noise schedules and schedule types

A1111/k-diffusion 🤗 Diffusers
Karras init with use_karras_sigmas=True
sgm_uniform init with timestep_spacing="trailing"
simple init with timestep_spacing="trailing"
exponential init with timestep_spacing="linspace", use_exponential_sigmas=True
beta init with timestep_spacing="linspace", use_beta_sigmas=True

Token diffusion schedulers

These schedulers operate over categorical token IDs instead of continuous latents. They are designed for discrete token diffusion models and expose the same set_timesteps/step interface as other schedulers.

Differences between the discrete token schedulers:

  • TokenDiffusionScheduler: token-level diffusion with per-token corruption (e.g. mask/uniform) and a single-step step to denoise logits.
  • BlockTokenDiffusionScheduler: block-wise token diffusion that updates fixed-size blocks in parallel.
  • HybridTokenDiffusionScheduler: hybrid transitions that combine token- and block-wise updates in the same schedule.
  • DFlashTokenDiffusionScheduler: block diffusion scheduler specialized for speculative decoding with a draft model and target acceptance.
  • SDARTokenDiffusionScheduler: block diffusion scheduler with remasking strategies (sequential/low-confidence/entropy-bounded) per step.

[[autodoc]] TokenDiffusionScheduler

[[autodoc]] BlockTokenDiffusionScheduler

[[autodoc]] HybridTokenDiffusionScheduler

[[autodoc]] DFlashTokenDiffusionScheduler

[[autodoc]] SDARTokenDiffusionScheduler

All schedulers are built from the base [SchedulerMixin] class which implements low level utilities shared by all schedulers.

SchedulerMixin

[[autodoc]] SchedulerMixin

SchedulerOutput

[[autodoc]] schedulers.scheduling_utils.SchedulerOutput

KarrasDiffusionSchedulers

[KarrasDiffusionSchedulers] are a broad generalization of schedulers in 🤗 Diffusers. The schedulers in this class are distinguished at a high level by their noise sampling strategy, the type of network and scaling, the training strategy, and how the loss is weighed.

The different schedulers in this class, depending on the ordinary differential equations (ODE) solver type, fall into the above taxonomy and provide a good abstraction for the design of the main schedulers implemented in 🤗 Diffusers. The schedulers in this class are given here.

PushToHubMixin

[[autodoc]] utils.PushToHubMixin