HyperKohaku

A diffusers based implementation of HyperDreamBooth

the code is based on the HyperDreamBooth implementation in the LyCORIS project.

TODOs

If you have any idea on these todos, just open issues or PRs. Thx!

• Implement in-run validation
• Implement section 4
• Implement better dreambooth training (like mask the face during training)
• Implement a more general hypernetwork (Already in LyCORIS)

HyperDreamBooth

This section is a brief introduction of HyperDreamBooth, I will split hyperdreambooth into 4 sections. And this projection will have 4 corresponding scripts

Section 1: Pre Optimize

Before we start to train the hypernetwork, we should train the lora(lilora) on each identites(instance) first. In this implementation, we took a batch of instances and then do a inner training loop to get the pre optimized weights.

Section 2: train the hypernetwork

Just train it. send the image into hypernetwork, get the weights, apply to unet. Calc the loss based on diffusion loss and weight loss.

Section 3: gen the weight

Use the image of the identity you want to train on, generate the weight from hypernetwork.

Section 4: further finetuning

resume from the generated weight, do a few step training (for about 20~50step)