2D ControlNet tutorial

[postmarone96]

Hi,

I have 3 question pertaining the Controlnet tutorial:

1. why do we also set the UNet into training mode during the training loop? I thought that we freeze the weights of the UNet and only train the ControlNet ?

2. During Validation: instead of using the inferer for noise prediction, why not set the ControlNet into validation mode and predict the noise via the frozen UNet model, like we did in the training phase ?

3. if I wanted to train inside a latent space of an AutoencoderKL, should I use the method autoencoderkl.encoder or autoencoderkl.encode, if I understood it correctly one encode and adds noise already and the other one only encodes the images into the latent space.

Thank you for your help and answer in advance.
Greetings,
Maroan

[ericspod]

@virginiafdez would be able to comment here? Thanks!

[virginiafdez]

Dear Maroan,

Here's a reply to your questions:

1. It does not really make a difference, because the requires_grad flag of the U-Net parameters are set to False.
2. I just checked the tutorial and the validation loop is wrong indeed. I'll fix it as soon as possible. We also recently implemented a new inferer for ControlNet, so the new code will be cleaner, but essentially, everything should be exactly as in training (unless you are sampling) except for the train / eval modes, as you say.
3. There is no noise added in the encoding function. With the autoencoder, you just apply the reparametrization trick to add noise after you calculate the mean and standard deviations of your input, but this whole process has nothing to do with the diffusion model. You always need to use encode_stage_2_inputs. This is done inside of the inferer forward method.

Hope this clarifies your doubts. I will create an issue and work on it for point 2.

Thanks for the post.

Virginia

[postmarone96]

Hi Virginia,
Thank you very much for your answer, looking forward to the fixes.
Maroan

[postmarone96]

Hi @virginiafdez ,
if I wanted to use the ControlNetLatentDiffusionInferer:

• what is the difference between cn_cond and conditioning ?
• should cn_cond have the same spacial dimensions as the latent ?
  and for the controlNet initialization:
• what do Both of conditioning_embedding_num_channels and conditioning_embedding_in_channels mean ?
  Thank you in advance for answering my questions.
  Marwan

[virginiafdez]

Hello,

• cn_cond is the ControlNet conditioning, which should be an image. The "conditioning"is the normal conditioning for the LDM, which can be concatenated (mode = "concat") or cross-attention. If you go to the normal LDM and inferers (without ControlNet), you will see that conditioning is still there, whereas cn_cond isn't.
• at the moment, we have no resizing mechanisms, so yes, the spatial dimensions of the input conditioning for ControlNet should have the same spatial size.
• before mixing the input and conditioning for ControlNet, the ControlNet conditioning (cn_cond) is brough to an embedding space, which outputs the same number of channels as the self.conv_in of the LDM / ControlNet [they have symmetric architectures], then you add both things. However, this stack of layers between the cn_cond input and the embeddings that are added to the output of self.conv_in(x) is flexible. You can specify how many intermediate channels you want ( conditioning_embedding_num_channels ). conditioning_embedding_in_channels is just the number of input channels of the input conditioning (cn_cond). You can see how that happens in line 383 (forward method) of controlnet.py.

Hope this helps!
Best wishes,

Virginia