I'm integrating cleanDIFT with some other modules for my dense prediction task. However, when I train the entire network (as in other modules of the network while the diffusion model is frozen), I am getting memory leaks (my GPU memory slowly keeps increasing subsequently leading to CUDA OOM error). Is there any reason for this phenomenon as a lot of people report memory leaks with different stable diffusion models but no real solution is available anywhere. Any help would be really appreciated!!
I'm integrating cleanDIFT with some other modules for my dense prediction task. However, when I train the entire network (as in other modules of the network while the diffusion model is frozen), I am getting memory leaks (my GPU memory slowly keeps increasing subsequently leading to CUDA OOM error). Is there any reason for this phenomenon as a lot of people report memory leaks with different stable diffusion models but no real solution is available anywhere. Any help would be really appreciated!!