I am training a conditional variational autoencoder on a dataset of faces. When I set my KLL Loss equal to my Reconstruction loss term, my autoencoder seems unable to produce varied samples. I always get the same types of faces appearing:
The problem is that the learned latent space is not smooth. If I try to perform a latent interpolation or generate a random sample, I get junk. When the KLL term has a small weight (0.001), I observe the following loss behavior: Notice that the VggLoss (the reconstruction term) decreases, while the KLLoss continues to increase.
I also tried increasing the dimensionality of the latent space, but this didn't work either.
Notice here, when the two loss terms are of equal weight, how the KLL term dominates but doesn't allow the reconstruction loss to decrease:
This results in terrible reconstructions. Are there any suggestions on how to balance these two loss terms or any other possible things to try so that my autoencoder learns a smooth, interpolative latent space while producing reasonable reconstructions?