I'm reading a RL article that is using a B-VAE to encode states. Here is the original article presenting reinforcement learning with imagined goals (RIG) Ashvin Nair et al., "Visual Reinforcement Learning with Imagined Goals" 2018.

In the presented algorithm, the authors put a step (page 6, Algorithm 1, line 3) that is : "Fit prior $p(z)$ to latent encodings $\{\mu_{\phi}(s^{(i)})\}$

I understand what $p(z)$ and $\{\mu_{\phi}(s^{(i)})\}$ are (or at least I think I do ...) but I have no idea about how to fit one to another. Any idea or helpful references?

  • $\begingroup$ What is a B-VAE? The article you link to does not mention the term. $\endgroup$ – Sycorax Mar 28 at 17:27
  • $\begingroup$ Presumably it’s the $\beta$-VAE in eq. (2). $\endgroup$ – Arya McCarthy Mar 28 at 17:41
  • $\begingroup$ yeah exactly, i don't know how to type beta $\endgroup$ – Hedwin Bonnavaud Mar 28 at 19:11
  • $\begingroup$ @Sycorax In VAE you have two losses. The first is the reconstruction loss. The second is the regularization term that responsible for generalization. The second term is the part that enables you to generelize and to be able to generate objects from the random latent space. $\beta$ helps you to control the tradeoff between the two losses. for example: In some cases you don't need to generate completely from random, so you can have less focus on the second regularization term i.e. set $\beta$ < 1 $\endgroup$ – ofer-a Mar 29 at 8:59

When training VAE you have auxiliary loss to fit the encoder output to P(Z) - a multivariate standard normal. But in practice the resulting distribution is not exactly standard normal. In this article they have another phase. They don't insist on standard normal, instead they are willing to change their prior to other gaussian distribution that will be closer to the encoder output. So the question is now, what is the final gaussian distribution of the encoder output. I believe that this can be found by minimizing the KL divergence between the encoder output and a new gaussian. i.e. find which mu, sigma of each dimension of the latent space, minimizes the KL divergence to the encoder output. KL divergence between two normal distribution has an analytic solution and therefore can be minimized by gradient descent algorithms


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.