In "β-VAE: LEARNING BASIC VISUAL CONCEPTS WITH A CONSTRAINED VARIATIONAL FRAMEWORK", the Introduction states that "With β > 1 the model is pushed to learn a more efficient latent representation of the data, which is disentangled ...".

In "Understanding disentangling in β-VAE", it is stated in section 4.3 that data locality pressure (due to the higher weight on the KL divergence) leads to latent features getting embedded in different latent axis. However, "this pressure alone would not discourage the representational axes from rotating relative to the factors". And, since the model has to allocate different variances to the different posteriors, p(z|x), the "the diagonal covariance of the posterior distribution restricts the model to doing this in different latent dimensions", ... "encouraging the latent dimensions to align with the factors".

While these intuitions help, I wonder if there is further quantitative evidence that a set of entangled features, say a linear combination of disentangled latent features using a unitary transformation (rotation), would not be just as efficient an encoding or if it would have a higher KL divergence between the estimated (approximate) posterior, q(z|x) and true posterior, p(z|x) because, if the two have the same KL divergence, how does the model converge to a disentangled representation by simply weighting it with β.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.