0
$\begingroup$

I recently trained a AE and a VAE and used the latent variables of each for a clustering task. It seemed to work well, sensible clusters. The main reason for training the VAE was too gain more interpretation from the learned variables.

To try and gain insight, I output the means and log vars of each learned variable from the encoder for a particular case, convert the log var to sd, and apply this to each of the learned variables to see what each is doing individually (ie holding the others as the mean while each is varied up to plus and minus 2 sd's in either direction.

The problem is, barely anything changes. Ie the variability for every variable is so small that any change is not noticeable.

My code appears to be correct, but I just wanted to ensure that my interpretation of how I should do this was correct before exploring other options or trying to train this differently. Thanks

Everything I tried is described above

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

You're better off varying the value of the target latent by plus or minus 2 SDs of the prior.

The SDs of the inferred variational beliefs can indeed be very small, if the network is simply very certain about the values of the latents given the input. For instance, it may be very certain that it saw a '2' in a given MNIST image, in which case the SD(s) along the digit-coding dimension(s) will be tiny. Sampling from the latent beliefs then will not produce significant variation in the decoded images (at least, not as far as the digit class is concerned).

Instead, what is typically done in order to explore the meaning of different latent dimensions, is to vary the value of each latent by a range that is set in proportion to the prior, since the prior is supposed to represent the range of values that are plausible on the whole. The prior of course is most often an uncorrelated Normal distribution with unit variance, and so varying a latent by up to 2 SDs of the prior would then simply mean a range of [-2, 2] around the inferred mean for that latent. Alternatively, you can also just sample values in the range [-2, 2] (or some other plausible range, like [-3, 3]), rather than centering the range on the inferred mean for a given input. This is arguably more theoretically sound, since centering the range on the inferred value for a given input may result in some values at one end of that range that are outside the training distribution (e.g., if a certain latent had an inferred mean of -1.7 for a given input, then a 2-SD range centered on that value would span from -3.7 to 0.3, and -3.7 is a very unlikely value under the prior that might not map to anything sensible).

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks Ruben. It appears I may have trained this incorrectly after all. Are you saying that the prior of each learned node should be N(0,1), and if it is, to simply vary using this to understand what each node is doing? I have plotted histograms of the latent space predictions for each node across all training instances. Mine are not distributed this way, since I used a relu dense layer in the latent space so wouldn't expect to be. I guess this was a mistake? I have seen linear activations used here also, but it's not obvious to me that would give me the desired property either. $\endgroup$
    – Nathan Thompo
    Commented Jul 16 at 10:02
  • $\begingroup$ N(0,1) is the typical prior used if your latent beliefs are Gaussian (but in principle you can use any prior). This prior shows up in the VAE cost function. It doesn't require or guarantee that in practice, the marginal distribution of your latents will follow this prior - it just acts as a regularizing force in the cost function. However, if you did use this prior (and if you don't know what prior you used, then chances are that this is the one you are using), then it doesn't make sense to have a ReLU activation layer at the end, as this will lead to strictly non-negative outputs. $\endgroup$
    – Ruben van Bergen
    Commented Jul 16 at 10:53
  • $\begingroup$ ReLU's also don't make sense to generate the log variances, since having the logvars be strictly non-negative means that the variances of your latent beliefs will always be greater than 1, which isn't what you want - you want your variances to be strictly positive, but they should be able to take values between 0-1. $\endgroup$
    – Ruben van Bergen
    Commented Jul 16 at 10:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.