I am learning about imposing structure on the latent variables in autoencoders. In that context I have looked at variational autoencoders (VAEs) and adversarial autoencoders (AAEs). This paper proposes a variation on the (Wasserstein) AAE that can be used on categorical input data.

I have observed in my own data that both VAEs and AAEs struggle when I use mixed or categorical input data. Why can categorical data not be captured by a continuous latent variable after several layers of nonlinear transformations? What kind of distribution would be the prior of choice (e.g. in the paper I linked)? I am having trouble picturing what that means for the latent variable.

Update: I found that the concerns about capturing categorical variables with continuous latent variables is well explained here for PCA, and it also applies to basic linear autoencoders. I assume the argument scales to nonlinear layers as well. I could be wrong but in the paper I linked the latent space is still continuous and the prior gaussian, but we create a map between samples from a normal distribution and the latent space that allows to use the AE as as a generative model. I still have many questions, but wanted to share this in case it is useful to anyone.

  • $\begingroup$ I don't agree with the arguments in that reddit post. First, using the covariance matrix doesn't mean PCA assumes the data is Gaussian (it doesn't, as it's simply a matrix factorization method and doesn't involve probability distributions). Second, the fact that PCA won't work well for uniformly distributed binary data isn't very compelling--it won't work well for uniformly distributed real data either, for exactly the same reasons. $\endgroup$ – user20160 Dec 5 '18 at 2:46
  • $\begingroup$ Interesting, thanks for this! It sort of made sense to me but I see I'll have to revisit this. $\endgroup$ – Luise Jan 2 at 14:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.