I am learning about imposing structure on the latent variables in autoencoders. In that context I have looked at variational autoencoders (VAEs) and adversarial autoencoders (AAEs). This paper proposes a variation on the (Wasserstein) AAE that can be used on categorical input data.
I have observed in my own data that both VAEs and AAEs struggle when I use mixed or categorical input data. Why can categorical data not be captured by a continuous latent variable after several layers of nonlinear transformations? What kind of distribution would be the prior of choice (e.g. in the paper I linked)? I am having trouble picturing what that means for the latent variable.
Update: I found that the concerns about capturing categorical variables with continuous latent variables is well explained here for PCA, and it also applies to basic linear autoencoders. I assume the argument scales to nonlinear layers as well. I could be wrong but in the paper I linked the latent space is still continuous and the prior gaussian, but we create a map between samples from a normal distribution and the latent space that allows to use the AE as as a generative model. I still have many questions, but wanted to share this in case it is useful to anyone.