2
$\begingroup$

For a problem I'm dealing with, I'm trying to understand if my approach could make sense.

I'm using a Variational AutoEncoder (VAE) having relatively low-dimensional inputs, say $x \in \mathbb{R}^n$. Since my dimension $n$ is small, I'm trying to use the VAE by substantially inverting the role of encoder and decoder (inverse-bottleneck) to create high dimensional latent representation $z \in \mathbb{R}^m$ where $m > n$ and then reducing its dimension again obtaining the reconstruction $\hat{x}$.

So by the means of neural networks I'm substantially trying to approximate a decoder to the high-dimensional space $\phi : \mathbb{R}^ n \rightarrow \mathbb{R}^m$ and then an encoder back to the original space $\psi : \mathbb{R}^m \rightarrow \mathbb{R}^n$.

This usage of VAE is not common at all (it is used instead as dimensionality reduction approach). But I'm still trying to understand why my approach could eventually be wrong at the bottom from a mathematical perspective.

I mean, in general it's reasonable to project features in higher dimensional space to enhance separation (kernel methods), so here we are just learning the best way to project these inputs to a such space..

$\endgroup$

1 Answer 1

4
$\begingroup$

You did not say, what your actual final goal is. But in any case, what a VAE will learn in this setting, contrary to when a proper bottleneck is used, will simply be (a noisy version of) the identity. And the (noisy) values $z\in\mathbb{R}^m$ might also not be of much help, since it can actually be anything since there is no restriction. Think of any arbitrary mapping from $x\in\mathbb{R}^n$ to (a distribution in) $\mathbb{R}^m$ and this mapping could, in principle, be obtained by this VAE. So I don't see that the latent values would give useful representations of your data.

However, adding some constraints might be worthwhile. E.g. people have, IIRC, tried to get better models via "anti-bottlenecks" by penalizing denseness of the $z$-vectors (or, favoring sparsity). Or you could try to severely restrict the architecture of your network.

$\endgroup$
6
  • $\begingroup$ My goal would be learning to generate potentially new instances of low-dimensional vectors. I get your point, so in the case of vae, we may only approximate a noisy identity function, but what about the case of projecting features to high-dimensional space instead? There the situation would be different? $\endgroup$ Apr 7, 2022 at 12:54
  • $\begingroup$ @JamesArten What means: "generate potentially new instances of low-dimensional vectors"? Do you want a low dimensional representation of your data, or an embedding as submanifold, or ...? And: What does "projecting features to high-dimensional space" refer to? $\endgroup$
    – frank
    Apr 7, 2022 at 13:23
  • $\begingroup$ I just want to use VAE as generative model to generate new instances of my original vectors: isn’t this a very common goal while using VAEs? Also, with projecting to high dimensional space I simply refer to the context while we start with original features, $x \in \mathbb{R}^d$ and by using a map $\phi : \mathbb{R}^d \rightarrow \mathbb{R}^m$ we project them into a space $\mathbb{R}^m$ with $m >> d$. $\endgroup$ Apr 7, 2022 at 14:02
  • $\begingroup$ The latter situation may occur when we want to obtain a better separation of our instances, eventually to fit an optimal decision hyperplane. $\endgroup$ Apr 7, 2022 at 14:11
  • 1
    $\begingroup$ Many thanks for your answers. $\endgroup$ Apr 7, 2022 at 15:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.