0
$\begingroup$

Setup:

Dataset: 40k uniformly distributed 13-dim samples (floats between 0 and 1)

AutoEncoder: (input: 13dim) - fc layer 13 dim, relu - latent layer - fc layer 13 dim, relu - (output: 13dim)

Loss: MSE

I am using this toy problem to check if I can get a good reconstruction of the input data.

Using a simple AutoEncoder (just reconstruction loss, MSE), if i keep the latent layer of also 13-dim, I get a good reconstruction after 4k epochs.

However, if I reduce the latent dim to 2, I am having problems obtaining a good reconstruction of the input data even after 5k epochs. Visualising the first two dimensions, the reconstruction looks squished w.r.t. the input.

Does anyone have experience with such a setting?

I am not sure if the problem is with the data being uniform or there is another bug I should look into.

$\endgroup$
1
$\begingroup$

I think this is perfectly normal because autoencoders are actually compressors. It's like you're zipping your files. If the information content inside your data is high, you can't compress it. There must be redundancies, which is typically caused by dependencies, and non-uniform data distribution. For example, if your image pixels are totally random, JPEG can't do much for you. Going back to original problem, you create a data with information content that cannot be efficiently compressed. The uniformity of data distribution makes it worse since every sample has the same likelihood. In the extreme case, think about a distribution where your samples are very much concentrated around zero. Then, the autoencoder can focus on reconstruction of those samples and catch redundancies better.

$\endgroup$
  • $\begingroup$ Just to clarify, by "I think this is perfectly normal" you refer to the bad reconstruction in the case when latent dimension is 2? Thanks $\endgroup$ – El Rakone Jun 21 at 10:33
  • $\begingroup$ Yes, I meant that. $\endgroup$ – gunes Jun 21 at 10:37
  • $\begingroup$ I reduced the dataset to just 10 samples, but cannot get it to over-fit when latent dimension is 2 (in fact any dim <8-10) Shouldn't it be possible to do that? (Using AdamOptimizer, tried different learning rates and batch sizes) $\endgroup$ – El Rakone Jun 23 at 16:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.