Adding random noise to latent representation increase the accuracy in the autoencoder I am working on an autoencoder project, it consists of dense layers like this :
dense ([10, 756] )--> dense ( [10, 512] ) --> latent ( [ 10, 256] ) --> dense ( [10, 512] ) --> dense ([10, 756])

This is a very simple setup, and training it with mean square error loss i am getting accuracy ~0.79
I thought to experiment with variational autoencoder but it didn't work out well on this problem, But the strange thing is if I am adding noise in latent representation, it actually increases the accuracy to ~ 0.85
Adding noise in latent then setup is
dense ([10, 756] )--> dense ( [10, 512] ) --> latent ( [ 10, 256] ) + random_normal([ 10, 256] ) --> dense ( [10, 512] ) --> dense ([10, 756])

It's giving good accuracy but I want to know the theory behind it, I thought it's denoising auto-encoder but since in denosing auto-encoder we add noise in input samples not in latent space so we can't say it's denosing auto-encoder. How should I explain why it's increasing the accuracy.
 A: I can give you two explanations, one is intuitive and the other is more mathematical:
The intuitive explanation:
During the training, each data point is seen by model several times (once per epoch for example) and the model is updated through out the training. So every time a data point is mapped to a different point in the latent space (due to changes of the encoder). But after a while, as the training goes on, these changes get smaller and smaller, until they eventually become negligible. So from this point on, each point is mapped to the same point in the latent space. And the decoder tries to reconstruct the data from that latent representation.
But if you add noise to the latent space, it's like instead of passing one data point several times, you are passing a little cluster of several different points. This has two effects:

*

*It makes decoder more robust and lessen the risk of overfitting, because it has to learn that very small changes in the latent space should not change the output drastically.

*It helps with the generalization, since you are effectively introducing a little premise which your data space is at least locally convex (meaning that if a point belongs to your distribution, then there exists a neighborhood around it that also belongs to your distribution)

The more mathematical explanation:
The autoencoder is trying to estimate the underlying distribution of your data. And it measures the error and tries to improve the estimated distribution. The problem is that in real world applications, the data space often has a low dimensional support. Meaning that the actual meaningful part of the data can be represented in a few dimensions. Even in very high dimensional problems (like working with images) still the actual structure of the data can be represented in very lower dimensions. For example even though an image may have several thousands of pixels, but we know that not all the possible combinations have meaning. And also there are lots of constraints on the data. Like if the picture is of a skyscraper, then it is probably a tall and straight thing. Or if it is a picture of a dog, then it probably has 4 legs and a face with two eyes, etc.
So the data space is high dimensional, but the support manifold is low dimensional. In such situation, any two distributions are most likely disjoint, meaning they don't collide (namely, the actual data distribution and the estimated distribution). And the error metrics usually can't handle this case very smoothly. The more famous case of this problem is in the GAN architecture that leads to the use of the Wasserstein distance instead of previously-used Jensen-Shannon.
But when you add noise, you are basically stretching the distribution all over the space, and even though this added tail contains no information, it causes the two distributions to not be disjoint. And that helps with the stability of error metrics.

Sorry for the long answer. Hope it helps
