# How does one know if normalizing is improves reconstructions in the task of auto-encoding?

I wanted to understand the performance on an algorithm in the auto-encoding task and compare understand if normalizing the data was a good idea or not and compare the performance when the data is normalized vs when its not (normalize as in $\frac{x}{\| x \|}$ ).

The algorithm is trained via stochastic gradient descent (say its some neural network) and I train it via with the data and when the data is not normalized. Naturally, when the data is normalized, its likely to see smaller errors (just because we are dividing by a quantity $\| x \| \geq 1$) so nearly by definition the error "seems" smaller because the data has been shrunk. However, this doesn't mean that the algorithm necessarily performed better (or worse). Therefore, I was trying to understand how should compare the performance of the algorithm on the same task when the data is normalized and when its not.

Naturally, if the problem I was trying to solve was instead classification, the problem would be very easy to solve, because, if the accuracy (in terms of correct labels predicted) increases, then we know that the algorithm is performing better when the data (say images) are normalized. But in the case of auto-encoding, its harder to see this.

Is the issue of having a hard time to compare the performance inherit in that I am considering the wrong loss function (i.e. $L(f(x), y) = \| f(x) - y \|^2$ doesn't capture the information I want to learn) or what is the issue? How do people overcome this issue in practice and know if its worth normalizing in the case of auto-encoding task?

Obviously, one can just observe the reconstruction for say, a fraction of the images and the human can asses which one is better, but this approach doesn't seem to scale. Is there a better way to do this or has this been addressed in practice or theoretically?

• What is $y$? Does the output activation linear? Feb 27 '16 at 6:36
• @yasin.yazici $y = x$ since its an auto-encoder. I think the issue I presented is not really an issue in particular to NN or any specific learning algorithm (so it doesn't matter if the activation is linear, its just any kernel or activation, linear, non-linear). The issue is that comparing the results/errors of an auto-encoder where the error is normalized vs when its not. The error is (potentially) artificially shrunk by normalization, but that doesn't necessarily mean the auto-encoder is doing better. Feb 28 '16 at 21:17

You can compute your error not on the on the original input image vs output image, but on scaled input image and scaled output image, which will solve your problems of different scales. Also you can use different similarity metrics, that are scale invariant, for example correlation or cosine similarity.

• so after the learning has been done one do as follow: $L(\frac{ f(x) }{ \| f(x) \| } , \frac{ x }{\| x \| } )$, right? Is there a way to prove that this is mathematially correct? What worried me about this approach was that $f(x)$ would be normalized in the method you mentioned, but the function learned on the normalized training data set is not necesserily normalized. Wouldn't that be an issue? Mar 17 '16 at 15:38

Did you consider scaling the loss of the normalized auto-encoder by your normalization factor? As in multiply by ||x|| again?

Be careful with squared error though. You'd have to apply the scaling factor inside the power (i.e. (f(x)*||x||)² not f(x)² * ||x||. Or just use absolute error instead.

.

One thing from the theoretical side to consider: IIRC, An auto-encoder layer with N nodes is identical to the rank-N truncated SVD of the input matrix.

For the SVD, if your data is not centered around the origin, the first singular vector will point to the center of the data (you can easily see this in an example).

So, for equal N, when your data is z-normalized (I know, this is not what you're doing here), i.e. mean = 0, std = 1, you have one more singular vector to "work with" to describe the variance of your data compared to when it's unnormalized.

This also makes sense intuitively, as you lose the information about where your data is located (say in an area between x = 2 and x = 5) during normalization, while you keep the information about its structure around that center point.

• as in, for any loss $L(f(x), y)$, do $\| x \| L(f(x), y)$? For the squared loss I mentioned would be $\| f(x) - y \|^2 \cdot \| x\|$, why would that be mathematically correct? Mar 17 '16 at 15:32
• Sorry, maybe I was imprecise. My line of thinking was that you "scale up" your loss by the normalization factor again. || (f(x) - y) * ||x|| ) || ² In a sense, you normalize your data, perform the auto-encoding and de-normalize the results back to the original scale, then compute the error. This way you're comparing apples to apples. What you mention here would not be mathematically correct, because (square of normalized loss) * normalization factor is not the same as the square of the unnormalized loss Alternatively you can choose a loss where this property holds, such as absolute error Mar 21 '16 at 9:05