When training an autoencoder on continuous data (that is, not binary), almost all papers/implementations I've seen minimize the $L_2$ reconstruction error between the feature vector $\mathbf{x}$ and the decoded sparse representation $\hat{\mathbf{x}}$, i.e., $\mathcal L = \sqrt{||\mathbf{x} - \hat{\mathbf{x}}||^2}$ (though usually without the sqrt so its differentiable everywhere). I've recently read On the Surprising Behavior of Distance Metrics in High Dimensional Space, which seems to suggest that for high dimension (the authors use ~ 20), the Euclidean distance metric does not provide a reasonable measure of closeness. Briefly, the authors state (section 2, first paragraph):

...the difference between the maximum and minimum distances to a given query point does not increase as fast as the nearest distance to any point in high dimensional space. This makes a proximity query meaningless and unstable because there is poor discrimination between the nearest and furthest neighbor.

While they mainly discuss this result in the context of measuring a nearest neighbor in a high dimensional space, I jumped to its implications for autoencoders.

A brief search did not show me any papers or studies in which autoencoders were trained with other than $L_2$ reconstruction error, as opposed to a generic $L_k$ loss (i.e., $k=1, 1/2, \dots$). Have $L_k$ reconstruction errors been considered before for autoencoders (I'm guessing yes)? And, given the results of the paper cited above, why are different metrics not used (besides just "tradition")?

Update: In the paper Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion, the authors demonstrate that minimizing the $L_2$ loss is equivalent to maximizing the mutual information between the reconstructed inputs and the original ones. This is a reasonable theoretical justification for the use of the $L_2$ metric over others.

  • 1
    $\begingroup$ I don't know about autorcoders, however using l2 norm is pretty common in nearly all optimization problems. L1 norm is robust but needs an especially designed optimizer since it is not differentiable. Other norms are not convex and difficult to establish theoretical result on them. $\endgroup$ – TPArrow Nov 8 '17 at 5:36
  • 1
    $\begingroup$ Also if the goal of the autoencoder is to reconstruct anything in the image that isn't noise, and you assume that noise is (independently and identically) Normally distributed, then minimizing the L2 norm of the residuals is equivalent to maximum likelihood estimation of the components of the image. $\endgroup$ – Ruben van Bergen Nov 8 '17 at 9:32
  • $\begingroup$ @RubenvanBergen Thanks for your insight. This gives a reasonable theoretical justification for why one would use an L2 loss, even for high dimensional data. $\endgroup$ – the higgs broson Nov 8 '17 at 14:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.