9
$\begingroup$

When training an autoencoder on continuous data (that is, not binary), almost all papers/implementations I've seen minimize the $L_2$ reconstruction error between the feature vector $\mathbf{x}$ and the decoded sparse representation $\hat{\mathbf{x}}$, i.e., $\mathcal L = \sqrt{||\mathbf{x} - \hat{\mathbf{x}}||^2}$ (though usually without the sqrt so its differentiable everywhere). I've recently read On the Surprising Behavior of Distance Metrics in High Dimensional Space, which seems to suggest that for high dimension (the authors use ~ 20), the Euclidean distance metric does not provide a reasonable measure of closeness. Briefly, the authors state (section 2, first paragraph):

...the difference between the maximum and minimum distances to a given query point does not increase as fast as the nearest distance to any point in high dimensional space. This makes a proximity query meaningless and unstable because there is poor discrimination between the nearest and furthest neighbor.

While they mainly discuss this result in the context of measuring a nearest neighbor in a high dimensional space, I jumped to its implications for autoencoders.

A brief search did not show me any papers or studies in which autoencoders were trained with other than $L_2$ reconstruction error, as opposed to a generic $L_k$ loss (i.e., $k=1, 1/2, \dots$). Have $L_k$ reconstruction errors been considered before for autoencoders (I'm guessing yes)? And, given the results of the paper cited above, why are different metrics not used (besides just "tradition")?

Update: In the paper Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion, the authors demonstrate that minimizing the $L_2$ loss is equivalent to maximizing the mutual information between the reconstructed inputs and the original ones. This is a reasonable theoretical justification for the use of the $L_2$ metric over others.

$\endgroup$
4
  • 2
    $\begingroup$ I don't know about autorcoders, however using l2 norm is pretty common in nearly all optimization problems. L1 norm is robust but needs an especially designed optimizer since it is not differentiable. Other norms are not convex and difficult to establish theoretical result on them. $\endgroup$
    – TPArrow
    Nov 8, 2017 at 5:36
  • 1
    $\begingroup$ Also if the goal of the autoencoder is to reconstruct anything in the image that isn't noise, and you assume that noise is (independently and identically) Normally distributed, then minimizing the L2 norm of the residuals is equivalent to maximum likelihood estimation of the components of the image. $\endgroup$ Nov 8, 2017 at 9:32
  • $\begingroup$ @RubenvanBergen Thanks for your insight. This gives a reasonable theoretical justification for why one would use an L2 loss, even for high dimensional data. $\endgroup$ Nov 8, 2017 at 14:34
  • $\begingroup$ When using L2, the closer we get to the optimum, the smaller the gradient gets and the slower we learn. Would it make sense to combine L1 (constant gradient even close at optimum) + L2? Otherwise the autoencoder will never be able to reconstruct the data 100% correctly. $\endgroup$ Apr 21, 2021 at 15:14

1 Answer 1

1
$\begingroup$

Autoencoders can be considered in the framework of [Variational Autoencoders][1] (VAEs), which effectively generalise deterministic autoencoders, where:

  • each data sample $x$ is mapped to a distribution $q(z|x)$ over latent space, rather than to a unique value of $z$ (as given by a deterministic encoder function)
  • similarly, each latent representation $z$ is mapped to a distribution over the data space $p(x|z)$, e.g. a small Gaussian around a learned mean;
  • the latent variables are fitted to a prior distribution $p(z)$

The point of considering VAEs is that they learn a proper latent variable model where terms of the loss function have a meaningful interpretation.

The deterministic autoencoder can be seen as a special case of the VAE framework where:

  • the variance of $q(z|x)$ is reduced towards 0, so that $q(z|x)$ in the limit tends/concentrates to a deterministic function of $x$;
  • the mean square ($L_2$) loss (mentioned in the question) relates to the reconstruction term of the VAE loss function and is equivalent to assuming $p(x|z)$ is Gaussian whose mean is learned as a function of $z$; and
  • the second (KL or regularisation) term of the VAE loss, including the prior over $z$, is dropped (so no assumed structure is imposed in the latent space).

The distribution choice for $p(x|z)$ can be varied, which corresponds to different metrics in $x$-space (e.g. $L_1$ equivalent to Laplacian, etc).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.