Why use $L_2$ distance as reconstruction error for an autoencoder as opposed to $L_k$

Question

When training an autoencoder on continuous data (that is, not binary), almost all papers/implementations I've seen minimize the $L_2$ reconstruction error between the feature vector $\mathbf{x}$ and the decoded sparse representation $\hat{\mathbf{x}}$, i.e., $\mathcal L = \sqrt{||\mathbf{x} - \hat{\mathbf{x}}||^2}$ (though usually without the sqrt so its differentiable everywhere). I've recently read On the Surprising Behavior of Distance Metrics in High Dimensional Space, which seems to suggest that for high dimension (the authors use ~ 20), the Euclidean distance metric does not provide a reasonable measure of closeness. Briefly, the authors state (section 2, first paragraph):

...the difference between the maximum and minimum distances to a given query point does not increase as fast as the nearest distance to any point in high dimensional space. This makes a proximity query meaningless and unstable because there is poor discrimination between the nearest and furthest neighbor.

While they mainly discuss this result in the context of measuring a nearest neighbor in a high dimensional space, I jumped to its implications for autoencoders.

A brief search did not show me any papers or studies in which autoencoders were trained with other than $L_2$ reconstruction error, as opposed to a generic $L_k$ loss (i.e., $k=1, 1/2, \dots$). Have $L_k$ reconstruction errors been considered before for autoencoders (I'm guessing yes)? And, given the results of the paper cited above, why are different metrics not used (besides just "tradition")?

Update: In the paper Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion, the authors demonstrate that minimizing the $L_2$ loss is equivalent to maximizing the mutual information between the reconstructed inputs and the original ones. This is a reasonable theoretical justification for the use of the $L_2$ metric over others.

Answer 1

Autoencoders can be considered in the framework of [Variational Autoencoders][1] (VAEs), which effectively generalise deterministic autoencoders, where:

• each data sample $x$ is mapped to a distribution $q(z|x)$ over latent space, rather than to a unique value of $z$ (as given by a deterministic encoder function)
• similarly, each latent representation $z$ is mapped to a distribution over the data space $p(x|z)$, e.g. a small Gaussian around a learned mean;
• the latent variables are fitted to a prior distribution $p(z)$

The point of considering VAEs is that they learn a proper latent variable model where terms of the loss function have a meaningful interpretation.

The deterministic autoencoder can be seen as a special case of the VAE framework where:

• the variance of $q(z|x)$ is reduced towards 0, so that $q(z|x)$ in the limit tends/concentrates to a deterministic function of $x$;
• the mean square ($L_2$) loss (mentioned in the question) relates to the reconstruction term of the VAE loss function and is equivalent to assuming $p(x|z)$ is Gaussian whose mean is learned as a function of $z$; and
• the second (KL or regularisation) term of the VAE loss, including the prior over $z$, is dropped (so no assumed structure is imposed in the latent space).

The distribution choice for $p(x|z)$ can be varied, which corresponds to different metrics in $x$-space (e.g. $L_1$ equivalent to Laplacian, etc).