8
$\begingroup$

Auto-encoders are used for dimensionality reduction, and as a tool for unsupervised feature learning. Further, auto-encoders have been used for building and training multi-layer neural networks.

When we talk about auto-encoders, we introduce a sparsity and I want to know, what is the purpose of introducing an sparsity term for auto-encoders?

$\endgroup$
1
  • 2
    $\begingroup$ I'd guess its because otherwise the auto-encoder would learn the identity mapping. Though, I forget the details, though the intuition is very simple, if you don't put any kind of constraints on the auto-encoder, then if the auto-encoder is trying to minimize some kind of cost function $\frac{1}{n}\sum_{t} \| x^{(t)} - \hat{x}^{(t)}\|_{\mathcal{H}}$, the it might be that $x^{(t)} - \hat{x}^{(t)} = 0 \iff x^{(t)} = \hat{x}^{(t)}$ is the obvious/trivial/vacuous minimizer. $\endgroup$ Jul 12, 2015 at 4:50

1 Answer 1

4
$\begingroup$

Remember that auto-encoders are trying to come up with the best summarized/compressed (hidden/laten) representation of the data such that the reconstruction error is minimized. This can be expressed in equations as follow:

$$ \min_{\hat{x} \in \mathcal{F}} \frac{1}{n} \sum^{n}_{t=1} cost(x^{(t)}, \hat{x}^{(t)} )$$

where $\hat{x}^{(t)} = \hat{x}(x^{(t)})$ is the reconstruction for data point $t$, $\mathcal{F}$ is the space of functions we are considering and $cost$ is some cost/distance function of your choice.

If you choose such a powerful model as mutli-layer neural networks that is a universality approximator, then it means you can basically copy any input data that you get. Therefore, the solution to the above problem could becomes vacuous and trivial, since your neural network might learn the identity mapping:

$\hat{x}^{(i)} = \hat{x}(x^{(i)}) = x^{(i)}$

This could happen if your cost function is such that its zero when the $cost(x^{(t)}, \hat{x}^{(t)} ) = cost(x^{(t)}, x^{(t)} )$. This is probably true for a lot of commonly used cost functions, like euclidean distance. For example:

$$ \min_{\hat{x} \in \mathcal{F}} \frac{1}{n} \sum^{n}_{t=1} \frac{1}{2} \| x^{(t)} - \hat{x}^{(t)} \|^2_{2}$$

is zero when $x^{(t)} - \hat{x}^{(t)} = 0 \iff x^{(t)} = \hat{x}^{(t)}$.

Therefore, without any further constraints on the space of functions you are considering, you would not learn a meaningful compressed/hidden representation of the data (since you system/algorithm is too powerful and basically is kind of "overfitting").

Requiring a sparse solution is just one way of getting around this problem. Its just imposing a prior on your given problem so that you can get a meaningful solution. Intuitively, its similar to getting around the "no-free-lunch" issue. Without some kind of prior, its hard to get something useful.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.