Auto-encoders are used for dimensionality reduction, and as a tool for unsupervised feature learning. Further, auto-encoders have been used for building and training multi-layer neural networks.
When we talk about auto-encoders, we introduce a sparsity and I want to know, what is the purpose of introducing an sparsity term for auto-encoders?
Remember that auto-encoders are trying to come up with the best summarized/compressed (hidden/laten) representation of the data such that the reconstruction error is minimized. This can be expressed in equations as follow:
$$ \min_{\hat{x} \in \mathcal{F}} \frac{1}{n} \sum^{n}_{t=1} cost(x^{(t)}, \hat{x}^{(t)} )$$
where $\hat{x}^{(t)} = \hat{x}(x^{(t)})$ is the reconstruction for data point $t$, $\mathcal{F}$ is the space of functions we are considering and $cost$ is some cost/distance function of your choice.
If you choose such a powerful model as mutli-layer neural networks that is a universality approximator, then it means you can basically copy any input data that you get. Therefore, the solution to the above problem could becomes vacuous and trivial, since your neural network might learn the identity mapping:
$\hat{x}^{(i)} = \hat{x}(x^{(i)}) = x^{(i)}$
This could happen if your cost function is such that its zero when the $cost(x^{(t)}, \hat{x}^{(t)} ) = cost(x^{(t)}, x^{(t)} )$. This is probably true for a lot of commonly used cost functions, like euclidean distance. For example:
$$ \min_{\hat{x} \in \mathcal{F}} \frac{1}{n} \sum^{n}_{t=1} \frac{1}{2} \| x^{(t)} - \hat{x}^{(t)} \|^2_{2}$$
is zero when $x^{(t)} - \hat{x}^{(t)} = 0 \iff x^{(t)} = \hat{x}^{(t)}$.
Therefore, without any further constraints on the space of functions you are considering, you would not learn a meaningful compressed/hidden representation of the data (since you system/algorithm is too powerful and basically is kind of "overfitting").
Requiring a sparse solution is just one way of getting around this problem. Its just imposing a prior on your given problem so that you can get a meaningful solution. Intuitively, its similar to getting around the "no-free-lunch" issue. Without some kind of prior, its hard to get something useful.
