What is the advantage of sparsity? In many problems the solution is sparse and the authors of methods usually present it as an advantage. How could one leverage the sparsity in general? Why is an advantage?
 A: I would not so much call it an advantage but a way out in otherwise intractable situations. In particular, in high dimensional problems, we face the situation that the number of available predictors $p$ is much larger than the number of observations $n$. As is well known, classical methods like OLS do not work (do not have a unique solution) in case $p>n$. Hence, regularized or sparse methods like the LASSO become attractive or even indispensable alternatives.
Now, if you want to for example demonstrate theoretical results like that LASSO is capable of finding the relevant predictors out of a large set of predictors, it turns out you need to make such sparsity assumptions that the set of these actually relevant variables is suitably "small".
See, e.g., Statistics for High-Dimensional Data - Methods, Theory and Applications by Bühlmann, Peter, van de Geer, Sara.
A: I found this interesting post on sparsity. In my opinion sparsity helps in even terms of statistical robustness of the solution. Having an over complete representation of the features, help averaging the statistical fluctuations introduced during the training phase. But this is an intuition, I have no proof.
