What are the benefits of sparse representations and sparse parameters? What are their benefits? I know sparse parameters are a different story than sparse representations, but I want to know how each of these can benefit us and which one is more important than the other one.  
 A: Sparse representations have two main purposes:  


*

*They are form of regularization, that pushes parameters to exact zeros. It works as any other form of regularization, so basically, it leads to simpler model by learning what parameters can be dropped, lowering their total number.  

*Another reason for using them, is to produce models that are smaller in terms of memory usage, for example, when you need to send them over network, or store on mobile devices. That's the same reason as for using sparse data formats. 
Think of a recommender system, where you have thousands of users and thousands of products, so the matrix of all the possible interactions is huge, by using sparse data format you can save a lot of memory. Same with sparse model representation. Deep learning models have a lot of redundancies, and in many cases you can get rid of a lot of weights by preserving the quality of results, as discussed for example by James Kwork in this talk on Compressed Deep Neural Networks. You can remove those redundant weights and use sparser solution, that needs less memory to be stored.
Notice that sparse models are defined in terms of sparse weights, they are not different story.
A: I can think of two benefits. 
The first one is that it simplifies interpretation considerably which makes understanding our models easier (probably not so relevant in the case of neural networks)
Another benefit of having sparse representations, that arise e.g. in Lasso-Regression is that the penalization that induces the sparsity reduces the estimation variance (at the cost of larger bias)
Consider e.g. a really noisy linear regression where there are only 2 covariates that have a real influence on $y$ and 100 noisy ones (that have no real influence). If we fit a linear model (including the noisy covariates) there will be a lot of estimation variance which will likely result in a bad fit. L1-regularization however can handle this pretty well by inducing sparsity in the parameters and thereby reducing the estimation variance. (see here)
