2
$\begingroup$

What are their benefits? I know sparse parameters are a different story than sparse representations, but I want to know how each of these can benefit us and which one is more important than the other one.

$\endgroup$

2 Answers 2

2
$\begingroup$

Sparse representations have two main purposes:

  • They are form of , that pushes parameters to exact zeros. It works as any other form of regularization, so basically, it leads to simpler model by learning what parameters can be dropped, lowering their total number.
  • Another reason for using them, is to produce models that are smaller in terms of memory usage, for example, when you need to send them over network, or store on mobile devices. That's the same reason as for using sparse data formats.

    Think of a recommender system, where you have thousands of users and thousands of products, so the matrix of all the possible interactions is huge, by using sparse data format you can save a lot of memory. Same with sparse model representation. Deep learning models have a lot of redundancies, and in many cases you can get rid of a lot of weights by preserving the quality of results, as discussed for example by James Kwork in this talk on Compressed Deep Neural Networks. You can remove those redundant weights and use sparser solution, that needs less memory to be stored.

Notice that sparse models are defined in terms of sparse weights, they are not different story.

$\endgroup$
2
  • $\begingroup$ Thanks a lot. what about the sparse parameters? if I impose a regularization term on activations of lets say a layer, so that their absolute value would go to zero, it will affect the parameters values, making them have small/near zero values. will this result in sparse representation as well? will these sparse parameters be of any use o us? $\endgroup$
    – Hossein
    Oct 6, 2019 at 15:55
  • 1
    $\begingroup$ @Breeze basically sparse representations are defined in terms of sparse parameters. The difference is that usual regularization, like $L_1$ or $L_2$, lead to parameters that are close to zero, while sparse representations drag them to exact zeros. $\endgroup$
    – Tim
    Oct 6, 2019 at 17:09
2
$\begingroup$

I can think of two benefits.

The first one is that it simplifies interpretation considerably which makes understanding our models easier (probably not so relevant in the case of neural networks)

Another benefit of having sparse representations, that arise e.g. in Lasso-Regression is that the penalization that induces the sparsity reduces the estimation variance (at the cost of larger bias)

Consider e.g. a really noisy linear regression where there are only 2 covariates that have a real influence on $y$ and 100 noisy ones (that have no real influence). If we fit a linear model (including the noisy covariates) there will be a lot of estimation variance which will likely result in a bad fit. L1-regularization however can handle this pretty well by inducing sparsity in the parameters and thereby reducing the estimation variance. (see here)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.