0
$\begingroup$

I am currently learning about Ridge and Lasso Regression, which leads me to learn about L1 and L2 regularization. There's a phrase saying L2 Regularization doesn't result in sparse model and L1 regularization can result in sparse models with few coefficients.

I know what sparse matrix means, a matrix with very few non-zero elements (hope I am not wrong about it) but I am not able to understand what a sparse model means. I tried to search about it on google but didn't find any satisfying answer. I am seeing this term a lot now, so It will be great if you can help me understand this.

I am not a math guy and so please be soft and use less terminology while answering.

$\endgroup$
3
$\begingroup$

Recall what L1 and L2 regularization do: they drive the parameters towards small values, zeroes in a perfect case. A sparse model has many parameters zeroed out by regularization. You can implement it to actually use the sparse data structures for the parameters to save memory and compute.

$\endgroup$
4
  • $\begingroup$ "You can implement it to actually use the sparse data structures to hold the parameters to save memory and compute." -Didn't get this line Are you saying that after applying l1 or l2 many features (parameter) will become zero(approx) and the model with many zeroed parameters is a sparse model right? Which then can be stored in any sparse data structure. Also what is sparse data structures $\endgroup$ Jul 31 at 9:43
  • 1
    $\begingroup$ @DarkstarDream sparse vector holds only non-zero elements saving space. Dot product with such vector means multiplying only the non-zero elements because all the others are zeros when summing , so less operations. But this is a technical comment, sparse model is just one with zeroed out parameters. $\endgroup$
    – Tim
    Jul 31 at 10:12
  • $\begingroup$ So can I just conclude that a sparse model is a model with many zeroed parameters resulted from the operation of L1 or L2 regularization? $\endgroup$ Jul 31 at 10:29
  • 1
    $\begingroup$ @DarkstarDream yes, it’s a model which has sparse parameters. $\endgroup$
    – Tim
    Jul 31 at 10:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.