I am using the glmnet package in R. When I set the alpha value = 0, I would expect that no variables are selected. When I look at the coefficients some of them are set to zero. What could be the explanation for this behaviour?
2 Answers
$\begingroup$
$\endgroup$
I found an explanation why I get some coefficients set to zero in my empirical data set. The reason seems to be, that some of my variables have the same information content over rows. I am working with SNP data. So in the case where a variable is set to zero they are e.g. homozygous for all indviduals. If I remove all this "monomorphic" positions I end up with the same amount of variables, which remain after fitting the ridge model and removing coefficients which were set to zero. So my assumption now is, as some predictors have no variance, the respective regression coefficients are not assessable and thus set to zero.
$\begingroup$
$\endgroup$
7
You should have a look at the glmnet's vignette. Having alpha set to zero does not mean that no variables are selected. It means that you only use the $\ell_2$-penalization. Let me know if you need more explanation ?
In fact, $\ell_1$ penalization tends to give you sparse estimates. If you have a look on the picture, you can see that it usually goes on the edge (i.e. forcing some coefficients to be zero). The $\ell_2$ penalization is supposed to "regroup variables" (harder to explain it with this picture).
Maybe I don't understand your question, but why did you think it would not select variables ?
-
$\begingroup$ Some explanation would be appreciated. I thought the l2-penalty can not set variables to zero. $\endgroup$– PatCommented May 28, 2014 at 9:25
-
$\begingroup$ I updated my answer. Yes, l2-penalty can set a variable to zero. Actually you can see it as a constraint, i.e., it tries to find a solution along the ellipsoids which minimizes the equation AND is in the black region. $\endgroup$ Commented May 28, 2014 at 9:38
-
-
$\begingroup$ Oh, I think I know what you mean. $\ell_1$ penalty tends to give you sparse estimates. It means that some of the coefficients are going to be zero, therefore you are going to pick only the coefficients with a value different from zero. If you only use $\ell_2$ penalty, you will not have this "behavior". This does not mean you cannot do "variable selection". The only different thing you have to do is to choose an $\epsilon$ and select the variables that are bigger than this $\epsilon$. You know what I mean ? $\endgroup$ Commented May 28, 2014 at 11:41
-
1$\begingroup$ With most real data sets it would highly unusual for any $l_2$ coefficients to be exactly zero (within numerical roundoff error). $\endgroup$– whuber ♦Commented May 28, 2014 at 15:43