3 added 22 characters in body edited Oct 20 '16 at 21:47 Matthias Schmidtblaicher 1,17677 silver badges1818 bronze badges When I re-run your code, I get that the coefficient of $$x_2$$ is numerically indistinguishable from zero. To understand better why LASSO sets that coefficient to zero, you should look at the relationship between LASSO and Least Angle Regression (LAR). LASSO can be seen as a LAR with a special modification. The algorithm of LAR is roughly like this: Start with an empty model (except for an intercept). Then add the predictor variable that is the most correlated with $$y$$, say $$x_j$$. Incrementally change that predictor's coefficient $$\beta_j$$, until the residual $$y - c - x_j\beta_j$$ is equally correlated with $$x_j$$ and another predictor variable $$x_k$$. Then change the coefficients of both $$x_j$$ and $$x_k$$ until a third predictor $$x_l$$ is equally correlated with the residual $$y - c - x_j\beta_j -x_k\beta_k$$ and so on. LASSO can be seen as LAR with the following twist: as soon as the coefficient of a predictor in your model (an "active" predictor) hits zero, drop that predictor from the model. This is what happens when you regress $$y$$ on the collinear predictors: both will get added to the model at the same time and, as their coefficients are changed, their respective correlation with the residuals remains the samewill change proportionately, but one of the predictors will get dropped from the active set first because it hits zero first. As for which of the two collinear predictors it will be, I don't know. [EDIT: When you reverse the order of $$x_1$$ and $$x_2$$, you can see that the coefficient of $$x_1$$ is set to zero. So the glmnet algorithm simply seems to set those coefficients to zero first that are ordered later in the design matrix.] A source that explains these things more in detail is Chapter 3 in "The Elements of Statistical Learning" by Friedman, Hastie and Tibshirani. When I re-run your code, I get that the coefficient of $$x_2$$ is numerically indistinguishable from zero. To understand better why LASSO sets that coefficient to zero, you should look at the relationship between LASSO and Least Angle Regression (LAR). LASSO can be seen as a LAR with a special modification. The algorithm of LAR is roughly like this: Start with an empty model (except for an intercept). Then add the predictor variable that is the most correlated with $$y$$, say $$x_j$$. Incrementally change that predictor's coefficient $$\beta_j$$, until the residual $$y - c - x_j\beta_j$$ is equally correlated with $$x_j$$ and another predictor variable $$x_k$$. Then change the coefficients of both $$x_j$$ and $$x_k$$ until a third predictor $$x_l$$ is equally correlated with the residual $$y - c - x_j\beta_j -x_k\beta_k$$ and so on. LASSO can be seen as LAR with the following twist: as soon as the coefficient of a predictor in your model (an "active" predictor) hits zero, drop that predictor from the model. This is what happens when you regress $$y$$ on the collinear predictors: both will get added to the model at the same time and, as their coefficients are changed, their correlation with the residuals remains the same, but one of the predictors will get dropped from the active set first because it hits zero first. As for which of the two collinear predictors it will be, I don't know. A source that explains these things more in detail is Chapter 3 in "The Elements of Statistical Learning" by Friedman, Hastie and Tibshirani. When I re-run your code, I get that the coefficient of $$x_2$$ is numerically indistinguishable from zero. To understand better why LASSO sets that coefficient to zero, you should look at the relationship between LASSO and Least Angle Regression (LAR). LASSO can be seen as a LAR with a special modification. The algorithm of LAR is roughly like this: Start with an empty model (except for an intercept). Then add the predictor variable that is the most correlated with $$y$$, say $$x_j$$. Incrementally change that predictor's coefficient $$\beta_j$$, until the residual $$y - c - x_j\beta_j$$ is equally correlated with $$x_j$$ and another predictor variable $$x_k$$. Then change the coefficients of both $$x_j$$ and $$x_k$$ until a third predictor $$x_l$$ is equally correlated with the residual $$y - c - x_j\beta_j -x_k\beta_k$$ and so on. LASSO can be seen as LAR with the following twist: as soon as the coefficient of a predictor in your model (an "active" predictor) hits zero, drop that predictor from the model. This is what happens when you regress $$y$$ on the collinear predictors: both will get added to the model at the same time and, as their coefficients are changed, their respective correlation with the residuals will change proportionately, but one of the predictors will get dropped from the active set first because it hits zero first. As for which of the two collinear predictors it will be, I don't know. [EDIT: When you reverse the order of $$x_1$$ and $$x_2$$, you can see that the coefficient of $$x_1$$ is set to zero. So the glmnet algorithm simply seems to set those coefficients to zero first that are ordered later in the design matrix.] A source that explains these things more in detail is Chapter 3 in "The Elements of Statistical Learning" by Friedman, Hastie and Tibshirani. Post Undeleted by Matthias Schmidtblaicher occurred Oct 20 '16 at 21:41 2 Completely rewritten - I first mistook the code for a ridge regression due to the wrong naming. edited Oct 20 '16 at 21:41 Matthias Schmidtblaicher 1,17677 silver badges1818 bronze badges First of all, you are not fitting a LASSO but a Ridge regression (alpha = 0 would correspond to LASSO)When I re-run your code, i.e. you estimateI get that the coefficient: $$\beta_{ridge} = (X'X + \lambda I)^{-1}X'y$$ So even though of $$X'X$$$$x_2$$ is singular, the added diagonal matrix makes the term in brackets invertiblenumerically indistinguishable from zero.   Ridge will in general not set coefficients equalTo understand better why LASSO sets that coefficient to zero, but shrink them all a bit, unless you have perfect collinearityshould look at the relationship between LASSO and Least Angle Regression (LAR). To see how it shrinks them, youLASSO can relate write its predicted valuebe seen as a LAR with a special modification. The algorithm of LAR is roughly like this: $$X\beta_{ridge} = U' \left[\begin{array}{cc} \frac{d_{1}^2}{d_{1}^2 + \lambda}& 0 \\ 0 & \frac{d_{k}^2}{d_{k}^2 + \lambda} \end{array}\right] U' y$$ Where Start with an empty model (except for an intercept). Then add the columns ofpredictor variable that is the most correlated with $$U$$ spanning$$y$$, say $$x_j$$. Incrementally change that predictor's coefficient $$\beta_j$$, until the column space ofresidual $$X$$$$y - c - x_j\beta_j$$ is equally correlated with $$x_j$$ and theanother predictor variable $$d_{j}^2$$$$x_k$$. Then change the eigenvaluescoefficients of both $$X'X$$. Since$$x_j$$ and $$X'X$$$$x_k$$ until a third predictor $$x_l$$ is singular in your case,equally correlated with the second eigenvalue is zero, so your system actually looks like this:residual $$X\beta_{ridge} = U' \left[\begin{array}{cc} \frac{d_{1}^2}{d_{1}^2 + \lambda}& 0 \\ 0 & 0 \end{array}\right] U' y$$$$y - c - x_j\beta_j -x_k\beta_k$$ When I run your code, I also get that the second coefficient is of the order $$10^{-15}$$,and so it is virtually zeroon.  Interestingly, LASSO will not set onecan be seen as LAR with the following twist: as soon as the coefficient equal toof a predictor in your model (an "active" predictor) hits zero but give positive values to both. To understand that, drop that predictor from the model. This is what happens when you should have a lookregress $$y$$ on the collinear predictors: both will get added to the model at the relationship between Least Angle Regressionsame time and LASSO, as their coefficients are changed, their correlation with the residuals remains the same, but one of the predictors will get dropped from the active set first because it hits zero first. As for which of the two collinear predictors it will be, I don't know. A source that I recommend to understandexplains these methods betterthings more in detail is Chapter 3 in "The Elements of Statistical Learning" by Friedman, Hastie and Tibshirani. First of all, you are not fitting a LASSO but a Ridge regression (alpha = 0 would correspond to LASSO), i.e. you estimate the coefficient: $$\beta_{ridge} = (X'X + \lambda I)^{-1}X'y$$ So even though $$X'X$$ is singular, the added diagonal matrix makes the term in brackets invertible.   Ridge will in general not set coefficients equal to zero, but shrink them all a bit, unless you have perfect collinearity. To see how it shrinks them, you can relate write its predicted value as: $$X\beta_{ridge} = U' \left[\begin{array}{cc} \frac{d_{1}^2}{d_{1}^2 + \lambda}& 0 \\ 0 & \frac{d_{k}^2}{d_{k}^2 + \lambda} \end{array}\right] U' y$$ Where the columns of $$U$$ spanning the column space of $$X$$ and the $$d_{j}^2$$ the eigenvalues of $$X'X$$. Since $$X'X$$ is singular in your case, the second eigenvalue is zero, so your system actually looks like this: $$X\beta_{ridge} = U' \left[\begin{array}{cc} \frac{d_{1}^2}{d_{1}^2 + \lambda}& 0 \\ 0 & 0 \end{array}\right] U' y$$ When I run your code, I also get that the second coefficient is of the order $$10^{-15}$$, so it is virtually zero. Interestingly, LASSO will not set one coefficient equal to zero but give positive values to both. To understand that, you should have a look at the relationship between Least Angle Regression and LASSO. A source that I recommend to understand these methods better is Chapter 3 in "The Elements of Statistical Learning" by Friedman, Hastie and Tibshirani. When I re-run your code, I get that the coefficient of $$x_2$$ is numerically indistinguishable from zero. To understand better why LASSO sets that coefficient to zero, you should look at the relationship between LASSO and Least Angle Regression (LAR). LASSO can be seen as a LAR with a special modification. The algorithm of LAR is roughly like this: Start with an empty model (except for an intercept). Then add the predictor variable that is the most correlated with $$y$$, say $$x_j$$. Incrementally change that predictor's coefficient $$\beta_j$$, until the residual $$y - c - x_j\beta_j$$ is equally correlated with $$x_j$$ and another predictor variable $$x_k$$. Then change the coefficients of both $$x_j$$ and $$x_k$$ until a third predictor $$x_l$$ is equally correlated with the residual $$y - c - x_j\beta_j -x_k\beta_k$$ and so on.  LASSO can be seen as LAR with the following twist: as soon as the coefficient of a predictor in your model (an "active" predictor) hits zero, drop that predictor from the model. This is what happens when you regress $$y$$ on the collinear predictors: both will get added to the model at the same time and, as their coefficients are changed, their correlation with the residuals remains the same, but one of the predictors will get dropped from the active set first because it hits zero first. As for which of the two collinear predictors it will be, I don't know. A source that explains these things more in detail is Chapter 3 in "The Elements of Statistical Learning" by Friedman, Hastie and Tibshirani. Post Deleted by Matthias Schmidtblaicher occurred Oct 20 '16 at 21:24 1 answered Oct 20 '16 at 21:22 Matthias Schmidtblaicher 1,17677 silver badges1818 bronze badges First of all, you are not fitting a LASSO but a Ridge regression (alpha = 0 would correspond to LASSO), i.e. you estimate the coefficient: $$\beta_{ridge} = (X'X + \lambda I)^{-1}X'y$$ So even though $$X'X$$ is singular, the added diagonal matrix makes the term in brackets invertible. Ridge will in general not set coefficients equal to zero, but shrink them all a bit, unless you have perfect collinearity. To see how it shrinks them, you can relate write its predicted value as: $$X\beta_{ridge} = U' \left[\begin{array}{cc} \frac{d_{1}^2}{d_{1}^2 + \lambda}& 0 \\ 0 & \frac{d_{k}^2}{d_{k}^2 + \lambda} \end{array}\right] U' y$$ Where the columns of $$U$$ spanning the column space of $$X$$ and the $$d_{j}^2$$ the eigenvalues of $$X'X$$. Since $$X'X$$ is singular in your case, the second eigenvalue is zero, so your system actually looks like this: $$X\beta_{ridge} = U' \left[\begin{array}{cc} \frac{d_{1}^2}{d_{1}^2 + \lambda}& 0 \\ 0 & 0 \end{array}\right] U' y$$ When I run your code, I also get that the second coefficient is of the order $$10^{-15}$$, so it is virtually zero. Interestingly, LASSO will not set one coefficient equal to zero but give positive values to both. To understand that, you should have a look at the relationship between Least Angle Regression and LASSO. A source that I recommend to understand these methods better is Chapter 3 in "The Elements of Statistical Learning" by Friedman, Hastie and Tibshirani.