It seems that the order of variable entry into a lasso model is significantly important! Is there any way that we make our model more meaningful? I mean is that OK to order the variables using another method (such as Stepwise) and give them to Lasso in that predefined order?
$\begingroup$
$\endgroup$
5
-
4$\begingroup$ Can you explain why you think that entry order is important? $\endgroup$– Glen_bCommented Nov 28, 2012 at 1:29
-
$\begingroup$ I used my data in two different formats (where the variables columns were the only difference), and I got different results. $\endgroup$– NioushaCommented Nov 28, 2012 at 1:39
-
3$\begingroup$ Can you give your data and code? This does not seem right. $\endgroup$– Peter FlomCommented Nov 28, 2012 at 1:53
-
3$\begingroup$ The LASSO criterion is $\lVert A x - y \rVert ^2 + \lambda \sum \lvert y_i \rvert$, which is clearly invariant to variable ordering. It's possible that something is randomized in the implementation you're using and you got two different non-optimal results, but something is going wrong here: like Peter said, please post the code. :) $\endgroup$– DanicaCommented Nov 28, 2012 at 4:51
-
$\begingroup$ Thanks so much for all your replies. I another try to see if my lasso function is working differently with two different inputs, but I figured that I'm actually getting the same results. Sorry for bothering:) $\endgroup$– NioushaCommented Nov 29, 2012 at 0:51
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
The order of the columns in the design matrix does not affect the lasso estimate $\hat \beta$. We can see this by noticing that, for $P$ a permutation matrix, \begin{align*} \min_\beta \|y-X\beta\|_2^2 + \lambda \|\beta\|_1 & = \min_\beta \|y - X P P^{-1} \beta\|_2^2 + \lambda \|P (P^{-1} \beta)\|_1 \\ & = \min_\alpha \|y - X P \alpha \|_2^2 + \lambda \|P \alpha\|_1 \\ & = \min_\alpha \|y - X P \alpha \|_2^2 + \lambda \|\alpha\|_1 \\ \end{align*} Therefore, mathematically, the ordering of the columns of $X$ does not influence the minimum of the LASSO optimization.
As the OP points out in the comments, they've made a mistake. Alternatively, such a result could have been the consequence of the algorithm used to compute the minimizer.
answered May 26, 2017 at 16:54
-
1$\begingroup$ +1. FWIW, you're implicitly relying on the (obvious) fact that $||\beta||_1$ is invariant under reordering, too. I'm not entirely sure there was a mistake: sometimes people get alarmed at tiny differences attributable to floating point error. It's possible that reordering $X$ causes the algorithm to proceed a little differently, which could produce tiny differences in the least significant figures of the results. When people tell us they got "different results," they really haven't told us anything... . $\endgroup$– whuber ♦Commented May 26, 2017 at 17:11
-
$\begingroup$ @whuber thanks for the comment! I've updated the answer to be more clear. (The last time a question reported a difference in estimators that shouldn't have been, I jumped on the algorithm giving the difference--only to later find out the OP had made a mistake, so I was more hesitant this time. Thanks for including that remark.) $\endgroup$ Commented May 26, 2017 at 17:41