0
$\begingroup$

The lasso and ridge regression are tuned to the same alpha parameter. No matter what I tune the parameter to [0,1], the results of all three regressions are always the same (linear, ridge, lasso), which is a bit befuddling because I had previously suspected that there was some multicollinearity in my data due to high variance inflation factors. I thought that either the lasso or ridge would address the issue of multicollinearity, but this does not seem to be the case. If anyone can provide some insight that would be great.

Thanks.

$\endgroup$

closed as unclear what you're asking by gung, Momo, Nick Cox, Andy, chl Sep 13 '14 at 10:38

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ Could you precise something about your data ? What is the dimension of the data ? What about the goodness of fit of the OLS regression ? $\endgroup$ – Stéphane Laurent Aug 21 '14 at 21:34
  • 3
    $\begingroup$ There's not enough information here to discern what's happening. A reproducible minimal example (i.e. the smallest example that shows the problem you describe) might throw light on what's happening. $\endgroup$ – Glen_b Aug 21 '14 at 23:49
2
$\begingroup$

It's hard to answer this question without more specific information, but it's important to understand that that if the parameter in ridge regression or LASSO is too small, then you'll get back very nearly the same answer as with linear regression.

For example, suppose that in ridge regression you're minimzing

$\min \| X \beta - y \|_{2}^{2} + \lambda \| \beta \|_{2}^{2}$

with $\lambda=1$, and that in the optimal solution, $\| \beta \|_{2}=10$, but $\| X \beta - y \|_{2}=10000000$. In this case, the ridge regression penalty term is so small as to be insignificant in comparison to the linear regression term and you'll end up with essentially the same answer with or without the penalty term.

So, in your examples, what is the size of $\| X \beta -y \|_{2}$? What about the ridge regression and LASSO penalty terms?

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.