Timeline for Is ridge regression useless in high dimensions ($n \ll p$)? How can OLS fail to overfit?
Current License: CC BY-SA 3.0
16 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Feb 15, 2018 at 22:09 | comment | added | amoeba | After the edits it mostly makes sense to me, but unfortunately I have to say that I still find the whole phenomenon as mysterious as before... In the end of the answer you show that lambda=0 with p=100 overfits (large erratic coefficients) but with p=1000 does not overfit (small coefficients that do not change over a large range of lambdas), but the question remains: WHY EXACTLY is this happening? As I said, the "common wisdom" is that p>n always yields bad overfitting... and this seems to be entirely wrong! | |
| Feb 15, 2018 at 21:15 | comment | added | Firebug | @amoeba Alright, I added some more edits, TLDR: I think the regularization path makes some difference in model coefficients (code now reflect this). This is starting to creep outside my expertise, though, so perhaps you could pick up from here and add an improved, and perhaps definitive, answer? | |
| Feb 15, 2018 at 21:14 | history | edited | Firebug | CC BY-SA 3.0 |
added 110 characters in body
|
| Feb 15, 2018 at 20:26 | comment | added | Firebug | @amoeba about the inner CV, you're correct, that's the reason there's none for that model, it runs the outer CV only. | |
| Feb 15, 2018 at 20:25 | comment | added | Firebug |
@amoeba Hmm, I've ran the cv.glmnet with different lambda sequences and it does make a difference on the result (that's likely the reason for the 0.7 figure). Now, we know glmnet is based on the regularization path problem, so perhaps it's due to numerical rounding at such extreme regularization parameters.
|
|
| Feb 15, 2018 at 20:05 | comment | added | amoeba | Thanks. To be honest I don't understand the output of your 1e-100 benchmark. E.g. for p=1100 it gives MSE=1.45. But here there is no hyperparameter tuning in the inner loop so basically one does not need inner CV loop at all. Meaning that the result should be the same as with non-nested CV at lambda=1e-100. But we see on the first figure that the MSE there is around 0.7. It does not make sense to me. | |
| Feb 15, 2018 at 18:42 | comment | added | Firebug |
@amoeba lambda.min. There's also a regr.cvglmnet learner, which probably allows one to select other rules.
|
|
| Feb 15, 2018 at 18:34 | comment | added | amoeba |
Do you know if mlr selects lambda.min or lambda.1se (in the cv.glmnet terminology)?
|
|
| Feb 15, 2018 at 17:15 | history | edited | Firebug | CC BY-SA 3.0 |
added 4 characters in body
|
| Feb 15, 2018 at 17:08 | history | edited | Firebug | CC BY-SA 3.0 |
added 4 characters in body
|
| Feb 15, 2018 at 16:53 | history | edited | Firebug | CC BY-SA 3.0 |
added 4 characters in body
|
| Feb 15, 2018 at 16:52 | comment | added | amoeba |
Also, I don't quite understand your last line. Look at the cv.glmnet output for p=100. It will have very different shape. So what affects this shape (asymptote on the left vs. no asymptote) when p=100 or p=1000?
|
|
| Feb 15, 2018 at 16:51 | comment | added | amoeba | +1 Thanks for doing these experiments! It seems you are getting a tiny minimum for some non-zero lambda (I am looking at your figure), but the curve is still really really flat to the left of it. So my main question remains as to why $\lambda\to 0$ does not noticeably overfit. I don't see an answer here yet. Do you expect this to be a general phenomenon? I.e. for any data with $n\ll p$, lambda=0 will perform [almost] as good as optimal lambda? Or is it something special about these data? If you look above in the comments, you'll see that many people did not even believe me that it's possible. | |
| Feb 15, 2018 at 16:51 | history | edited | Firebug | CC BY-SA 3.0 |
added 376 characters in body
|
| Feb 15, 2018 at 16:37 | history | edited | Firebug | CC BY-SA 3.0 |
deleted 6 characters in body
|
| Feb 15, 2018 at 16:29 | history | answered | Firebug | CC BY-SA 3.0 |