Timeline for Can a Linear Regression Model (with no higher order coefficients) over-fit?
Current License: CC BY-SA 4.0
13 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Dec 25, 2020 at 11:48 | comment | added | Sextus Empiricus | @user137795 I think your answer wasn't wrong. | |
| Jan 31, 2020 at 16:17 | history | edited | user137795 | CC BY-SA 4.0 |
added 132 characters in body
|
| Jan 31, 2020 at 16:02 | history | edited | user137795 | CC BY-SA 4.0 |
deleted 350 characters in body
|
| Jan 31, 2020 at 15:50 | comment | added | user137795 | @gwg No problem, I will remove this answer to inspire what you want. | |
| Jan 31, 2020 at 15:34 | comment | added | jds | Okay, you've convinced me. (Sadly, SO won't let me change my vote.) | |
| Jan 31, 2020 at 0:21 | comment | added | user137795 | @gwg You mention two regularization methods, so I suppose that you want an answer for this question like "Point out a non-overfitting version of OLS, thus the overfitting in vanilla OLS is obvious to see." While I agreed with it, I think it may be off topic. Overfitting in true model specification $Y=X\beta+\epsilon$, in essence, is due to noise $\epsilon$ as indicated by me, even under any regularization. Unless you specify a too strong regularization like "assign 0 to all parameter", in that case, you will get both zero overfitting and zero fitting if the true model is not a "zero model". | |
| Jan 30, 2020 at 12:10 | comment | added | jds | See Tibshirani's justification for the Lasso: math.yorku.ca/~hkj/Teaching/6621Winter2017/Coverage/lasso.pdf. He mentions (1) greater prediction accuracy because OLS estimates can have high variance and (2) interpretability. Also see Wikipedia's justification of Tikhonov regularization: en.wikipedia.org/wiki/Tikhonov_regularization. Neither frame the problem as overfitting as OP means it. Many people would agree with what you've written, but I think a good answer to OP's question would add a lot more nuance. | |
| Jan 30, 2020 at 11:49 | comment | added | jds | Sure. I think you're describing shrinkage, which has to do with selecting features, reducing the variance of the estimator, and increasing predictive power. Wikipedia says, "This idea is complementary to overfitting and, separately, to the standard adjustment made in the coefficient of determination to compensate for the subjunctive effects of further sampling." | |
| Jan 30, 2020 at 1:48 | comment | added | user137795 | @gwg, I can't see what I can do to be more specific. Maybe something like this? Say we want "predict" NPC income (Y) in a simulation game (so we can define "true" model). The game includes IQ (X1) and leg length (X2) as factors. Since the leg length (X2) is exactly not related to income (Y) in that game, "true" $\beta_2=0$. But in general $\hat{\beta_2}$ will get non-zero value due to $\epsilon$. Thus if you use this model to predict $Y$, you will suffer from wrong $\beta_2$ configuration, it's exactly what over-fitting denotes, an wrong pattern (leg length affects income) fitted from data. | |
| Jan 30, 2020 at 1:04 | comment | added | jds | Why is this true? Can you be more specific? I see upvotes, but no justification. | |
| Sep 1, 2019 at 21:07 | comment | added | ViktorStein | What is $\hat{\beta}_2$? | |
| Apr 11, 2017 at 15:23 | vote | accept | Batool | ||
| Dec 29, 2020 at 2:34 | |||||
| Apr 11, 2017 at 4:23 | history | answered | user137795 | CC BY-SA 3.0 |