13
$\begingroup$

How can a straight line (or plane) over-fit? My question is not about polynomial regression (although it too is considered 'linear'), but regarding a linear regression model with no higher order features, such as the following equation:

$$ y = \theta_0 + \theta_1 X_1 + \theta_2 X_2 + \theta_3 X_3 + \theta_4 X_4 + ... $$

I did go through other answered questions on this web which speak of poor generalization of a linear regression model with too many features (and that is in fact over fitting), but geometrically speaking, I cannot understand how can a linear model over-fit.

Here is Prof Andrew Ng's example of over-fitting shown geometrically. As far as I can see, a linear model (with no higher order features) can only under fit (the first figure depicting logistic regression):

enter image description here

Similar question: Overfitting a logistic regression model

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ I removed my accepted answer due to a helpful downvote to inspire a better answer, @gwg. But for someone may find a compact answer useful, I place it here as comment: Assuming the real model is: $y_i = \beta_0 + \beta_1 X_{i1} + \epsilon_i$ but you add a factor $X_{2i}$ which is not related the $y_i$ to model and fit the new model $y_i = \beta_0 + \beta_1 X_{i1} + \beta_2 X_{i2} + \epsilon_i$ In general, you will get a $\hat{\beta_2} \neq 0$ , then if you run the model to predict something including factor $X_{i2}$ , you will suffer over-fit. $\endgroup$
    – user137795
    Commented Jan 31, 2020 at 16:01
  • $\begingroup$ I posted a simulation I like over at the Data Science Stack: datascience.stackexchange.com/a/79994/73930. $\endgroup$
    – Dave
    Commented Dec 25, 2020 at 1:42

2 Answers 2

Reset to default
2
$\begingroup$

This is an old post but I just came across it. I think the question refers to how can a line become "curvy" when over estimation occurs. If we have 2 3D points we should be over fitting. The algorithm will try to fit a plane through 2 points. An infinite number of planes can go through 2 points. It can 'tilt' any way to fit a 3rd point. If we add another dimension and another point it will tilt towards it and so on. That's where the problem is. The 2D plots they use to illustrate the concept are an abstraction or refer to polynomial regression.

Improve this answer
$\endgroup$
4
$\begingroup$

Deleted, see question comment for content. (This post remains here because I can't delete an accepted answer, see https://meta.stackexchange.com/questions/14932/allow-author-of-accepted-answer-to-delete-it-in-certain-circumstances).

Improve this answer
$\endgroup$
9
  • $\begingroup$ What is $\hat{\beta}_2$? $\endgroup$
    – ViktorStein
    Commented Sep 1, 2019 at 21:07
  • $\begingroup$ Why is this true? Can you be more specific? I see upvotes, but no justification. $\endgroup$
    – jds
    Commented Jan 30, 2020 at 1:04
  • $\begingroup$ @gwg, I can't see what I can do to be more specific. Maybe something like this? Say we want "predict" NPC income (Y) in a simulation game (so we can define "true" model). The game includes IQ (X1) and leg length (X2) as factors. Since the leg length (X2) is exactly not related to income (Y) in that game, "true" $\beta_2=0$. But in general $\hat{\beta_2}$ will get non-zero value due to $\epsilon$. Thus if you use this model to predict $Y$, you will suffer from wrong $\beta_2$ configuration, it's exactly what over-fitting denotes, an wrong pattern (leg length affects income) fitted from data. $\endgroup$
    – user137795
    Commented Jan 30, 2020 at 1:48
  • $\begingroup$ Sure. I think you're describing shrinkage, which has to do with selecting features, reducing the variance of the estimator, and increasing predictive power. Wikipedia says, "This idea is complementary to overfitting and, separately, to the standard adjustment made in the coefficient of determination to compensate for the subjunctive effects of further sampling." $\endgroup$
    – jds
    Commented Jan 30, 2020 at 11:49
  • $\begingroup$ See Tibshirani's justification for the Lasso: math.yorku.ca/~hkj/Teaching/6621Winter2017/Coverage/lasso.pdf. He mentions (1) greater prediction accuracy because OLS estimates can have high variance and (2) interpretability. Also see Wikipedia's justification of Tikhonov regularization: en.wikipedia.org/wiki/Tikhonov_regularization. Neither frame the problem as overfitting as OP means it. Many people would agree with what you've written, but I think a good answer to OP's question would add a lot more nuance. $\endgroup$
    – jds
    Commented Jan 30, 2020 at 12:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.