How can a straight line (or plane) over-fit? My question is not about polynomial regression (although it too is considered 'linear'), but regarding a linear regression model with no higher order features, such as the following equation:

$$ y = \theta_0 + \theta_1 X_1 + \theta_2 X_2 + \theta_3 X_3 + \theta_4 X_4 + ... $$

I did go through other answered questions on this web which speak of poor generalization of a linear regression model with too many features (and that is in fact over fitting), but geometrically speaking, I cannot understand how can a linear model over-fit.

Here is Prof Andrew Ng's example of over-fitting shown geometrically. As far as I can see, a linear model (with no higher order features) can only under fit (the first figure depicting logistic regression):

enter image description here

Similar question: Overfitting a logistic regression model


Assuming the real model is: $$ y_i = \beta_0 + \beta_1 X_{i1} + \epsilon_i $$ but you add a factor $X_{2i}$ which not relate the $y_i$ to model and fit the new model $$ y_i = \beta_0 + \beta_1 X_{i1} + \beta_2 X_{i2} + \epsilon_i $$ In general, you will get a $\hat{\beta_2} \neq 0$, then if you run the model to predict something including factor $X_{i2}$, you will suffer over-fit.

  • $\begingroup$ What is $\hat{\beta}_2$? $\endgroup$ – Viktor Glombik Sep 1 '19 at 21:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.