How can a straight line (or plane) over-fit? My question is not about polynomial regression (although it too is considered 'linear'), but regarding a linear regression model with no higher order features, such as the following equation:
$$ y = \theta_0 + \theta_1 X_1 + \theta_2 X_2 + \theta_3 X_3 + \theta_4 X_4 + ... $$
I did go through other answered questions on this web which speak of poor generalization of a linear regression model with too many features (and that is in fact over fitting), but geometrically speaking, I cannot understand how can a linear model over-fit.
Here is Prof Andrew Ng's example of over-fitting shown geometrically. As far as I can see, a linear model (with no higher order features) can only under fit (the first figure depicting logistic regression):
Similar question: Overfitting a logistic regression model