What is an intuitive way to think about why high variance in predictions is associated with overfitting of a model? I read that for linear models, when more variables are added to the regression, typically the bias of the predictions decreases and the variance increases. That is:

*

*Too few covariates yields high bias, known as underfitting.

while


*Too many covariates yields high variance, known as overfitting.

I am having a hard time intuitively understanding why too many covariates would yield high variance. Isn't it the case too many covariates sort of "pin" the estimates and would actually reduce the variance? And is there a way to also think about how the bias would decrease? Thanks.
 A: In this context, "variance" refers to the variance of the training data which is predicted by the model. You are exactly correct, more covariates will "pin" the estimates to real data points because there are more degrees of freedom for your model to manipulate to actually fit the data.
However, consider that including completely superfluous variables in a linear model will impact model predictions with limited data. In this case, your model will be "pinning" estimates to points in your input space where most of the dimensions are nonsense. In this case, the bias will be lower because the training data will fit better, the model will perform badly because it will overestimate the importance of these superfluous variables.
At the extreme limit, consider you have $N$ 1-D data points which is captured in a linear model with 1 variable with an $R^2$ of 0.95. Now, generate $N-1$ random numbers and append them to each of the $N$ data points such that these data points are now $N$-dimensional. Now, when you fit your linear regression model, your variance increases and bias decreases, in fact, you get $R^2= 1.00$! However, this is clearly an overfit model because it using random numbers to predict outcomes.
