# Can I remove variance of OLS estimators through averaging coefficients from cross-validation?

Imagine working on a linear multiple regression problem with a design-matrix $$\Phi$$ based upon some independent variables $$x_k, k\in{}[1, r]$$. The goal is to find an equation that explains the "true" relationship between a dependent variable $$y$$ and independent variables $$x_k, k\in{}[1, r]$$ e.g. of a recorded physical experiment. It is defined as $$$$\bar{y}=\Phi\theta$$$$ with $$\theta$$ being the regression coefficients and $$\bar{y}$$ being predictions of the estimator. Why do I need penalized regression or subset-selection?

From what I have read, additional unnecessary columns in $$\Phi$$ create more variance in the estimators parameters $$\theta$$ but no bias (p.94, Kennedy: A guide to econometrics). (I assume this is because any correlation with relevant variables will result in some non-zero coefficient for that term. Coefficients could also act in opposing directions (such as with sin(x) and x at low x values). Is that correct? If you could help me understand this with the covariance matrix, that would be great.)

Why am I not able to remove this variance by using cross-validation and by averaging coefficients of the best sets? Isn't their mean free of bias? I read about this approach in (Brunton: DATA DRIVEN SCIENCE & ENGINEERING. http://databookuw.com/databook.pdf) but could not find it anywhere else. It also suggests thresholding small coefficients.

I read https://stats.stackexchange.com/questions/472202/when-to-use-regularization-vs-cross-validation#:~:text=Cross%20validation%20is%20about%20choosing,to%2C%20result%20in%20similar%20solutions and understand that cross-validation does not do the same thing as regularization. But shouldn't it converge to the true parameters for infinite data?

If you know a good piece of literature covering this, please share it. I am a mechanical engineering student and this topic is way above anything we have ever done in classes. Thanks in advance!

• Could you explain how one could ever get started doing a regression analysis with a "completely unknown" matrix of explanatory variables??
– whuber
Mar 31 at 17:06
• @whuber I meant that their transformations are unknown. It does not really contribute to the question, so I removed it. It could consist out of many possible transformation, such as trigonometric functions, polynomiales, ..., that could be part of the equation.
– Timo
Mar 31 at 17:15