# Is there any theoretical problem with averaging regression coefficients to build a model?

I want to build a regression model that is an average of multiple OLS models, each based on a subset of the full data. The idea behind this is based on this paper. I create k folds and build k OLS models, each on data without one of the folds. I then average the regression coefficients to get the final model.

This strikes me as similar to something like random forest regression, in which multiple regression trees are built and averaged. However, performance of the averaged OLS model seems worse than simply building one OLS model on the entire data. My question is: is there a theoretical reason why averaging multiple OLS models is wrong or undesirable? Can we expect averaging multiple OLS models to reduce overfitting? Below is an R example.

#Load and prepare data
library(MASS)
data(Boston)
trn <- Boston[1:400,]
tst <- Boston[401:nrow(Boston),]

#Create function to build k averaging OLS model
lmave <- function(formula, data, k, ...){
lmall <- lm(formula, data, ...)
folds <- cut(seq(1, nrow(data)), breaks=k, labels=FALSE)
for(i in 1:k){
tstIdx <- which(folds==i, arr.ind = TRUE)
tst <- data[tstIdx, ]
trn <- data[-tstIdx, ]
assign(paste0('lm', i), lm(formula, data = trn, ...))
}

coefs <- data.frame(lm1=numeric(length(lm1$coefficients))) for(i in 1:k){ coefs[, paste0('lm', i)] <- get(paste0('lm', i))$coefficients
}
lmnames <- names(lmall$coefficients) lmall$coefficients <- rowMeans(coefs)
names(lmall$coefficients) <- lmnames lmall$fitted.values <- predict(lmall, data)
target <- trimws(gsub('~.*$', '', formula)) lmall$residuals <- data[, target] - lmall$fitted.values return(lmall) } #Build OLS model on all trn data olsfit <- lm(medv ~ ., data=trn) #Build model averaging five OLS olsavefit <- lmave('medv ~ .', data=trn, k=5) #Build random forest model library(randomForest) set.seed(10) rffit <- randomForest(medv ~ ., data=trn) #Get RMSE of predicted fits on tst library(Metrics) rmse(tst$medv, predict(olsfit, tst))
[1] 6.155792
rmse(tst$medv, predict(olsavefit, tst)) [1] 7.661 ##Performs worse than olsfit and rffit rmse(tst$medv, predict(rffit, tst))
[1] 4.259403

• You might try using the median coefficient values, rather than the average coefficient values. I have seen that this technique can sometimes yield better results. Commented Jul 1, 2018 at 23:01
• It will probably not give you a performance boost or reduce over fitting, but it does have other useful applications. This chap uses to select the correct trend for his streamed time series data youtube.com/watch?v=0zpg9ODE6Ww&index=64&list=WL
– josh
Commented Jul 2, 2018 at 8:22
• See also this post, stats.stackexchange.com/q/68030/28746 Commented Jul 2, 2018 at 12:57
• Here is a new way in regression analysis. Average regression paper, by Dr Abdul Rahim Wong Link data.mendeley.com/datasets/ytz7pyg8y9/1 DOI: 10.17632/ytz7pyg8y9.1 Commented Oct 30, 2023 at 9:01

Given that OLS minimizes the MSE of the residuals amongst all unbiased linear estimators (by the Gauss-Markov theorem) , and that a weighted average of unbiased linear estimators (e.g., the estimated linear functions from each of your $k$ folds) is itself an unbiased linear estimator, it must be that OLS applied to the entire data set will outperform the weighted average of the $k$ linear regressions unless, by chance, the two give identical results.