Averaging the linear regression coefficients

Question

Isn't it always better to make a number of linear regression models on different samples (sampled out using random sampling) of data set and then taking average of the regression coefficients for all variables?

The reason I am asking this question is that I have read somewhere that the regression coefficient of a variable is distributed normally with mean equal to the population parameter.Although , I am not very clear about this concept , so please correct me if I am wrong.

Also , if I am correct , is it necessary to make same type of model (having same features) each time I take a new sample to train with (so that the regression coefficients are distributed normally).

Answer 1

I can spot two aspects in your question.

The first is on mixing theory and practical application. The second one is on ensemble modeling.

Mixing theory and practical application

If you are willing to impose many strict assumptions on your linear regression, indeed regression coefficients (seen as random variables) follow a multivariate normal distribution. This allows to calculate classic p values, standard errors and confidence intervals, which is often quite useful. Since these p values etc. are a result from mathematical statistical theory, in applications you don't have to run multiple linear regressions on multiple subsets by your own. You run one model and theory provides you with nice p values etc.

It is the same with the arithmetic mean: You calculate it for your random sample and thanks to the Central Limit Theorem, you know this mean is approximately normal and thus a confidence interval is immediately available without resampling.

Just as an exercise, you might split your sample into 20 parts, run a simple linear regression on each of the 20 parts and look at the distribution of the 20 slopes. It will look more like a Gaussian distribution than, say, a uniform distribution.

Ensemble modeling

You mention different ways on how to run multiple models (and then combining them). Both are typical elements of so called ensemble models. They are used as a way to reduce overfitting and eventually, increase true model performance on fresh data.

1. Random sampling: If you draw rows with replacement, this is called bootstrap aggregating or bagging and is one of the key features of a random forest. Sampling without replacement is also common, e.g. with gradient boosting.

2. Random subspace method: Each model has access only to a random subset of covariables. This is heavily used e.g. with gradient boosting methods.

Both element can be applied to linear regression as well, but it is very untypical. Why? Firstly, linear regressions are often very simple reflections of reality and as such do not overfit in a problematic way. Secondly, it is not too simple to derive classic p values from such ensemble model.