0
$\begingroup$

Isn't it always better to make a number of linear regression models on different samples (sampled out using random sampling) of data set and then taking average of the regression coefficients for all variables?

The reason I am asking this question is that I have read somewhere that the regression coefficient of a variable is distributed normally with mean equal to the population parameter.Although , I am not very clear about this concept , so please correct me if I am wrong.

Also , if I am correct , is it necessary to make same type of model (having same features) each time I take a new sample to train with (so that the regression coefficients are distributed normally).

Improve this question
$\endgroup$
1
  • $\begingroup$ "Isn't it always better to..." Better in what sense? What sort of improvement do you have in mind? Also, better compared to what alternative? $\endgroup$
    – Sextus Empiricus
    Apr 20 at 17:32

1 Answer 1

Reset to default
0
$\begingroup$

I can spot two aspects in your question.

The first is on mixing theory and practical application. The second one is on ensemble modeling.

Mixing theory and practical application

If you are willing to impose many strict assumptions on your linear regression, indeed regression coefficients (seen as random variables) follow a multivariate normal distribution. This allows to calculate classic p values, standard errors and confidence intervals, which is often quite useful. Since these p values etc. are a result from mathematical statistical theory, in applications you don't have to run multiple linear regressions on multiple subsets by your own. You run one model and theory provides you with nice p values etc.

It is the same with the arithmetic mean: You calculate it for your random sample and thanks to the Central Limit Theorem, you know this mean is approximately normal and thus a confidence interval is immediately available without resampling.

Just as an exercise, you might split your sample into 20 parts, run a simple linear regression on each of the 20 parts and look at the distribution of the 20 slopes. It will look more like a Gaussian distribution than, say, a uniform distribution.

Ensemble modeling

You mention different ways on how to run multiple models (and then combining them). Both are typical elements of so called ensemble models. They are used as a way to reduce overfitting and eventually, increase true model performance on fresh data.

  1. Random sampling: If you draw rows with replacement, this is called bootstrap aggregating or bagging and is one of the key features of a random forest. Sampling without replacement is also common, e.g. with gradient boosting.

  2. Random subspace method: Each model has access only to a random subset of covariables. This is heavily used e.g. with gradient boosting methods.

Both element can be applied to linear regression as well, but it is very untypical. Why? Firstly, linear regressions are often very simple reflections of reality and as such do not overfit in a problematic way. Secondly, it is not too simple to derive classic p values from such ensemble model.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for this thorough answer! Talking about the first part (mixing theory) you said that we don't need to run different models since we have p-values ,but then p-value just tells us if the variable is meaningful (correct me if I am wrong). So,what I was thinking was to take average of coefficients coming from different subsets so that our estimate will become more accurate (since mean of the coefficients would be equal to true estimate). And if this is correct than is it necessary to train same type of linear model ,so that coefficients are normally distributed (my 2nd question). $\endgroup$
    – Ajinkya Dandvate
    Apr 25, 2020 at 4:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.