Combining coefficients and standard errors from repeated models

Question

In my current project I'm working with a previously developed linear model and asking how many training observations do I need to reproduce the model. The original model was developed with ~1000 observations, and I want to see how many I actually need to reproduce the original model parameters. To address this, I plan to sample the training data (keeping the response distribution representative of the training set) and develop new models for numerous repetitions - perhaps 100-1000 repeats. Each time the model is trained, I will use the same set of covariates used in the original model, and I would like to see how the coefficient estimate and standard error compare to the original, but, I'm unclear on how to combine the coefficients and standard errors from 1000 different models. The value for the coefficient could simply be the average or all trials, but how do I combine the standard errors for those same trials?

Let me clarify my thoughts on "reproduce the model parameters:"

What I'm trying to do is to determine how many observations I actually need from the training set to get the same, or close to the same, coefficients. The original model used 937 observations, but I would like to reduce that number. I realize that if I change the number of observations, that will lead to a slightly different set of coefficients with different standard errors from the original full training. The actual value of the coefficients and their standard errors will vary depending on the sample I take from my set of observations. In order to get a handle on the uncertainty, I will repeat my data sampling many times and aggregate the coefficients and their standard errors in order to compare them to the original.

For example, let's say I start with 100 observations. I'll pick 100 observations using a stratified sampling of the original data, train a new model using just those observations, and I'll have a different set of coefficients and standard errors. If I then repeat that process a second time, my coefficients and standard errors will be different due to the different subsample of observations. If I repeat that process 1000 times, I would like to aggregate the 1000 sets of coefficients and standard errors to see how well using 100 observations approximates the coefficients and standard errors I get when using the full dataset.

The question is, how do I combine 1000 sets of coefficients and standard errors in a meaningful way?

Answer 1

If you distrust your original 937 observations so much, you should be very cautious before you use those data for any purpose. I assume that you have thought that issue through and have nevertheless decided to use those untrustworthy observations to obtain a rough estimate of the number of re-tests you have to perform in order to get an adequately precise final model based on later, technically sound observations.

If you are keeping the same set of covariates in each of your new models based on subsets of the original observations, the fact that you have a linear model means that your work is essentially already done. Each coefficient standard error reported in your model with all 937 cases essentially contains a factor close to inversely proportional to the square root of the number of cases. If you analyze subsets with 100 cases each, your coefficients in the subset models will have about $\sqrt{\frac{937}{100}}$ higher standard errors, about a factor of 3.

You certainly can test this by doing 1000 models based on samples of 100 from the original data if you wish. Examine the standard deviations of the point estimates of each of the regression coefficients among the 1000 models. That tells you how much different models based on 100 cases each are likely to differ, which is what you apparently care about. The average value of each coefficient among the 1000 models should be close to the value from your original model.

I don't see a need to combine the standard errors of the models separately. If you do choose to do so, you should, for each regression coefficient, square the standard errors to obtain variances, average those variances among the 1000 models, and then take the square root of the average variance. The pooled standard error obtained that way should be pretty similar to the standard deviation of the point estimate obtained as in the previous paragraph.

Often when someone speaks of "training" a model there is some variable selection involved to obtain the final set of covariates. If that was the case in the development of your original model, you also need to consider the reproducibility of the covariate-selection process itself. In that case, keeping the same set of covariates for all of your sub-sample models might not be a good way to proceed.