# Dealing with results from thousands of regression models

I have a dataset containing 100 observations and 4 independent variables. These 100 observations can be categorised into 30 groups. In other words, for any given iteration, one observation per group is picked to end up with 30 observations with 4 independent variables. There are several possible combinations of 30 observations and, therefore, I built several regression models by randomly extracting 30 observations. I've been testing for several thousand iterations. From each model, I extracted the R², rmse and mae.

Furthermore, I need to modify the independent variables and repeat the iterations.

At this point, I have two sets of values containing the respective R², rmse and mae values. One before modifying the independent variables and one after. The only test I can think of is the t-test to compare the means of the R², rmse and mae values.

I have the following questions:

1. Do I have to worry about the significance of the independent variables in each iteration? As I randomly subset the data to arrive at 30 observations, there is going to be a variation in the significance of the independent variables.

2. Is there any test, other than the t-test, to compare all these thousands of R², rmse and mae values.

3. Is there something that can be inferred from the thousands of coefficients for the independent variables?

If you want to know better what the data looks like, I posted another question here about handling data in R.

Thankyou!

• I don't understand exactly the point of what you are trying to do. Regarding your second question, maybe you should look into bootstrapping to arrive at confidence intervals for your fit values. This would allow to test their differences for significance. I'm not sure how useful that would be, though. Jan 29, 2021 at 11:05

3. You will probably see that the accumulated $$beta$$ coefficients from sampled models will have a probability density quite close to Normal. The mean of each of the four coefficients will approach an expected value of the coefficient for the model built on 100% of data. The standard deviation of sampled coefficients will approach the 100%-model standard error in the coefficients.