I have a dataset containing 100 observations and 4 independent variables. These 100 observations can be categorised into 30 groups. In other words, for any given iteration, one observation per group is picked to end up with 30 observations with 4 independent variables. There are several possible combinations of 30 observations and, therefore, I built several regression models by randomly extracting 30 observations. I've been testing for several thousand iterations. From each model, I extracted the R², rmse and mae.

Furthermore, I need to modify the independent variables and repeat the iterations.

At this point, I have two sets of values containing the respective R², rmse and mae values. One before modifying the independent variables and one after. The only test I can think of is the t-test to compare the means of the R², rmse and mae values.

I have the following questions:

  1. Do I have to worry about the significance of the independent variables in each iteration? As I randomly subset the data to arrive at 30 observations, there is going to be a variation in the significance of the independent variables.

  2. Is there any test, other than the t-test, to compare all these thousands of R², rmse and mae values.

  3. Is there something that can be inferred from the thousands of coefficients for the independent variables?

If you want to know better what the data looks like, I posted another question here about handling data in R.


  • $\begingroup$ I don't understand exactly the point of what you are trying to do. Regarding your second question, maybe you should look into bootstrapping to arrive at confidence intervals for your fit values. This would allow to test their differences for significance. I'm not sure how useful that would be, though. $\endgroup$
    – Max J.
    Jan 29, 2021 at 11:05

1 Answer 1

  1. You make something similar to bootstraping to build models, except for in the boostraping one samples with replacements, which results in not exactly independent samples. Of course, you can expect that some of the models will be less significant than others due to sampling effects. Moreover, depending on your dataset, a majority of the models can be insignificant / poorly formulated as OLS models in terms of the requiremets. It is a quite much broader topic.

  2. In my opinion you can use T-test for two samples to check for the absence of the significant difference between the sample means. One caveat that I see here is that each model's output should be independent so that your data are independent, which is a general requirement for making hypothesis checking. I think this point could be better explained by other members.

  3. You will probably see that the accumulated $beta$ coefficients from sampled models will have a probability density quite close to Normal. The mean of each of the four coefficients will approach an expected value of the coefficient for the model built on 100% of data. The standard deviation of sampled coefficients will approach the 100%-model standard error in the coefficients.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.