4
$\begingroup$

I have two linear regression models (with the same predictors) that try to estimate two different (although related) features of the same population. I am analyzing the hypothesis that these predictors are not as good for the second feature as they are for the first. Indeed, the RMSE of the second model is 7% worse than the RMSE of the first: it is not a huge difference, but for my purposes it is fine.

The problem is: I have been asked the statistical significance of this result (the 7% worse).

I am a computer scientist, not a statistics expert, and I am at a loss here on which test I should use, given that I am trying to compare the squared errors of the two (paired) sets of predictions (I am using SPSS software).

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Is the comparison of the RMSE even meaningful? There may be some cases where it is, but often it will not be. If one of the measures were heights of people then you could easily change the RMSE by changing the units of measurement, i.e. the RMSE would be higher for millimeters than for inches and lower for feet or meters than for inches. So saying that one RMSE is 7% worse could be reflecting units and a change of one of the units would make that RMSE better instead of worse (or much worse).

If the responses are on a scale where the comparison is meaningful, then your best chance is probably doing a simulation (aka parametric bootstrap). Simulate data that represents your analysis (same sample size, same effects of predictor variables) but with the condition that the 2 RMSEs are the same (on a population/simulation distribution level), then compare the 2 estimated RMSEs. Repeat this a bunch of times (in the thousands) and see how often you see a 7% or greater difference. If you often see 7% or more differences in the simulations, then your original difference is easily explained by chance (not statistically significant), but if you only rarely (less than alpha) see anything that extreme, then that shows that chance is not the likely explanation (statistically significant).

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ An alternative: do a nonparametric bootstrap. Resample from your original data, fit the two models for the two DVs, record the difference in RMSE, do this many times. See where in this distribution the original RMSE falls. - @GregSnow: thanks for your comment on my "answer", where I indeed misunderstood the question - I deleted it, it was misleading as it stood. $\endgroup$
    – Stephan Kolassa
    Mar 28, 2014 at 19:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.