I have 50 values set as "true" in a simulation and attempts to recover these "true" parameters using 2 different models. Here's 3 (instead of 50) runs just to show the data layout (these are made up for illustration).

Run ,  "true" parameter ,  estimate model 1  , estimate model 2
1   ,       10          ,      9.5           ,      9.6
2   ,        8          ,      7.5           ,      8.1
3   ,        7          ,      7.1          ,       7.2

For each of the two models, I have calculated root mean squared error (error is difference between "truth" and estimate), mean absolute error, and Pearson correlation (correlation between 50 "truth" measures and 50 estimates). For each accuracy measure, I would like to compare the performance between the two models. Normality assumptions are not satisfied, so I would like to use bootstrap methods.

Since the data is paired, I've thought I could resample by runs (with replacement, 50 draws, and as many replications as feasible). I would then calculate the statistic of interest for each model and save the ratio of these for each run (RMSE model 1/RMSE model 2 for instance) and then determine confidence intervals using a percentile (or other) method.

I would also like to use a direct hypothesis testing approach via resampling. For this, I would shuffle the model outputs within pairs (so the estimated values for model 1 and model 2 would be switched, for instance) at random and then calculate my various ratios as before. This would give a range expected given a null hypothesis of interchangeability to compare to my observed values in order to attain a p-value.

I have ordered Efron and Tibshirani's Intro to the Bootstrap and will read parts that apply, but I have not found much literature on what I would like to attempt. Perhaps I am looking in the wrong place or it is an obvious dead end.

I am open to any advice or obvious flaws in my approach.

Thank you

  • $\begingroup$ This is my first post. Sorry that I did not format the toy example so that it would not be "destroyed". I think clarity is still retained. $\endgroup$
    – wvguy8258
    Aug 4, 2011 at 3:01
  • $\begingroup$ Formatting is easy to do using the tools just above the edit textbox. $\endgroup$
    – whuber
    Aug 4, 2011 at 3:33
  • 2
    $\begingroup$ Sounds good to me. For your second approach (permutation test), a standard reference seems to be Good, PI (2005) Permutation, Parametric, and Bootstrap Tests of Hypotheses. Is there a specific reason for choosing the ratio of your performance measures as a test statistic instead of their difference? WRT boostrap CIs: BCa intervals are supposed to be more accurate when the bootstrap-distribution exhibits skew and bias. $\endgroup$
    – caracal
    Aug 4, 2011 at 18:48
  • $\begingroup$ Thanks. I was perhaps following the shadow of the F-test when thinking to use ratios, is using differences defensible? $\endgroup$
    – wvguy8258
    Aug 4, 2011 at 21:13
  • $\begingroup$ That's why I was asking: writing the ANOVA or regression F-ratio from the model-comparison point of view, the numerator is the difference in RSS of the restricted and the full model (divided by the difference in their df). I.e., the comparison between two models is done by taking the difference of a performance measure that is almost your RMSE. Of course you don't have two nested models, but I think the idea is natural to carry over. Perhaps not to correlations though, these are normally compared using Fisher's Z-transormation. $\endgroup$
    – caracal
    Aug 4, 2011 at 21:38

1 Answer 1


Your permutation test is correct for testing if there is a difference in the quality of the 2 models. I don't understand enough about your bootstrap approach to know if it is correct or not. Another book to consider is "Bootstrap Methods and their Application" by Davison and Hinkley. It is a bit more recent and I believe more applied than Efron and Tibshirani.

  • $\begingroup$ Thanks. For permutation test, I would be respecting the paired nature of the data. So, I would switch values within runs, between models. So, in run 1 I may switch values so that model estimate 1 would then be 9.6 (not 9.5). I hope that was clear and the correct approach for a paired randomization/permutation test. $\endgroup$
    – wvguy8258
    Aug 4, 2011 at 21:02
  • $\begingroup$ As far as the bootstrap approach, it is an extension of selection with replacement to get a distribution for a test statistic. In the one sample approach, I would just select from model 1 estimates and calculate a summary stat for each draw. Since I am interested in the difference between the summary stats, and the data is paired, I would select runs at random. I would then summarize the data separately for each model among the selected runs. I would then calculate the ratio or difference between these two summarized samples. No shuffling of the values between the two models would occur. $\endgroup$
    – wvguy8258
    Aug 4, 2011 at 21:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.