# Bootstrap and randomization tests to compare paired data sets

I have 50 values set as "true" in a simulation and attempts to recover these "true" parameters using 2 different models. Here's 3 (instead of 50) runs just to show the data layout (these are made up for illustration).

Run ,  "true" parameter ,  estimate model 1  , estimate model 2
1   ,       10          ,      9.5           ,      9.6
2   ,        8          ,      7.5           ,      8.1
3   ,        7          ,      7.1          ,       7.2


For each of the two models, I have calculated root mean squared error (error is difference between "truth" and estimate), mean absolute error, and Pearson correlation (correlation between 50 "truth" measures and 50 estimates). For each accuracy measure, I would like to compare the performance between the two models. Normality assumptions are not satisfied, so I would like to use bootstrap methods.

Since the data is paired, I've thought I could resample by runs (with replacement, 50 draws, and as many replications as feasible). I would then calculate the statistic of interest for each model and save the ratio of these for each run (RMSE model 1/RMSE model 2 for instance) and then determine confidence intervals using a percentile (or other) method.

I would also like to use a direct hypothesis testing approach via resampling. For this, I would shuffle the model outputs within pairs (so the estimated values for model 1 and model 2 would be switched, for instance) at random and then calculate my various ratios as before. This would give a range expected given a null hypothesis of interchangeability to compare to my observed values in order to attain a p-value.

I have ordered Efron and Tibshirani's Intro to the Bootstrap and will read parts that apply, but I have not found much literature on what I would like to attempt. Perhaps I am looking in the wrong place or it is an obvious dead end.

I am open to any advice or obvious flaws in my approach.

Thank you

-
This is my first post. Sorry that I did not format the toy example so that it would not be "destroyed". I think clarity is still retained. –  wvguy8258 Aug 4 '11 at 3:01
Formatting is easy to do using the tools just above the edit textbox. –  whuber Aug 4 '11 at 3:33
Sounds good to me. For your second approach (permutation test), a standard reference seems to be Good, PI (2005) Permutation, Parametric, and Bootstrap Tests of Hypotheses. Is there a specific reason for choosing the ratio of your performance measures as a test statistic instead of their difference? WRT boostrap CIs: BCa intervals are supposed to be more accurate when the bootstrap-distribution exhibits skew and bias. –  caracal Aug 4 '11 at 18:48
Thanks. I was perhaps following the shadow of the F-test when thinking to use ratios, is using differences defensible? –  wvguy8258 Aug 4 '11 at 21:13
That's why I was asking: writing the ANOVA or regression F-ratio from the model-comparison point of view, the numerator is the difference in RSS of the restricted and the full model (divided by the difference in their df). I.e., the comparison between two models is done by taking the difference of a performance measure that is almost your RMSE. Of course you don't have two nested models, but I think the idea is natural to carry over. Perhaps not to correlations though, these are normally compared using Fisher's Z-transormation. –  caracal Aug 4 '11 at 21:38
show 1 more comment