# Determine which method predicts best given results from a simulation exercise

This question is related to one I have already asked but the answer I got that suggests I should adopt a new tack to address my research question.

I repeat the substantive part of the original question to show the features of my particular computer simulation/experiment:

I conducted a computer-based assessment of different methods of fitting a particular type of model used in the palaeo sciences. I had a large-ish training set and so I randomly (stratified random sampling) set aside a test set. I fitted $m$ different methods to the training set samples and using the $m$ resulting models I predicted the response for the test set samples and computed a RMSEP over the samples in the test set. This is a single run.

I then repeated this process a large number of times, each time I chose a different training set by randomly sampling a new test set.

Having done this I want to investigate if any of the $m$ methods has better or worse RMSEP performance. I also would like to do multiple comparisons of the pair-wise methods.

Say I did 50 Runs so I have $50 \times m$ observations of the RMSEP. I wish to determine if all $m$ methods yield the same RMSEP (test a null of equal RMSEP for each method). Assuming a difference in RMSEP between models I would also like to know which models differ in the predictive performance.

Note that the reason I did 50 Runs was to avoid the issue of getting a particular result just because I chose that particular training set / test set combination. Computationally I can probably afford to do and order or magnitude more Runs quite easily if required.

You could use pairwise t-tests for each of the $\frac{m*(m-1)}{2}$ combinations of methods. I would also treat it as a paired test for each comparison, since each method operates on the same training data.

That is, if you had only 2 methods, you could just do a paired t-test on the two vectors of RMSEP results (all 50 of them). If you have more than two methods, you can do the same test on every pair of methods, and then adjust the p-values to account for the number of tests conducted.

In R, you could use:

pairwise.t.test(x,g,paired=TRUE)


Where $x$ is the vector of all $50*M$ RMSEP concatenated together, and $g$ is a grouping vector indicating, for every element of $x$ which method was used.

The output is an $M*M$ table of p-values, where the $i,j$th entry us the p-value (adjusted for multiple hypothesis testing) of methods $i,j$ having identical performance under a paired t-test.

FOLLOWUP: I think your comments make some good points, but I have to disagree with the claim that you cannot treat the pairs as being independent of one another in this case. You are drawing training-test splits from a distribution of possible splits, with replacement. This means the training-test splits should be i.i.d. random draws, so a paired t-test should be fine. Of course, this assumes that your initial data set is representative of what you want to study, and I'm not completely sure what the stratified sampling will do.

Another possibility: Treat the use bootstrapping to estimate the mean performance of each method, and compare the overlapping confidence intervals.

• Usual assumptions for paired t.test are dependence between members of the same pair but independence between pairs... – phaneron Oct 19 '12 at 19:39
• Thanks but I'm with @phaneron here; the reason I can't rely on the LME results (in the linked Q) is that I don't have independence between sets of RMSEP. – Gavin Simpson Oct 21 '12 at 20:37
• Was going to write a comment response, but it got long. Going to put in the question now. – John Doucette Oct 22 '12 at 7:27