This question is related to one I have already asked but the answer I got that suggests I should adopt a new tack to address my research question.
I repeat the substantive part of the original question to show the features of my particular computer simulation/experiment:
I conducted a computer-based assessment of different methods of fitting a particular type of model used in the palaeo sciences. I had a large-ish training set and so I randomly (stratified random sampling) set aside a test set. I fitted $m$ different methods to the training set samples and using the $m$ resulting models I predicted the response for the test set samples and computed a RMSEP over the samples in the test set. This is a single run.
I then repeated this process a large number of times, each time I chose a different training set by randomly sampling a new test set.
Having done this I want to investigate if any of the $m$ methods has better or worse RMSEP performance. I also would like to do multiple comparisons of the pair-wise methods.
Say I did 50 Runs so I have $50 \times m$ observations of the RMSEP. I wish to determine if all $m$ methods yield the same RMSEP (test a null of equal RMSEP for each method). Assuming a difference in RMSEP between models I would also like to know which models differ in the predictive performance.
How might I go about addressing this question statistically?
Note that the reason I did 50 Runs was to avoid the issue of getting a particular result just because I chose that particular training set / test set combination. Computationally I can probably afford to do and order or magnitude more Runs quite easily if required.