2
$\begingroup$

I am writing a paper that develops a new model, A. I want to compare this model to previous models, B, C, D, E and F.

I have a performance metric for each model for each of 100 randomly generated simulation datasets.

My hypothesis is that A is better than B, C, D, E and/or F. Therefore, I thought that a one-tailed t-test would be appropriate, testing A > B, A > C, etc. However, two reviewers suggested that such a test may not be appropriate.

I originally also considered an ANOVA with post-hoc test. However, this post: https://www.researchgate.net/post/Which_post_hoc_test_is_best

Post-hoc is a suitable for data collected on "fishing expeditions" experiments or observations where no clear hypothesis is clear beforehand regarding which treatments should be compared, so then you must compare all terms with each other. "Run the flag and see who salutes!"

Suggested that post hoc tests are meant for "fishing expeditions" whereas my hypothesis is more specific.

So, which test is most appropriate? Should I use an ANOVA post hoc, or something else?

$\endgroup$
  • $\begingroup$ If you are simulating data sets, why use only 100? Why not 10,000? Then any comparison you perform will be essentially comparing the population values for each outcome, meaning you don't have to do significance tests. BTW, you can look into the literature on meta-models in simulations. $\endgroup$ – Noah Aug 30 '17 at 21:51
0
$\begingroup$

I would be inclined to use the $R^2$ statistics to assess the amount of explained variation and/or the RMSE to assess the size of the error in the model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.