I am using ROC curves to compare different methods but not sure if I need to re-simulate datasets using different seeds in R in order to reduce the "by-chance" issue for a particular output. Here is a brief outline of my simulation:
generate.datais used to simulate data of some distribution, and by simulation, I know which data are true positives. The random number generator is controlled by fixing the
check.modelsis used to test a total of 5 methods, and return the quantities used to draw a ROC curve for each method. Also for each curve (method), the AUC is reported.
plot.rocis used for plotting.
In step #1, there are some other factors to change so that the data are under different "alternatives". When I run steps #1 and #2 above using
seed=123 and pick up the method with the highest AUC, I got one set of results. However, when I re-run using a different seed (say
seed=456), I got another set of results not identical to the first run. Therefore, I think rigorously I should run my simulation across different
seed's in R to generate data in step #1, so that the "by-chance" issue of using a particular dataset is reduced.
Am I correct? If so, then I should report the average of the AUC's for each method across (say, 1000) simulations, and pick up the highest among the methods compared? Thanks!