I am trying to understand how to set up simulations to test the False Positive and False Negative Rates (and build a ROC curve) under a Bayesian model.
The results of the analyses are support for one Bayesian model over all others (non-exclusive) models considered, and the results are posterior probabilities. So for example we have posterior probabilities supporting model 1 vs. models 2,3,4,5,6,8.
I have simulated data under model 1, and now I want to find the False Positive and False Negative Rates under model 1 for a particular threshold. The true positives for this model are the number of simulations correctly identified as model 1, so number of simulations with posterior probability for model 1 > threshold. How do I then find the False negatives? I do not have a "true" negatives since I simulated all under model 1, so I am confused on how to go about defining these.
Do I need to add simulations for all the models? If I do that, then I would have to redefine my true positives as the number of simulations with posterior probability for model 1 > threshold OR model 2>threshold OR model 3>threshold and so on, while my False negatives would be the number of simulations under model 1 that are wrongly classified as model 2>threshold OR model 3>threshold and so on. Is this correct? Thank you for your help!