0
$\begingroup$

I am trying to understand how to set up simulations to test the False Positive and False Negative Rates (and build a ROC curve) under a Bayesian model.

The results of the analyses are support for one Bayesian model over all others (non-exclusive) models considered, and the results are posterior probabilities. So for example we have posterior probabilities supporting model 1 vs. models 2,3,4,5,6,8.

I have simulated data under model 1, and now I want to find the False Positive and False Negative Rates under model 1 for a particular threshold. The true positives for this model are the number of simulations correctly identified as model 1, so number of simulations with posterior probability for model 1 > threshold. How do I then find the False negatives? I do not have a "true" negatives since I simulated all under model 1, so I am confused on how to go about defining these.

Do I need to add simulations for all the models? If I do that, then I would have to redefine my true positives as the number of simulations with posterior probability for model 1 > threshold OR model 2>threshold OR model 3>threshold and so on, while my False negatives would be the number of simulations under model 1 that are wrongly classified as model 2>threshold OR model 3>threshold and so on. Is this correct? Thank you for your help!

$\endgroup$
0
$\begingroup$

I think the way to look at this is to pick a reasonable alternative model. It seems that the best choice would be the one with the second highest posterior probability. You have a threshold to define type I error. The threshold set under the null hypothesis allows you to determine "true" negatives without necessarily checking all the alternative models. You will know which of the models 2, 3, 4, 5, 6, 8 has the next highest posterior probability. So do the simulation with that one.

$\endgroup$
  • $\begingroup$ Hi Michael, thank you for your help! To understand what you suggest step by step: I would choose for example model 3 as the alternative to model 1, do simulations under model 1 and another set of simulations under model 3, then analyze these simulations together and find the true negatives if model 3 > threshold, am I understanding this correctly? Or are you suggesting to just use the simulations under model 1? $\endgroup$ – user971102 Jan 12 '17 at 12:32
  • $\begingroup$ @user971102 That is exactly what I have in mind. Does it make sense? $\endgroup$ – Michael Chernick Jan 12 '17 at 12:36
  • $\begingroup$ Yes, and I had tried to do this only using the simulations under model 1 but the rates looked off (and so did the ROC curve), but I haven't tried this under both simulations of model 1 and model 3, I will try this right now! Thanks again! $\endgroup$ – user971102 Jan 12 '17 at 12:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.