12
$\begingroup$

Background

I am designing a Monte Carlo simulation that combines the outputs of series of models, and I want to be sure that the simulation will allow me to make reasonable claims about the probability of the simulated outcome and the precision of that probability estimate.

The simulation will find the probability that a jury drawn from a specified community will convict a certain defendant. These are the steps of the simulation:

  1. Using existing data, generate a logistic probability model (M) by regressing “juror first ballot vote” on demographic predictors.

  2. Use Monte Carlo methods to simulate 1,000 versions of M (i.e., 1000 versions of the coefficients for the model parameters).

  3. Select one of the 1,000 versions of the model (Mi).

  4. Empanel 1,000 juries by randomly selecting 1,000 sets of 12 “jurors” from a “community” (C) of individuals with specified demographic characteristic distributions.

  5. Deterministically calculate the probability of a first ballot guilty vote for each juror using Mi.

  6. Render each "juror’s" probable vote into a determinate vote (based on whether it is greater or less than randomly selected value between 0-1).

  7. Determine each "jury’s" “final vote” by using a model (derived from empirical data) of the probability a jury will convict, conditional on the proportion of jurors voting for conviction on the first ballot.

  8. Store the proportion of guilty verdicts for the 1000 juries (PGi).

  9. Repeat steps 3-8 for each of the 1,000 simulated versions of M.

  10. Calculate the mean value of PG and report that as the point estimate of the probability of conviction in C.

  11. Identify the 2.5 & 97.5 percentile values for PG and report that as 0.95 confidence interval.

I am currently using 1,000 jurors and 1,000 juries on the theory that 1,000 random draws from a probability distribution—demographic characteristics of C or versions of M—will fill out that distribution.

Questions

Will this allow me to accurately determine the precision of my estimate? If so, how many juries do I need to empanel for each PGi calculation to cover C's probability distribution (so I avoid selection bias); may I use fewer than 1,000?

Thank you so much for any help!

Improve this question
$\endgroup$
10
  • $\begingroup$ Just out of curiosity: is anything in this model conditional on whether the accused is actually guilty? $\endgroup$
    – whuber ♦
    Apr 6, 2011 at 20:51
  • $\begingroup$ The model is based off of survey responses to a single fact pattern, so actual guilt doesn't vary. I'm predicting how different juries would come out in a single contested case. $\endgroup$
    – Maggie
    Apr 6, 2011 at 22:36
  • $\begingroup$ OK, kidding aside, you report three estimates: the mean and 2.5 and 97.5 percentiles of PG. For which one(s) do you need an "accurate" determination and how accurate must it be? $\endgroup$
    – whuber ♦
    Apr 6, 2011 at 23:10
  • $\begingroup$ Also, step (6) is mysterious. Could you explain what it is intended to do? Is there a different "randomly selected value" for each juror(5), each jury(4), each model(3), or some combination thereof? $\endgroup$
    – whuber ♦
    Apr 6, 2011 at 23:16
  • 1
    $\begingroup$ (See above comment) I think I can reduce (a), the number of juries. Sampling error is a function of the number of samples. With 1,000 juries per model, I have a million samples total. Sampling error associated with 10^6 samples is ~0.1%. If I use only 35 juries per model, I'd have 3.5*10^4 samples, and ~0.5% sampling error. This sampling error is much smaller than my measurement error of ~5.0%. Therefore, I should be able to use 35 juries per model and just use the measurement error to estimate my confidence interval. $\endgroup$
    – Maggie
    Apr 9, 2011 at 2:34

2 Answers 2

Reset to default
4
+100
$\begingroup$

There is one general and "in-universe" criterion for goodness of Monte Carlo -- convergence.

Stick to one M and check how the PG behaves with the number of juries -- it should converge, so will show you a number of repetitions for which you will have a reasonable (for your application) number of significant digits. Repeat this benchmark for few other Ms to be sure you wasn't lucky with M selection, then proceed to the whole simulation.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ not sure anyone has been fully responsive to question. It has two parts: (1) Does the modeling strategy described supply a defensible solution to the problem she wants to solve—namely, what is the likelihood that a jury drawn randomly from a community, C, with specified demographic characteristics will vote to find a defendant guilty? And (2) If the modeling strategy is reasonable, how many “juries” must she select, and how many “verdicts” must she simulate for each, to report a defensible estimate of the prob of conviction & 0.95 CI? She wants to economize on computing. See her last comment $\endgroup$
    – dmk38
    Apr 9, 2011 at 23:29
0
$\begingroup$

It seems to me that the problem here is whether the model is too complex to look out without using Monte Carlo simulation.

If the model is all relatively simple then it should be possible to look at it through conventioanl statistics and derive a solution to the question being asked, without re-running the model multiple times. This is a bit of an over simplication, but if all your model did was produce points based on a normal distribution, then you could easily derive the sort of answers you are looking for. Of course, if the model is this simple then you are unlikely to need to do a Monte Carlo simulation find your answers.

If the problem is complex and it is not possible to break it down to more elementary, the Monte-Carlo is the right type of model to use, but I don't think there is any way of defining confidence limits without runing the model. Ultimately to get the type of confidence limits described he model would have to be run a number of times, a probability distribution would have to be fit to the outputs and from there the confidnce limits could be defined. One of the challenges with Monte-Carlo simulation is that models give good and regular answers for distributions in the mid range but the tails often give much more variable results, which ultimately means more runs to define the shape of the outputs at the 2.5% and 97.5% percentiles.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.