I am running a chain of Monte Carlo simulations to estimate uncertainty in some model estimates (which are derived from a series of estimates). An important variables used in one of the steps is a sample proportion, which sometimes is near or at zero successes (e.g., 0 out of 20 fish were of hatchery-origin).

Here is a simplified example of my problem.

Simulate sample proportions and estimate confidence interval from the distribution

simulated_proportions = rbinom(n = 1000, size = 20, prob = 0) / 20 quantile(simulated_proportions, probs = c(0.5, 0.025, 0.975)) 50% 2.5% 97.5% 0 0 0

Compare simulated confidence intervals to Wilson-Score approximation

library(Hmisc) binconf(x = 0, n = 20, alpha = 0.05, method = "wilson") PointEst Lower Upper 0 0 0.1611252

My simulated data underestimate potential uncertainty in the sample proportion. It is important that uncertainty in this step is included in this step of the MC simulations.

Any advice on how to address this problem?

  • $\begingroup$ A coworker and myself have been banging our heads against our desks trying to figure this out today. We want to be able to capture the case where you made x observations, and didn't see any of what you're looking for, but there could be some of those. That's where rbinom breaks down, because a p=0 always gives 0. We think there might be some way to approximate it using the PropCIs package, and the scoreci function (which returns CI's based on the Wilson score) -- but haven't figured it out yet. Did you end up figuring out a way to do this? $\endgroup$
    – BDavis
    May 10 '19 at 23:12
  • $\begingroup$ @BDavis You appear to confuse the underlying probability with the outcome. The distinction is this: you can flip a coin with a chance $p\gt 0$ of landing heads $n$ times, yet it is possible (maybe even probable) that you will see no heads, whence the observed proportion is zero. This doesn't mean $p$ is zero! Please re-read Maarten Buis's answer. $\endgroup$
    – whuber
    May 11 '19 at 14:14
  • $\begingroup$ @BDavis - Yes, that is the problem I am running into. I have not found a solution. If you have any additional ideas, please share. $\endgroup$
    – RFish
    May 14 '19 at 22:12

People have quite a specific idea of what a proportion is for these types of techniques: they imagine a given number of trials $n$, and the proportion is the number of "successes" divided by the number of trials. This means that there is only a limited number of proportions possible. For example: if you have 3 trials, then there are only 4 proportions possible: 0, 1/3, 2/3, 1. This discreteness of the parameter space makes the confidence interval for proportions such a well known hard problem.

Rbinom is thus not biased, it accurately represents the data generating mechanism often used for proportions. You seem to expect a continuous parameter space for a proportion. If you want to keep that, you will first need to think of a data generating mechanism that would result in such a continuous parameter space.

  • $\begingroup$ Thanks! I agree completely. Do you have any recommendations for a data generating mechanism? $\endgroup$
    – RFish
    May 14 '19 at 22:14
  • $\begingroup$ That is the wrong way around. You start with a real problem, this implies a data generating mechanism, which you then implement in a simulation. Now you have a simulation, and are looking for a problem. Don't start looking for problems, there are enough out there already... $\endgroup$ May 15 '19 at 7:48
  • $\begingroup$ I rephrased the question and problem. Maybe that will add some clarity. $\endgroup$
    – RFish
    May 16 '19 at 16:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.