1
$\begingroup$

I have a long series of entities:

x1, x2, x3, ... xn

for each of which, there is a probability of an event occurring. The probability for each x may be different, but they are all independent, and they are all known:

p1, p2, p3, ... pn

I might provide a point-estimate for how many of said event occurred in that series by simply summing the probabilities p, but is it possible to also calculate a confidence interval or some information about the error in a close-form fashion?

The resistance to using some sampling method is that the series of x many be prohibitively long, and there may be many such series upon which the same estimates need be delivered.


EDIT - Refactoring of the question

For a toy example of the problem, imagine I have a large batch of n widgets, each of which has a known probability of failing within the year. Each widget may have a different known probability of failing, some high, some low (say they belong to different widget classes, those classes well-studied in the past). I'm tasked with estimating the number of widgets that will fail at the end of the year for that batch.

I believe I can provide a point estimate by simply summing the probabilities of each widget in the batch. Stakeholders also request a confidence interval, which may not be possible, but I should be able to provide a sense of error/uncertainty/variance around that point estimate.

My understanding is that I can treat my widgets failing/not-failing as a series of Bernoulli trials, and that likely the "number that fail" can be modelled as a Poisson Binomial Distribution, but from there I get stuck.

$\endgroup$
13
  • 1
    $\begingroup$ This doesn't read like a confidence interval question: it looks more like you want to compute a probability. But the vague senses of "entity" and "event" preclude anything more than a guess concerning what you are trying to ask. Could you perhaps give a small concrete example? $\endgroup$
    – whuber
    Commented Mar 12, 2022 at 3:23
  • $\begingroup$ @whuber An analogous case -- Imagine I made a billion widgets in a batch, and each has a known probability of breaking at the end of some period. I'd like to estimate the total number from that batch that will break and give some sense of the error in my estimate (ideally a confidence interval). $\endgroup$
    – Josh
    Commented Mar 12, 2022 at 6:00
  • $\begingroup$ That's a probability. Confidence intervals are guesses based on data, whereas you assert all this information is known--there's no guessing involved. However, the second part of your question appears to contradict this: if all the relevant information is known, why are you even considering sampling? $\endgroup$
    – whuber
    Commented Mar 12, 2022 at 14:07
  • $\begingroup$ @whuber I'm afraid I'm not following your conclusion. Sure, you could state that the final answer of (say) 800M widgets out of 1B widgets is a probability by virtue of its fractional nature, but that is hiding a lot of the story (and possible variance!). Imagine running monte carlo on all those different probabilities to get the estimate - while it would converge to the same, you could also get a sense of the variance. $\endgroup$
    – Josh
    Commented Mar 12, 2022 at 14:20
  • 1
    $\begingroup$ @whuber I've refactored the question as best I know how -- my own ignorance limiting further ability. The question linked is helpful though, and I think the Poisson Binomial Distribution sounds like the right direction, but I've got more digesting to do to get there. $\endgroup$
    – Josh
    Commented Mar 14, 2022 at 22:17

2 Answers 2

1
$\begingroup$

I wanted to sum up what ultimately I've found works well for this case and roll up what's been discussed in some of the comments.

Since I am basically wanting to know how many events occurred out of series of independent Bernoulli trials, we can model this as a Poisson Binomial distribution.

The mean of that distribution (which is my estimate) is the sum of all the probabilities.

The variance of that distribution is also known: the sum of all (1 - pi)pi.

That already gives me the standard deviation (the square root of the variance), so I've gotten a little bit of what the stakeholder wants.

We know that often the Poisson Binomial approaches normal with larger number of parameters (constituent probabilities from the trials), and luckily because of the nature of my problem, I will almost always have a large number of parameters. When that's the case, we can get an interval by simply taking the mean +/- 1.96 * sdev.

That's not an assumption I can just make easily though, so I did a bit of modelling to find out find out what are the typical probabilities for a given widget batch (the example scenario above), and what are the more extreme probability distributions, each at the different likely parameter/probability sizes.

I then calculated the Poisson Binomial mean, sdev, and 95% intervals using the approaches noted and ran monte carlo against them, doing this many times (with different draws from the middle and the extremes) to try and get as much coverage of the possible probability distributions I could run into. I compared the mean, sdev, and interval estimates against those retrieved from monte carlo by their percent error (treating the monte carlo result as the "right" answer).

Long story short, the mean and sdev values are in line with Poisson Binomial quite well. And the confidence intervals, while possibly being up to 5% off in the more extreme cases, are well within our business need for low parameter sizes, and have almost no error for higher parameter sizes (which is most of our cases).

So ultimately I used what I can from the known moments from the Poisson Binomial distribution directly, and then back those into intervals assuming normality (which works well enough for our specific use case).

$\endgroup$
0
$\begingroup$

The total number of failures will follow a binomial distribution, assuming the probability of failure is constant, and you can just calculate that binomial probability. You can look up “Binomial distribution” for how to do this; Blitzstein and Hwang’s textbook Probability is a great intro for this.

If n is large, the binomial can be approximated as normal.

If the probability is not constant, the calculation becomes more involved. Instead the total number of failures will need to be calculated, either painfully/tediously by hand, or more likely by Monte Carlo methods.

Note that this calculation gives a tolerance interval, not a confidence interval. A confidence interval is for a parameter, e.g. the probabilities, not the observations.

$\endgroup$
5
  • 1
    $\begingroup$ I note that the probability for each widget can differ, that is there are many unique probabilities going into this. My understanding is that the vanilla Binomial distribution is not appropriate in this case. I'm hoping to not resort to Monte Carlo as my conceptual "batch" can be quite larger, and I can have many batches, making it untenable in practice. $\endgroup$
    – Josh
    Commented Mar 14, 2022 at 22:49
  • $\begingroup$ If your batch is large then the only way to do this in a computationally feasible way will be to either use Monte Carlo, use approximations, or do something like assume a conjugate prior. $\endgroup$ Commented Mar 14, 2022 at 22:56
  • $\begingroup$ I'm perfectly up for approximations so long as the pitfalls it brings are relatively well understand. I'm naive as to what a conjugate prior would entail. $\endgroup$
    – Josh
    Commented Mar 14, 2022 at 23:07
  • $\begingroup$ The thread I linked to in a previous comment (to the question) contains various answers, some of which show how to perform exact calculations for very large values of $n.$ They are neither "painful" nor "tedious." Moreover, the Normal approximation rarely holds for this Poisson Binomial form of distribution. Finally, this interval is neither a tolerance interval nor a confidence interval (both of which are estimates based on data): it is a prediction interval for a random variable: namely, the number of failures. $\endgroup$
    – whuber
    Commented Mar 15, 2022 at 13:49
  • $\begingroup$ @whuber Thanks, you're right about it being a prediction interval. That being said, the Wikipedia page on the Poisson-Binomial definitely makes it sound like the calculations involved are difficult, as they require techniques like discrete Fourier transforms. $\endgroup$ Commented Mar 19, 2022 at 17:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.