0
$\begingroup$

I have N distinct bernoulli trials with a distinct probability for each trial given by, P=(p1, p2, ..., pN). I want to know the distribution of the number of successes. Given that I know P, I can fully describe the distribution with a poisson-binomial distribution. As expected, the result is almost identical to the poisson distribution with rate=N*mean(P), since N=5000 is large and mean(P)=0.00114 is small. Poisson-binomial Distribution

I can also easily compute the parameters to a beta distribution from P, and describe the distribution of successes with a beta-binomial distribution.

Beta-Binomial Distribution

Why are the two distributions so different? This is my main question.

Clearly they respond to variance in P differently. When I artificially reduce the variance in P for the BB distribution by 1/20 the BB distribution mimics the shape of the PB. PB (black), BB with less variance (green)

When I examine a histogram of actual data for a fairly large sample (80k+ data points, each with N trials), the resulting distribution looks like the PB and BB distributions have been combined. So I am unsure which best describes the data, and why.

acutal distribution

Thanks!

$\endgroup$
  • $\begingroup$ are those 0 probabilities being assigned to integral values? The density curve should not look like a saw wave. $\endgroup$ – AdamO Feb 14 at 19:30
1
$\begingroup$

My question stemmed from my own ignorance at the time, but I thought I'd post the answer in case anyone else has the same misunderstanding I did.

When I posed the question I erroneously understood that the beta-binomial distribution assumed that every one of the N Bernoulli trials performed in one run of the experiment would draw a unique probability parameter from the defined beta distribution.

Now I understand that a single probability parameter is drawn from the defined beta distribution to be applied in ALL Bernoulli trials performed in one run of the experiment. But different experimental trials of the beta-binomial may each have distinct probability parameters. In the WikiPedia definition (https://en.wikipedia.org/wiki/Beta-binomial_distribution), "The beta-binomial distribution is the binomial distribution in which the probability of success at each trial is fixed but randomly drawn from a beta distribution prior to n Bernoulli trials" the key phrase is "...drawn...prior to n Bernoulli trials".

With that, it's clear that for a given set of trial probabilities, P, in a poisson binomial distribution, one cannot simply estimate the beta distribution describing that set of probabilities and plug it in to a beta-binomial. So my question really makes no sense.

Of course, by artificially reducing the variance to approach the mean probability value, the beta-binomial will resemble the equivalent poisson (and poisson binomial) distributions as I described in the question.

Sorry for the confusion, but hopefully this will help someone else.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.