0
$\begingroup$

I have observed N (a_i, p_i) pairs each drawn from a different Bernoulli distribution. Here a_i are observed amplitudes and p_i are observed probabilities of success for the i^{th} draw.

I would like to model the full likelihood distribution (and not just via the MLE) with a view to identifying which draws belong to the successful class (and hence the statistics of just these draws, especially any irregular uncertainty distributions etc.).

For example, I have

a_i p_i p_true
10 0.2 0
100 0.9 1
11 0.1 0
99 0.93 1
12 0.25 0

I know the PDFs of the model and of the data (both are obviously Bernoulli distributions), but how do I combine them to obtain residuals that I can use to explore the joint distribution? What distribution does the joint distribution follow and how?

I have tried to unpack the cross-entropy and data and model but don't have a clear solution.

Is the continuous Bernoulli distribution a complete red herring?

Note that ordering doesn't matter in my example. Some other points in response to comments:

  1. The p_i values come from an oracle - these represent the probabilities that the datapoints belong to a common class.

  2. The amplitudes a_i are just weights for the corresponding Bernoulli distributions. The higher the amplitude, the higher the scaling of the Bernoulli distribution in its contribution to the overall process.

See also:

"Weighted" Poisson binomial distribution

Weighted sum of Bernoulli distributions

https://math.stackexchange.com/questions/3481907/sum-of-weighted-independent-bernoulli-rvs

What is the CDF of the sum of weighted Bernoulli random variables?

Please let me know if you need any more information to assist in answering. Thanks as ever!

$\endgroup$
12
  • $\begingroup$ Is p_i directly observed or is it some modeled value based on a_i? $\endgroup$ Apr 4, 2023 at 13:27
  • $\begingroup$ "a_i are observed amplitudes" amplitudes of what? $\endgroup$ Apr 4, 2023 at 13:28
  • $\begingroup$ What is 'p_true' in your table? Is it the true value of p? How do you know it and why does it only have values 0 and 1? $\endgroup$ Apr 4, 2023 at 13:30
  • 1
    $\begingroup$ What model? Could you explain it. $\endgroup$ Apr 4, 2023 at 13:31
  • 1
    $\begingroup$ "Is the continuous Bernoulli distribution a complete red herring?" This sentence is unclear. What is the 'continuous Bernoulli distribution' and how does it connect with the rest of the text? $\endgroup$ Apr 4, 2023 at 13:33

1 Answer 1

0
$\begingroup$

I know the PDFs of the model and of the data (both are obviously Bernoulli distributions),

Bernoulli distributions are not characterized by a probability density function (PDF) and instead are characterized by a probability mass function (PMF).

but how do I combine them to obtain residuals that I can use to explore the joint distribution?

In many problems the observations are assumed to be independent and are multiplied. For instance if you have two pmf's depending on some parameter $p$ like $$P(X_1=x_1;p) = f_1(x_1,p)$$ and $$P(X_2=x_2;p) = f_2(x_2,p)$$ then the pmf for the joint probability is $$P(X_1=x_1 \land X_2=x_2 ;p) = f_1(x_1,p)f_2(x_2,p)$$.

For Bernoulli distributions the observations $X_i$ are discrete, but the parameter $p$ is continuous. So if you use this to make a likelihood function, then you get a continuous function.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.