1
$\begingroup$

I have a problem where the hidden random variables I want to know about are continuous (i.e. my parameters can take any real number), and so I have modeled them as having a Gaussian p.d.f. (because this is typically done in my context). However, the likelihood is a Bernoulli (or binomial), given that I want to perform binary classification. In other words, you can imagine that there's the data-generating process $\tilde{x} \sim p(x \mid \theta)$, where $x$ is discrete and $\theta$ is continuous, so $p(\theta)$ is a Gaussian and the likelihood $\theta \mapsto p(\tilde{x} \mid \theta)$ is a Bernoulli.

Now, I know that a Bernoulli likelihood with a beta prior leads to a beta posterior and that a Gaussian prior and a Gaussian likelihood lead to a Gaussian posterior (i.e. these are the conjugate priors).

However, what family is the posterior predictive distribution (p.p.d.) in when the likelihood is a Bernoulli (or binomial) and the prior and sampling distributions (i.e. the distribution I use to compute the p.p.d.) are Gaussians? Why do I want to know this? Because I want to estimate the p.p.d. by sampling and I want to know which moments I can estimate. For example, would it make sense to compute the mean or variance of the samples if the p.p.d. is not a Gaussian? Probably not.

Now, in my case, I am approximating the posterior with a Gaussian, so the posterior I will use the compute the p.p.d. will be a Gaussian, but the likelihood will still be a Bernoulli.

$\endgroup$
1
$\begingroup$

The posterior distribution has simple, trackable form only if you are using conjugate priors. Gaussian is not a conjugate prior for Bernoulli, or binomial, hence there is no named distribution that posterior follows. To get it, you need to apply Bayes theorem directly, or approximate it, e.g. by MCMC sampling. On another hand, if you knew what the posterior distribution is, there would be no need for approximating it.

Why computing mean or variance of non-Gaussian would not make sense? Sure, there are pathological cases like Cauchy distribution, where those are undefined, but they are exceptions rather than rule. It is common to calculate basic statistics like mean, median, variance etc. form Monte Carlo samples.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Well, how are you supposed to compute the statistics for different distributions? If you can support your claims about the statistics with a reference (i.e. a book or research paper), I would really appreciate it. $\endgroup$ – nbro Jul 30 at 21:00
  • 1
    $\begingroup$ @nbro the usual way is to take samples from this distribution using MCMC and calculate sample statistics. $\endgroup$ – Tim Jul 31 at 5:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.