What family is the posterior predictive distribution in when the likelihood is a Bernoulli and the prior (and posterior) is Gaussian?

Question

I have a problem where the hidden random variables I want to know about are continuous (i.e. my parameters can take any real number), and so I have modeled them as having a Gaussian p.d.f. (because this is typically done in my context). However, the likelihood is a Bernoulli (or binomial), given that I want to perform binary classification. In other words, you can imagine that there's the data-generating process $\tilde{x} \sim p(x \mid \theta)$, where $x$ is discrete and $\theta$ is continuous, so $p(\theta)$ is a Gaussian and the likelihood $\theta \mapsto p(\tilde{x} \mid \theta)$ is a Bernoulli.

Now, I know that a Bernoulli likelihood with a beta prior leads to a beta posterior and that a Gaussian prior and a Gaussian likelihood lead to a Gaussian posterior (i.e. these are the conjugate priors).

However, what family is the posterior predictive distribution (p.p.d.) in when the likelihood is a Bernoulli (or binomial) and the prior and sampling distributions (i.e. the distribution I use to compute the p.p.d.) are Gaussians? Why do I want to know this? Because I want to estimate the p.p.d. by sampling and I want to know which moments I can estimate. For example, would it make sense to compute the mean or variance of the samples if the p.p.d. is not a Gaussian? Probably not.

Now, in my case, I am approximating the posterior with a Gaussian, so the posterior I will use the compute the p.p.d. will be a Gaussian, but the likelihood will still be a Bernoulli.

Answer 1

The posterior distribution has simple, trackable form only if you are using conjugate priors. Gaussian is not a conjugate prior for Bernoulli, or binomial, hence there is no named distribution that posterior follows. To get it, you need to apply Bayes theorem directly, or approximate it, e.g. by MCMC sampling. On another hand, if you knew what the posterior distribution is, there would be no need for approximating it.

Why computing mean or variance of non-Gaussian would not make sense? Sure, there are pathological cases like Cauchy distribution, where those are undefined, but they are exceptions rather than rule. It is common to calculate basic statistics like mean, median, variance etc. form Monte Carlo samples.