Expectation and Variance of the Posterior Distribution Example: Seeking Elaboration on Normalising Constant

Question

I have the following example:

Assume that we have an observation $Y$ from a Binomial distribution with parameter $n=20$ and success probability $p: [ Y \sim \mathrm{Bin}(20,p)]$

Further assume that we observed $y=12$ and what to make inference about the value of $p$ using Zellner's prior, i.e. we want to use the following PDF as prior for $p: [ f(p) = c p^p (1-p)^{(1-p)} ]$ where $c$ is such that $\int_0^1 f(p)\,dp=1$. We can determine this constant by numeric integration in R:

```{r}
integrand <- function(p) p^p * (1-p)^(1-p)
res <- integrate(integrand, lower=0, upper=1)
res
(c <- 1/res$val)
```

It then continues as follows:

Given that we have observed $y=12$, the likelihood $L(p|y)$ is $[ L(p|y)= \binom{20}{12} p^{12} (1-p)^8]$ The posterior of $p$ is $[ f(p|y) \propto L(p|y) f(p) = \binom{20}{12} p^{12} (1-p)^8 c p^p (1-p)^{(1-p)}]$.

Whence $[ f(p|y) \propto p^{12+p} (1-p)^{9-p}]$.

Again, we could find the normalising constant for this density by numerical intergration:

```{r}
    integrand <- function(p) p^(12+p) * (1-p)^(9-p)
    res1 <- integrate(integrand, lower=0, upper=1)
    res1
    (c1 <- 1/res1$val)
    ```

We could also use numerical integration to calculate the posterior mean and posterior variance of $p$. The following calculates $\mathbb{E}[p|y]$, the expectation of the posterior distribution:

```{r}
integrand <- function(p) c1 * p * p^(12+p) * (1-p)^(9-p)
(resEp <- integrate(integrand, lower=0, upper=1))
Ep <- resEp$val
```

Now we calculate $\mathbb{E}[p^2|y]$, from which we can determine $\mathrm{Var}[p|y]=\mathbb{E}[p^2|y]- (\mathbb{E}[p|y])^2$, the variance of the posterior distribution:

```{r}
integrand <- function(p) c1 * p^2 * p^(12+p) * (1-p)^(9-p)
(resEp2 <- integrate(integrand, lower=0, upper=1))
Ep2 <- resEp2$val
(Varp <- Ep2 - Ep*Ep)
```

---

I have only just started learning Bayesian probability, so please be patient with me.

I understand that the $p^{12+p} \cdot (1-p)^{9-p}$ is $f(p|y)$ -- the posterior of $p$.

I understand that, in accordance with the formula for the expected value of a continuous random variable $E[X] = \int_\mathbb{R} xf(x)$, we multiply $f(p|y)$, which is essentially the density function $f(x)$, in this case, by $p$, which is essentially $x$, the realisations/outcomes/observations of our random variable, in this case, since $p$ is the random variable we are trying to find a posterior distribution for, as is in accordance with the Bayesian methodology.

I understand that the normalising constant is used to reduce any probability function to a probability density function between $0$ and $1$.

I don't understand the following:

1. Why is the normalising constant needed here (in general)?
2. Why do we need to multiply by the normalising constant in the formula for expectation? It has that $\mathbb{E}[p|y] = c_1 \cdot p \cdot p^{12+p} \cdot (1-p)^{9-p}$, and, as stated above, I think I understand all of it except for the normalising constant.

I would greatly appreciate it if people could please take take the time to clarify this.

Answer 1

Unfortunately, you are a little off in several of your beliefs.

1. $p^{12+p}(1-p)^{9-p}$ is not the posterior of $p$, as it hasn't been normalized to integrate to 1. It is proportional to the posterior, but without the normalization constant, it is not a probability distribution and cannot be used as such.

2. The normalizing constant is not there to "reduce any probability [density - I assume this should have been included] function to a probability density function between 0 and 1." It is there to make the probability density function integrate to exactly 1. The values of the density function itself need not be $\leq 1$; consider a Uniform distribution over the interval $(0, 1/2)$. In order to make that integrate to 1, the density $f(x)$ must equal 2 everywhere.

If the posterior density integrates to some number $> 1$, then you are saying "the probability of the parameter taking on a value - any value - is greater than 1", which is ... problematic ... in several respects. Similarly, if the posterior density integrates to a value $<1$, you are saying "the probability of the parameter taking on a value - any value - is less than 1". You can see that, too, is problematic in several respects. For example, what happens if the event "the parameter doesn't take on any value" occurs?

The implications for your specific questions are as follows. The normalizing constant is there so the distribution integrates to 1. If you don't multiply by the normalizing constant, you aren't calculating an expected value, you're just calculating an integral; it's not an expected value because you aren't working with a probability distribution, and you're not working with a probability distribution because your function $f(\cdot)$ doesn't integrate to 1.