Proportionality in Bayesian Models: What Is Absorbed?

Question

Considering two Bayesian models:

• Poisson Likelihood & Beta Prior:

$p(y|\lambda) \sim \text{Pois}(\lambda)$, $p(\lambda) \sim \text{Be}(a, b)$:

$$ p(\lambda|y) \propto \lambda^{a-1}e^{-b\lambda} \times \lambda^{y}e^{-\theta} $$

$$ = \lambda^{a+y-1}e^{-\lambda(1+b)} $$

$$ \sim \text{Ga}(a+y, b+1) $$

• Normal likelihood & Normal Prior:

$p(x|\theta, \phi) \sim \text{N}(\theta, \phi)$, $p(\theta) \sim \text{N}(\theta_{0}, \phi_{0})$:

$$ p(\theta|x) \propto \text{exp}\left\{-\frac{1}{2}(\theta-\theta_{0})^{2}/\phi_{0}\right\} \times \text{exp}\left\{-\frac{1}{2}(x-\theta)^{2}/\phi\right\} $$

$$ = \text{exp}\left\{-\frac{1}{2}\left(\frac{\theta^{2}-2\theta\theta_{0}+\theta_{0}^{2}}{\phi_{0}}+\frac{x^{2}-2x\theta+\theta^{2}}{\phi}\right)\right\} $$

$$ \propto \text{exp}\left\{-\frac{1}{2}\theta^{2}(\phi_{0}^{-1}+\phi^{-1})+\theta\left(\frac{\theta_{0}}{\phi_{0}}+\frac{x}{\phi}\right)\right\} $$

$$ \sim \text{N}(\theta_{1}, \phi_{1}) $$

where:

$\phi_{1} = \frac{1}{\phi_{0}^{-1}+\phi^{-1}}$, $\theta_{1} = \phi_{1}\left(\frac{\theta_{0}}{\phi_{0}}+\frac{x}{\phi}\right)$.

Why is it, in the first model, the terms ($a$, $b$, $y$, $n$) in the first model are retained, but in the second model, the terms $\left(\frac{\theta_{0}^{2}}{\phi_{0}}, \frac{x^{2}}{\phi}\right)$ are dropped?

Finally, once again in the second model, why is it that the posterior mean is equal to $\theta_{1}$ and the posterior variance is equal to $\phi_{1}$?

EDIT

I'm aware that if some term is a multiple of the parameter of interest (e.g. $c\theta$ or even $\frac{\theta}{c}$), then the scalar ($c$, in this instance) is absorbed by the proportionality sign, leaving just the parameter ($\theta$), but, if possible, could somebody explain the general rules of proportionality with regard to exponents?

A few examples:

$e^{c+\theta}$, $e^{c\theta}$, $\theta^{c+x}$, $\theta^{cx}$

Assuming $\theta$ is the parameter of interest, what terms would a proportionality sign absorb in the above expressions?

Answer 1

The core idea with the $\propto$ short-cut is that in Bayesian posterior calculations, you can drop any multiplicative *constant* because you know you will recover the right constant in imposing that your function integrates to one. Hence, $f(\theta,x,\mu) \propto \tilde f(\theta,x,\mu)$ means that $$ f(\theta,x,\mu) = \tilde f(\theta,x,\mu)\bigg/ \int \tilde f(\theta,x,\mu) \text{d}\theta $$ Now in this example I use three entries, $\theta$, $x$, and $\mu$, to make you aware that there are terms in the function that both vary and are considered as constant for the proportionality sign. E.g., $x$ is the data, which "vary" until observed, and $\mu$ is the hyperparameter, which may vary depending on the experiment(er). (This is not a question of "parameter of interest" but rather of "parameter of the sampling distribution". Your remark about the scalar transform of $\theta$ in the EDIT part is just plain wrong.)

For instance, in your Poisson example, $\theta=\lambda$, $x=y$, and $\mu=(a,b)$. In the normal example, $\theta=\theta$, $x=x$, and $\mu=(\theta_0,\phi,\phi_0)$.

This $\propto$ short-cut means that, when $f(\theta,x,\mu) = \tilde f(\theta,x,\mu)\times g(x,\mu)$ you can drop $g$ from the calculation, i.e. $f(\theta,x,\mu) \propto f(\theta,x,\mu)$.

As to answer why in the normal example the posterior mean and the posterior variances are such expressions, this is straightforward calculus, turning the second degree expression in the exponential into a perfect square.