1
$\begingroup$

Considering two Bayesian models:

  • Poisson Likelihood & Beta Prior:

$p(y|\lambda) \sim \text{Pois}(\lambda)$, $p(\lambda) \sim \text{Be}(a, b)$:

$$ p(\lambda|y) \propto \lambda^{a-1}e^{-b\lambda} \times \lambda^{y}e^{-\theta} $$

$$ = \lambda^{a+y-1}e^{-\lambda(1+b)} $$

$$ \sim \text{Ga}(a+y, b+1) $$

  • Normal likelihood & Normal Prior:

$p(x|\theta, \phi) \sim \text{N}(\theta, \phi)$, $p(\theta) \sim \text{N}(\theta_{0}, \phi_{0})$:

$$ p(\theta|x) \propto \text{exp}\left\{-\frac{1}{2}(\theta-\theta_{0})^{2}/\phi_{0}\right\} \times \text{exp}\left\{-\frac{1}{2}(x-\theta)^{2}/\phi\right\} $$

$$ = \text{exp}\left\{-\frac{1}{2}\left(\frac{\theta^{2}-2\theta\theta_{0}+\theta_{0}^{2}}{\phi_{0}}+\frac{x^{2}-2x\theta+\theta^{2}}{\phi}\right)\right\} $$

$$ \propto \text{exp}\left\{-\frac{1}{2}\theta^{2}(\phi_{0}^{-1}+\phi^{-1})+\theta\left(\frac{\theta_{0}}{\phi_{0}}+\frac{x}{\phi}\right)\right\} $$

$$ \sim \text{N}(\theta_{1}, \phi_{1}) $$

where:

$\phi_{1} = \frac{1}{\phi_{0}^{-1}+\phi^{-1}}$, $\theta_{1} = \phi_{1}\left(\frac{\theta_{0}}{\phi_{0}}+\frac{x}{\phi}\right)$.

Why is it, in the first model, the terms ($a$, $b$, $y$, $n$) in the first model are retained, but in the second model, the terms $\left(\frac{\theta_{0}^{2}}{\phi_{0}}, \frac{x^{2}}{\phi}\right)$ are dropped?

Finally, once again in the second model, why is it that the posterior mean is equal to $\theta_{1}$ and the posterior variance is equal to $\phi_{1}$?

EDIT

I'm aware that if some term is a multiple of the parameter of interest (e.g. $c\theta$ or even $\frac{\theta}{c}$), then the scalar ($c$, in this instance) is absorbed by the proportionality sign, leaving just the parameter ($\theta$), but, if possible, could somebody explain the general rules of proportionality with regard to exponents?

A few examples:

$e^{c+\theta}$, $e^{c\theta}$, $\theta^{c+x}$, $\theta^{cx}$

Assuming $\theta$ is the parameter of interest, what terms would a proportionality sign absorb in the above expressions?

$\endgroup$
  • 1
    $\begingroup$ The second example is confusing because $x$ appears to play two distinct roles, both of data and parameter. It also is not apparent what you mean by "terms" being "dropped" when those terms have not explicitly appeared anywhere. I suspect you might just be a victim of sloppy notation, in which case being more precise in the notation and in how you carry out the calculations may answer the question for you. $\endgroup$ – whuber Nov 26 '12 at 21:04
  • $\begingroup$ Sorry about that. I believe I have fixed it now. For the second example, $x$ is certainly the data whilst $\theta$ is the parameter of interest. If somebody could provide answers to the shorter examples under the latest section marked "EDIT", I'm sure I could work it out from there. Thanks! $\endgroup$ – Will Clyne Nov 27 '12 at 1:49
4
$\begingroup$

The core idea with the $\propto$ short-cut is that in Bayesian posterior calculations, you can drop any multiplicative *constant* because you know you will recover the right constant in imposing that your function integrates to one. Hence, $f(\theta,x,\mu) \propto \tilde f(\theta,x,\mu)$ means that $$ f(\theta,x,\mu) = \tilde f(\theta,x,\mu)\bigg/ \int \tilde f(\theta,x,\mu) \text{d}\theta $$ Now in this example I use three entries, $\theta$, $x$, and $\mu$, to make you aware that there are terms in the function that both vary and are considered as constant for the proportionality sign. E.g., $x$ is the data, which "vary" until observed, and $\mu$ is the hyperparameter, which may vary depending on the experiment(er). (This is not a question of "parameter of interest" but rather of "parameter of the sampling distribution". Your remark about the scalar transform of $\theta$ in the EDIT part is just plain wrong.)

For instance, in your Poisson example, $\theta=\lambda$, $x=y$, and $\mu=(a,b)$. In the normal example, $\theta=\theta$, $x=x$, and $\mu=(\theta_0,\phi,\phi_0)$.

This $\propto$ short-cut means that, when $f(\theta,x,\mu) = \tilde f(\theta,x,\mu)\times g(x,\mu)$ you can drop $g$ from the calculation, i.e. $f(\theta,x,\mu) \propto f(\theta,x,\mu)$.

As to answer why in the normal example the posterior mean and the posterior variances are such expressions, this is straightforward calculus, turning the second degree expression in the exponential into a perfect square.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.