In a Bayesian model, we normally have that:

$$ p(\boldsymbol\mu|\boldsymbol X) = \dfrac{p(\boldsymbol X|\boldsymbol \mu)p(\boldsymbol \mu)}{p( \boldsymbol X)} $$

Now suppose that $\boldsymbol \mu \sim N(\boldsymbol \mu_0, A)$ and that $\boldsymbol X | \boldsymbol \mu \sim N(\boldsymbol \mu, B)$. In this case, by conjugation properties, the posterior, $p(\boldsymbol\mu|\boldsymbol X) $ is also normal.

Suppose now that I want to find the marginal density of $\boldsymbol X$. Then normally we would integrate $p(\boldsymbol X|\boldsymbol \mu)p(\boldsymbol \mu)$ with respect to $\boldsymbol \mu$.

HOWEVER, another method is to just use:

$$ p( \boldsymbol X) = \dfrac{p(\boldsymbol X|\boldsymbol \mu)p(\boldsymbol \mu)}{p(\boldsymbol\mu|\boldsymbol X)} $$

and to just drop all terms not in $\boldsymbol X$, in essence, to find the kernel of $\boldsymbol X$, which should be an exponential form. After this, we just fill in the constants by way of identification of the kernel.

It appears that here this technique works. However I am wondering if in general this result holds.

My question is: What allows us to know that $p(\boldsymbol X)$ is a valid probability density function just by looking at the kernel? If the likelihood, posterior, and prior are all valid probability distribution functions summing to $1$, is it enough for me to just "fill" in constants by way of lookinga tthe kernel?


Bayes theorem is

$$ f_{X\mid Y}(x \mid y) = \frac{ f_{Y\mid X}(y \mid x) \; f_X(x) }{ f_Y(y) } = \frac{ f_{Y\mid X}(y \mid x) \; f_X(x) }{ \int f_{Y\mid X}(y \mid x) \; f_X(x) \;dx } $$


$$ f_Y(y) = \int f_{Y\mid X}(y \mid x) \; f_X(x) \;dx $$

by the law of total probability. So it follows from the probability theory.

  • $\begingroup$ Is this dependent on the prior being proper? Also would having the kernel form in the denominator be enough to identify the distribution? $\endgroup$
    – user321627
    Nov 3 '16 at 10:24
  • $\begingroup$ @user321627 it holds for proper probability distributions (pmf, pdf) in other cases it does not have to hold. $\endgroup$
    – Tim
    Nov 3 '16 at 10:27
  • $\begingroup$ @user321627 moreover, if you want only to estimate the unconditional distribution of data, then why do you want to use Bayes theorem for it? Why not simply use KDE? $\endgroup$
    – Tim
    Nov 3 '16 at 11:05
  • $\begingroup$ Sorry for posting my questin here, but I see you had answered perfectly several questions about Bauesian theorem. This answer is the best match my need. I would like to know from the formula you posted in this answer how the posterior distribution is derived in practice ? Do we apply this formula for every $x$ value of $X$ and every $y$ in the data ? thank you in advance $\endgroup$
    – Nizar
    Aug 26 '17 at 13:40
  • $\begingroup$ @Nizar the answer is "yes", but I encourage you to post it as a question and describe in greater detail what you understand and what is unclear. $\endgroup$
    – Tim
    Aug 26 '17 at 21:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.