0
$\begingroup$

Bayes Theorem

I know that Bayes theorem is: Posterior = Likelihood * Prior / Evidence

However, I am confused about the above notation in the picture.

  1. How do we get to the above three notation?
  2. How does integration come into the picture in the evidence part?
  3. Where does alpha come from?
  4. Why is Marginal likelihood written in the way they are in the above three equations?

Please explain your answer.

$\endgroup$
  • 1
    $\begingroup$ Can you share your resource please for us to get better understanding of the parameter $\alpha$ before making further assumptions? $\endgroup$ – gunes Sep 11 at 14:28
  • 2
    $\begingroup$ Small comment about that formula: the joint probability term should be $p (X,\theta|\alpha) $ instead of $p (X,\theta) $ $\endgroup$ – Sextus Empiricus Sep 11 at 17:27
  • $\begingroup$ @MartijnWeterings Thank you for pointing out. Can you explain how? $\endgroup$ – Akash Dubey Sep 11 at 17:39
4
$\begingroup$

How do we get to the above three notation?

How does integration come into the picture in the evidence part?

Where does alpha come from?

Why is Marginal likelihood written in the way they are in the above three equations?

I believe the confusion is being created by $\alpha$. This is a rarely used notation. The prior is usually just written as $p(\theta)$ and not $p(\theta|\alpha)$. They are equivalent.

The latter form is simply an acknowledgment that you did not randomly choose your prior distribution. All priors must be based on prior information. It does not require that the prior information is data. It also links the prior as the posterior of other information.

$\alpha$ is the information you have from outside the dataset that you are including to update your beliefs. The reason that $p(\theta)=p(\theta|\alpha)$ is that anything in $\alpha$ has been encoded in the prior. Effectively, it is gone, or rather, it's indistinguishable from the prior. The prior is the marginalization of the information in $\alpha.$

Now to questions 1,2, and 4.

How do we get to the above three notation?

The three are equivalents by the basic laws of probability. For example,$$\Pr(A;B)=\Pr(A|B)\Pr(B).$$ All of them are simply rearrangements of theorems or identities.

How does integration come into the picture in the evidence part?

The evidence is the expected likelihood of observing $X$ over the set of all possible explanations, with parameters and/or models being explanations. The numerator provides the point likelihood of observing the data, the denominator is an expectation. It is the expected likelihood. By the law of total probability, $$\int_\theta{f(X|\theta)}p(\theta)=p(X).$$

Adding the $\alpha$ is an acknowledgment of the role prior information played in forming the assessment of the expected probability of $X$. It directly follows from marginalization.

Why is Marginal likelihood written in the way they are in the above three equations?

$$\int_Bp(A|B)p(B|C)=p(A|C).$$ This is just marginalization. What this provides to a data scientist are multiple possible calculation rules that you could use. There are times when multiple formulations are useful. Bayes theorem can sometimes be solved readily under one construction but not so easily under another.

A simple example of this would be Bayes theorem for the normal likelihood with a conjugate prior. $$f(X|\mu;\sigma^2)p(\mu;\sigma^2)=f(X|\mu;\sigma^2)p(\mu|\sigma^2)p(\sigma^2).$$

The right-hand side has a pretty simple formulation. The left-hand side doesn't have an obvious formulation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.