2
$\begingroup$

When talking about conjugate distributions on a online video, it applies the Bayes theorem as:

$$ Pr(\theta|X) = \frac{Pr(X|\theta) Pr(\theta)}{Pr(X)} $$

and says that $Pr(X|\theta)$ is fixed by our model, $Pr(X)$ is fixed by the data while $Pr(\theta)$ is up to us.

I am assuming $Pr(\theta)$ is up to us in the sense that we use some prior knowledge or make an assumption about it as if it were true.

But what does it mean that the likelihood is fixed by the model?

Also $Pr(X)$ is fixed by data since we got observations of events. But if you consider a dataset with many points, it's difficult to have more than one entry for the same point: does $Pr(X)$ represent a distribution rather than a single probability?

$\endgroup$

1 Answer 1

5
$\begingroup$

The whole point of doing Bayesian Inference, is that you stipulate your data was generated by a model with unknown parameters. An example of this would be "the samples were all drawn independently from the same normal distribution, but we don't know the mean or variance of that distribution"

We then try to use the data to come up with beliefs about the likely "true values" of the parameters of the model. Once you have stipulated your model, then $Pr(X|\theta)$ can be evaluated for any given $\theta$ (in the above example, $\theta$ refers to two parameters, the mean and variance/std of your normal dist). That's what "likelihood is fixed by the model" means.

We then seek to come up with a posterior. When writing out Bayes Theorem as you did in your original post, people usually leave out this "|M" part or see it as implied, but it is probably more instructive to write:

$Pr(\theta| X, M) = \frac{P(X|\theta, M)P(\theta |M)}{P(X| M)}$

$Pr(X|M)$ is a bit hard to interpret. The most simple (this isn't the deepest, but it's probably the best explanation to run with, when you're still learning) interpretation, is simply that $Pr(\theta|X, M) \propto Pr(X|\theta , M)Pr(\theta | M)$. You require however, that $Pr(\theta|X, M)$ is normalised wrt $\theta$, and thus

$$Pr(\theta|X ,M)=\frac{Pr(X|\theta , M)P(\theta|M)}{\int Pr(X|\theta, M)Pr(\theta | M)d\theta}$$

Of course, from this, you can see that $Pr(X | M) = \int Pr(X|\theta, M)Pr(\theta |M)d\theta$ - when people say "it's fixed by the data", that means that it doesn't depend on $\theta$, all $\theta$-dependence has been integrated out. It does however still depend on which ever model you stipulated, which is easier to see when you don't leave out the "|M"

In a sense, the above formula is just a statement about conditional probability, you could replace $\theta$ in it with k or "fred" and it would remain true. It just so happens that we know how to write down a formula for $Pr(X|\theta, |M)$, because our model stipulates it, and we also have our prior $Pr(\theta |M)$, so only if we expand in terms of $\theta$, do we know how to evaluate it (i.e. $Pr(fred|M)$ is not defined)

If you want more of an interpretational view, $P(X|M)$ is the probability of seeing the data we have seen, given that Model M generated it, but making no claim about what those parameters were. Instead, it's the weighted sum (integral) of the probability of Model M with parameters $\theta$ having generated the data, weighted by your prior beliefs about how plausible different values of $\theta$ are.

Does this help clear things up?

$\endgroup$
7
  • $\begingroup$ Yes it helps: it clarifies $Pr(X|\theta)$. It would be more correct if all $Pr$ are $Pr$ are rewritten specifying they depend on a specific model i.e., $Pr(\theta|M)$, $Pr(\theta|X,M)$ and so on. $Pr(X)$ as a normalization factor does not convince me very much: it is a little confusing in the sense that the numerator is the product of two probabilities so it will be in $[0,1]$. Don't get me wrong I know it is correct and I know the general Bayes formula derivation and that it can be re-expressed as that integral: I think I will go with this. $\endgroup$ Commented Oct 2, 2019 at 7:38
  • $\begingroup$ When you say we have the prior, do you mean we have chosen a model for the prior with given parameters, i.e., a normal distribution with given mean and variance? This model is independent from the model chosen for the likelihood right? $\endgroup$ Commented Oct 2, 2019 at 7:39
  • 1
    $\begingroup$ To your first comment, the "|M" is implied everywhere, the correct statement of Bayes here is: $Pr(\theta|X,M) = \frac{Pr(X|\theta,M)Pr(\theta|M)}{Pr(X|M)}$ The prior, in words, should be interpreted as "we have stipulated that the data was generated by a certain type of model, which has some parameters $\theta$. Before we look at the data present, what are plausible values of these parameters?". In the case of a coin toss, you might say, before performing an experiment "based on simple physics, I think it's unlikely I'll get heads 80%+ of the time" $\endgroup$
    – gazza89
    Commented Oct 2, 2019 at 9:20
  • $\begingroup$ And as regards "P(X|M)" ... You should think of it as "what is the probability of seeing the data I've seen, if it was generated by Model M". If it was generated by Model M however, that makes no stipulation about the value of the parameters, so it's a weighted average (i.e. an integral) of the probability of seeing this data over all parameter values possible for this model. The weighting is against the prior $\endgroup$
    – gazza89
    Commented Oct 2, 2019 at 9:22
  • $\begingroup$ Yes the Bayes formula with each probability conditioned on the model is exactly what I meant. Thanks the second comment clarifies and it is clearer. Thanks for your patience :) Nice answers. $\endgroup$ Commented Oct 2, 2019 at 9:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.