9
$\begingroup$

If my prior is modelled as a continuous probability distribution, say, a beta distribution skewed to reflect my bias towards certain models, how can I calculate the posterior probability?

The challenge for me is calculating the probability of a given model, since the continuous distribution will only give me estimates for intervals.

Please forgive the naivety of the question, I've only recently started studying Bayesian statistics.

$\endgroup$
  • $\begingroup$ I guess the correct question would be "How can I calculate the probability of the model given a data sample?" I can easily calculate probability of the data given the model, but I don't know how to estimate the probability of the model. And yes, I'm interested in model comparison. $\endgroup$ – Rafa Apr 21 '15 at 17:51
3
$\begingroup$

For comparing models, say $$\mathfrak{M}_1=\{f_1(\cdot|\theta_1);\ \theta_1\in\Theta_1\}$$ and $$\mathfrak{M}_2=\{f_2(\cdot|\theta_2);\ \theta_2\in\Theta_2\}$$the classical Bayesian answer is (Jeffreys, 1939) to produce a Bayes factor $$\mathfrak{B}_{12}(x)=\frac{\int_{\Theta_1} f_1(x|\theta_1)\pi_1(\text{d}\theta_1)}{\int_{\Theta_2} f_2(x|\theta_2)\pi_2(\text{d}\theta_2)}$$When $\mathfrak{B}_{12}(x)$ is larger than $1$ the data favours model $\mathfrak{M}_1$; when $\mathfrak{B}_{12}(x)$ is smaller than $1$, the data favours model $\mathfrak{M}_2$.

| cite | improve this answer | |
$\endgroup$
6
$\begingroup$

Bayes theorem is: $$ P(A|B) = \frac{P(B|A)P(A)}{P(B)} $$

In a case where you have some data and a parameter, it is common to use $\theta$ for the parameter (or parameter vector) and $x$ for the data.

You might place a prior on $\theta$, $p(\theta)$, and you might have a model $p(x|\theta)$ which gives the probability of your data given the model. You can then use Bayes rule/theorem to "invert" this and get $p(\theta|x)$.

Only in a relatively small set of examples is it possible to get closed form solutions for $p(\theta|x)$. For arbitrary cases, often you approximate the posterior distribution using some standard methods in Bayesian statistics -- for example, the two most common broad approaches are markov chain monte carlo or variational Bayes.

Suppose you are interested in a simple case where a closed form posterior exists. An example of this would be if $p(\theta)$ is a standard normal (Gaussian with unit variance and zero mean) and $p(x|\theta)$ is a normal with mean value of $\theta$ and unit variance.

I will omit normalization factors for convenience. Also note that the denominator in Bayes rule tends to simply renormalize things: $$ p(\theta|x) \propto e^{-(x-\theta)^2/2} e^{-\theta^2/2}\\ $$ Let's combine the exponents and complete the square $$ -(x-\theta)^2/2 - \theta^2/2 \propto - (x^2 - 2\theta x + \theta^2) - \theta^2 $$ Recall that x is fixed here because it has been observed and we want expect our answer will be in terms of it. Complete the square and see that the exponent is $\propto -(\theta - x/2)^2$ with other terms of which depend on x. So: $$ p(\theta|x) \propto e^{-a(\theta - x/2)^2} $$
where 'a' is a factor that can be obtained by book-keeping. Notice that the posterior is a normal distribution with mean value x/2. Attempt to compute the variance for yourself.

Note that our answer makes intuitive sense...the prior said that $\theta$ is zero and we observe a sample $x$ that has expected value of $\theta$. Since the variance of the prior and the distribution $p(x|\theta)$ are equal magnitude, we trust them equally. Accordingly, our posterior is a distribution with a mean that is the average of $x$ and 0 and which ends up having smaller variance than the initial $p(x|\theta)$ or $p(x)$ (not shown here).

For model comparison, you could look at a ratio: $$ \frac{p(x|\theta_1)}{p(x|\theta_2)} $$

This is called the likelihood ratio (see wikipedia or elsewhere). Here you don't need the posterior, you simply are looking at how (relatively) likely your data (or observations) are given either $\theta_1$ or $\theta_2$ being the parameter of the model that generated your observations.

Hope this helps.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Sorry your answer is incorrect. The Bayes factor is not defined this way! $\endgroup$ – Xi'an Apr 21 '15 at 20:43
  • $\begingroup$ For model comparison, I described the likelihood ratio. Initially I mistakenly used the term Bayes factor. $\endgroup$ – Josh Apr 21 '15 at 20:51
  • $\begingroup$ Except that you do not know $\theta_1$ and $\theta_2$ that generated the observations. $\endgroup$ – Xi'an Apr 21 '15 at 20:54
  • $\begingroup$ Just meant to describe the simple case in which you have two hypothetical values of the model parameters and you wish to compare how well the data follows from them. Agreed that if you have two model forms and would like to compare them without knowledge of the specific parameters, your answer provides the correct approach. $\endgroup$ – Josh Apr 21 '15 at 21:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.