# Bayesian evidence

I am currently trying to solve an exercise (Information Theory, Inference, and Learning Algorithms, by David J.C. MacKay, ch 28 exercise 28.1)

The exercise is as follows: Random variables $x$ come independently from a probability distribution $P(x)$. According to model $H_{0}$, $P(x)$ is a uniform distribution.

$$P(x|H_0) = \frac{1}{2} \qquad x \in (-1,1).$$

According to model $H_{1}$, $P(x)$ is a nonuniform distribution with an unknown parameter $m \in$ $(-1,1)$:

$$P(x|m,H_{1}) = \frac{1}{2}(1+mx) \qquad x\in(-1,1).$$ Given the data $D = \{0.3, 0.5, 0.7, 0.8, 0.9\}$, what is the evidence for $H_{0}$ and $H_{1}$?

The evidence $P(D|H_{0})$ is as follows:

Edit 1 (thanks to Siong Thye Goh): Let $D = \{d_{i}|i = 1,\dots,5\}$. Since we assume independence, we have

$$P(D|H_{0}) = \prod_{i=1}^{5}P(d_{i}|H_{0}) = \frac{1}{2}^{5}.$$

Question 2: However, now I am stuck with $P(D|H_{1})$.

$$P(D|H_{1}) = \int_{-1}^{1}P(D|H_{1},m)\cdot P(m|H_{1})dm.$$

In the book, it is stated that the evidence can be approximated by: $$P(D|H_{1}) \approx P(D|m_{mp},H_{1})\cdot P(m_{mp}|H_{1})\sigma_{m|D}.$$ Here,

• $m_{mp}$ defines the most probable value for the parameter $m$.
• $\sigma_{m|d}$ defines the posterior uncertainty in $m$, as a part of an occam factor.
• $P(m_{mp}|H_{1})$ = $\frac{1}{\sigma_m}$, represents the range of values of $m$ that were possible a priori.

I'm guessing here that $\frac{1}{\sigma_{m}} = \frac{1}{2}$ since $m \in (-1,1)$ and that $P(d_{i}|m_{mp},H_{1}) = \frac{1}{2}(1+m_{mp}d_{i})$, so that the approximation for the evidence becomes

$$P(d_{i}|H_{1}) \approx \frac{1}{2}(1+m_{mp}d_{i})\cdot \frac{\sigma_{m|D}}{2}.$$

But now:

• what would $\sigma_{m|D}$ be in this case (if this even is the right approach)?
• How do we evaluate what $m_{mp}$ is? Can this just be done via maximum likelihood, as we do in classical statistics?
• There is a crucial piece of information not provided in the exercise: The prior distribution of $m$ (i.e. $P(m|H_{1})$). I would suggest using a uniform distribution, making $P(m|H_1)=1/2$. Now you have all ingredients to do the integration. No Laplace approximation needed. – COOLSerdash Sep 16 '17 at 21:16
• I think the exercise was made like this one purpose so that there would be no prior, or else it would indeed be quite easy. At least that is what I can guess from the difficulty indicator next to the exercise. For now I will just mark this question as answered. – Stephan Sep 16 '17 at 22:13

Let $D = \{ d_i |i = 1, \ldots, 5\}$,
$$P(D|H_0) = \prod_{i=1}^5P(d_i |H_0 ) = \left( \frac12 \right)^5$$
• Thank you for your answer, but I have one question about this. Since we assume that the models are continuous, wouldn't $P(d_{i}|H_{0}) = 0$ for all i, since the probability of any single point in a continuous distribution is infinitely small? – Stephan Sep 16 '17 at 19:12
• For continuous random variable, view $P(d_i|H_0)$ as a density, rather than a probability. I would prefer writing $f$ rather than $P$. – Siong Thye Goh Sep 16 '17 at 19:15