2
$\begingroup$

I am currently trying to solve an exercise (Information Theory, Inference, and Learning Algorithms, by David J.C. MacKay, ch 28 exercise 28.1)

The exercise is as follows: Random variables $x$ come independently from a probability distribution $P(x)$. According to model $H_{0}$, $P(x)$ is a uniform distribution.

\begin{equation} P(x|H_0) = \frac{1}{2} \qquad x \in (-1,1). \end{equation}

According to model $H_{1}$, $P(x)$ is a nonuniform distribution with an unknown parameter $m \in$ $(-1,1)$:

\begin{equation} P(x|m,H_{1}) = \frac{1}{2}(1+mx) \qquad x\in(-1,1). \end{equation} Given the data $D = \{0.3, 0.5, 0.7, 0.8, 0.9\}$, what is the evidence for $H_{0}$ and $H_{1}$?

The evidence $P(D|H_{0})$ is as follows:

Edit 1 (thanks to Siong Thye Goh): Let $D = \{d_{i}|i = 1,\dots,5\}$. Since we assume independence, we have

\begin{equation} P(D|H_{0}) = \prod_{i=1}^{5}P(d_{i}|H_{0}) = \frac{1}{2}^{5}. \end{equation}

Question 2: However, now I am stuck with $P(D|H_{1})$.

\begin{equation} P(D|H_{1}) = \int_{-1}^{1}P(D|H_{1},m)\cdot P(m|H_{1})dm. \end{equation}

In the book, it is stated that the evidence can be approximated by: \begin{equation} P(D|H_{1}) \approx P(D|m_{mp},H_{1})\cdot P(m_{mp}|H_{1})\sigma_{m|D}. \end{equation} Here,

  • $m_{mp}$ defines the most probable value for the parameter $m$.
  • $\sigma_{m|d}$ defines the posterior uncertainty in $m$, as a part of an occam factor.
  • $P(m_{mp}|H_{1})$ = $\frac{1}{\sigma_m}$, represents the range of values of $m$ that were possible a priori.

I'm guessing here that $\frac{1}{\sigma_{m}} = \frac{1}{2}$ since $m \in (-1,1)$ and that $P(d_{i}|m_{mp},H_{1}) = \frac{1}{2}(1+m_{mp}d_{i})$, so that the approximation for the evidence becomes

\begin{equation} P(d_{i}|H_{1}) \approx \frac{1}{2}(1+m_{mp}d_{i})\cdot \frac{\sigma_{m|D}}{2}. \end{equation}

But now:

  • what would $\sigma_{m|D}$ be in this case (if this even is the right approach)?
  • How do we evaluate what $m_{mp}$ is? Can this just be done via maximum likelihood, as we do in classical statistics?
|cite|improve this question|||||
$\endgroup$
  • 2
    $\begingroup$ There is a crucial piece of information not provided in the exercise: The prior distribution of $m$ (i.e. $P(m|H_{1})$). I would suggest using a uniform distribution, making $P(m|H_1)=1/2$. Now you have all ingredients to do the integration. No Laplace approximation needed. $\endgroup$ – COOLSerdash Sep 16 '17 at 21:16
  • $\begingroup$ I think the exercise was made like this one purpose so that there would be no prior, or else it would indeed be quite easy. At least that is what I can guess from the difficulty indicator next to the exercise. For now I will just mark this question as answered. $\endgroup$ – Stephan Sep 16 '17 at 22:13
3
$\begingroup$

Assuming the data are independent,

Let $D = \{ d_i |i = 1, \ldots, 5\}$,

$$P(D|H_0) = \prod_{i=1}^5P(d_i |H_0 ) = \left( \frac12 \right)^5$$

|cite|improve this answer|||||
$\endgroup$
  • $\begingroup$ Thank you for your answer, but I have one question about this. Since we assume that the models are continuous, wouldn't $P(d_{i}|H_{0}) = 0$ for all i, since the probability of any single point in a continuous distribution is infinitely small? $\endgroup$ – Stephan Sep 16 '17 at 19:12
  • $\begingroup$ For continuous random variable, view $P(d_i|H_0)$ as a density, rather than a probability. I would prefer writing $f$ rather than $P$. $\endgroup$ – Siong Thye Goh Sep 16 '17 at 19:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.