When I am reading Murphy's Machine Learning A Probabilistic Perspective. In chapter 3.2. I have some doubt.

I think the author want to express is two things. First, we can use Bayes formula to computer posterior, combine likelihood and prior. Second, if data is big enough, we can compute it directly. $1$ or $0$ depend on the hypothesis which maximum itself($p(h|\mathcal{D})$) is exist or not. I can't see these conclusion clearly. And I feel indigestion. the below is some detail I can't go through.

  • Likelihood. He give an formula:$$p(\mathcal{D}|h)=\left[\frac{1}{\text{size}(h)}\right]^N=\left[\frac{1}{|h|}\right]^N$$

    And the author say this is the probability of independently sampling $N$ items(with replacement) from $h$.

    Yes, if the probability mass function is uniform. this is equal to $\left[\frac{1}{|h|}\right]^N$. but I think the the notation $p(\mathcal{D}|h)$ is not reasonable. It represents the probability of event $\mathcal{D}$ when $h$ is happen. under the assumption pmf is uniform. it must equal to $$\frac{\text{size}(\mathcal{D})}{\text{size}(h)}$$.

  • Posterior , the formula is $$p(h|\mathcal{D})=\frac{p(\mathcal{D}|h)p(h)}{p(\mathcal{D})}$$ It is exactly Bayes formula. Then ,the author say if we have enough data, the prior is not important. the likelihood almost determine everything.then he give a formula: $$p(h|\mathcal{D})\rightarrow \delta_{\hat{h}^{MAP}}(h)$$ where $\hat{h}^{MAP} = \max_{h}p(h|\mathcal{D})$ and $\delta$ is Dirac measure. so does it mean if exist hypothesis $\hat{h}$ in $\{h\}$ maximum $p(h|\mathcal{D})$. then $p(h|\mathcal{D}) \rightarrow 1$?

    To find $\hat{h}$ maximum $p(h|\mathcal{D})$ is equal to maximum $\log p(\mathcal{D}|h) + \log h$. Then the author say if we have enough data, only to find $\hat{h}$ maximum $\log p(\mathcal{D}|h)$ is enough. Since the likelihood term depends exponentially on $N$, and the prior stays constant. But I don't think so. Intuitively, If we have a lot of data. the prior is not important. But for this formula, both $p(\mathcal{D}|h)$ and $p(h)$ are probability. they $\in[0,1]$. Although $\mathcal{D}$ is big. for different hypothesis $h$, $p(\mathcal{D}|h)$ is different. but it $\in [0,1]$ the same as $p(h)$. So I think we can't ignore $p(h)$.

  • $\endgroup$
    • 2
      $\begingroup$ There are definitely problems with some of the topics you raise (for me, that delta function is the worst). One thing that isn't clear though is this: what is your question? Your post contains more of a "thinking out loud" process rather than a specific question. You are kind of right about $ p (h) $ and the answer is "it depends" - sometimes it matters, sometimes not. If you think in terms of "logic" then the prior is important when it is " informative" (meaning it tells you something that the data/likelihood does not). $\endgroup$ Commented Jul 13, 2014 at 7:33
    • $\begingroup$ @probabilityislogic thanks for you advice. I think maybe I don't have the suffice background to read this book now. Now I am turn to read Pattern Recognition and Machine Learning. $\endgroup$
      – Laura
      Commented Jul 15, 2014 at 6:24


    Your Answer

    By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.