Chapter 13 of Kevin Murphy's book Machine Learning: A Probabilistic Perspective discusses Sparse Linear Models. After a short introduction on the benefits of sparse models, he introduces the following problem:

enter image description here

How does he derive equation 13.1 above? i.e. why does it take that form, and what is $f$ supposed to represent here?

  • 1
    $\begingroup$ It may lead somewhere to play with the algebra here. Notice that the two logs in the definition of $f$ can be combined (sum of logs is the log of a product). Subsequently, plugging this simplified version of $f$ into 13.1 might lead to an interpretable expression. The Law of Total Probability comes to mind. $\endgroup$ – zkurtz Oct 13 '13 at 21:24

The divisor is just a normalizing constant, so we can ignore it for the moment. If we plug in $f(\gamma)$ it simplifies to $p(D|\gamma)p(\gamma)$, by Bayes' rule is equal to $p(\gamma|D)p(D)$. Now since $p(D)$ isn't a function of $\gamma$ it falls into the normalizing constant. Thus it simplifies to $p(\gamma|D)$.

This expansion seems pointless until you realize that the $\gamma$ that $p(\gamma|D)$ is at its maximum is the same $\gamma$ that $f(\gamma)$ is at its minimum. So we can study $f(\gamma)$ by itself.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.