# Bayesian variable selection

Chapter 13 of Kevin Murphy's book Machine Learning: A Probabilistic Perspective discusses Sparse Linear Models. After a short introduction on the benefits of sparse models, he introduces the following problem:

How does he derive equation 13.1 above? i.e. why does it take that form, and what is $f$ supposed to represent here?

• It may lead somewhere to play with the algebra here. Notice that the two logs in the definition of $f$ can be combined (sum of logs is the log of a product). Subsequently, plugging this simplified version of $f$ into 13.1 might lead to an interpretable expression. The Law of Total Probability comes to mind. – zkurtz Oct 13 '13 at 21:24

The divisor is just a normalizing constant, so we can ignore it for the moment. If we plug in $f(\gamma)$ it simplifies to $p(D|\gamma)p(\gamma)$, by Bayes' rule is equal to $p(\gamma|D)p(D)$. Now since $p(D)$ isn't a function of $\gamma$ it falls into the normalizing constant. Thus it simplifies to $p(\gamma|D)$.
This expansion seems pointless until you realize that the $\gamma$ that $p(\gamma|D)$ is at its maximum is the same $\gamma$ that $f(\gamma)$ is at its minimum. So we can study $f(\gamma)$ by itself.