In Bishop's PRML chapter 3, he presents the following approximation for the marginal likelihood in equation 3.72

$\ln P(D) \approx \ln P(D| \boldsymbol{w}_{map}) + \ln M\frac{\Delta w_{posterior}}{\Delta w_{prior}}$

Where $D$ is the data, and $\boldsymbol{w}_{map}$ is a vector of model parameters around which there is a peak in the posterior. This is all dependent one of several models $M_i$, but that notation is dropped to reduce clutter

We can ignore the second term on the right-hand side for now. I want to understand something he says about the first term, $P(D|\boldsymbol{w}_{map})$

First, he notes that for a flat prior, the $\ln{P(D|\boldsymbol{w}_{map})}$ corresponds to the log-likelihood, which makes sense (although I don't see the dependence on the prior's shape).

My trouble starts with his second claim about $\ln{P(D|\boldsymbol{w}_{map})}$. He states

"As we increase the complexity of the model, the first term will typically decrease, because a more complex model is better able to fit the data..."

Why would the likelihood decrease with a better fit? The log is a monotonic increasing function of its input and my understanding of the likelihood is that it will increase with better fit. Why then, as we add parameters and increase the complexity of the model, would we see the likelihood decrease?



1 Answer 1


This is a typo: the first term increases with $M$ and the second term decreases with $M$ since $$ \frac{\Delta w_{posterior}}{\Delta w_{prior}}<1 $$ In the Errata published in 2011 by Svensén & Bishop, the exchange between increasing and decreasing is acknowledged:

enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.