This post relates to my original question here, but this time focusing on a more fundamental misunderstanding of how MCMC actually works.

When using Bayesian MCMC for parameter inference with a single model, it is usual to ignore the "evidence" integral and write:

$$P(\theta | D) \propto P(D | \theta) P(\theta)$$

For numerical reasons, it is also common to work with logarithms:

$$ln[P(\theta | D)] \propto ln[P(D | \theta)] + ln[P(\theta)]$$

I've used this successfully on a number of occasions, typically using Python and emcee. However, for reasons explained in my other question, I've recently been wondering about "normalising" my likelihood by dividing it by the number of observations (I realise this probably sounds odd, but please see my other question for why I thought it was worth a try).

My likelihood assumes i.i.d. Gaussian errors, so my log-likelihood is just the sum of the individual data likelihoods. My prior is uniform and I'm currently using an improper log-prior function, which simply returns zero if the parameter values are "within range" and '-inf' otherwise.

I initially assumed that dividing by the number of data points wouldn't make any difference, because the RHS is still proportional to the log-posterior:

$$ln[P(\theta | D)] \propto \frac{1}{n}ln[P(D | \theta)] + ln[P(\theta)]$$

(My prior only ever returns 0 or '-inf', so multiplying that by $1/n$ makes no difference).

However, when I re-run my MCMC, I get very different results. As a sense check, I tried a simple OLS linear regression using the same code.

With my original (not normalised) likelihood, I get sensible results: the MCMC recovers the "true" parameters and the 95% CI broadly agrees with the Frequentist results from 'statsmodels':

Original (not normalised) likelihood

However, with the "normalised" likelihood, I get very different (and obviously wrong) results:

enter image description here

I think I understand why this is, at least from the point of view of a simple Metropolis algorithm. The probability of accepting the proposed jump, $x_{i+1}$, compared to the current location, $x_i$, is:

$$\alpha = \frac{P(x_{i+1})}{P(x_i)}$$

Which, re-writing in terms of my log-posterior, $L$, becomes:

$$\alpha = \frac{exp(L(x_{i+1}))}{exp(L(x_i))} = exp(L(x_{i+1}) - L(x_i))$$

On the other hand, for my "normalised" version, all the likelihoods are divided by $n$, so this becomes:

$$\alpha = [exp(L(x_{i+1}) - L(x_i))]^\frac{1}{n}$$

which is different, and might explain why my uncertainty bounds suddenly become much larger. (I understand that emcee is more sophisticated than just basic Metropolis, but I guess a similar principle applies?).

My question:

What am I missing/misunderstanding to reconcile these two results, please? On the one hand, it's OK to ignore the "normalising constant" in Bayes' equation, because anything proportional to the log-posterior is sufficient. On the other hand, multiplying the log-posterior by a constant $\frac{1}{n}$ seems to have a dramatic effect on my results?

Thank you!

  • 2
    $\begingroup$ If you look at the denominator in Bayes theorem on the log scale (which you're working on) it would involve subtracting a constant, not dividing by one. $\endgroup$ – Glen_b Mar 29 '19 at 16:30

Multiplying the likelihood by a constant makes no difference, as you state. You can take $\tilde L(x)=\frac1n L(x)$ with $K>0$ and obtain the same result.

When you take the log, you will then get $\log\tilde L(x) = \log L(x) -n$: on the log scale, you can add a constant, but not multiply by a constant.

Multiplying the log-likelihood by a constant $k$ (or equivalently raise the likelihood to the $k$th power) gives what is called a tempered likelihood. When $0<k<1$, the tempered likelihood is flatter than the likelihood. The mode is preserved, but the tails of the distribution are heavier.

| cite | improve this answer | |
  • $\begingroup$ Gah! Of course, apologies - basic algebra fail :-( I had a feeling it would be something blindingly obvious, but nevertheless spent half the afternoon tying myself in knots! Thanks to you and @Glen_b for putting me back on track! $\endgroup$ – JamesS Mar 29 '19 at 18:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.