2
$\begingroup$

I am doing Bayesian inference. I have a normal prior probability distribution of some theoretical parameter $\theta$ and I am trying to update my knowledge of $\theta$ using some data $D$ and a model $M(\theta)$ using Markov chain Monte Carlo.

The likelihood is given by a multidimensional Normal:

\begin{equation} \mathcal{L}(\boldsymbol\theta | \boldsymbol D) = \frac{\exp\left(-\frac 1 2 ({M(\theta)}-{\boldsymbol\mu})^\mathrm{T}{\boldsymbol\Sigma}^{-1}({M(\theta)}-{\boldsymbol\mu})\right)}{\sqrt{(2\pi)^k|\boldsymbol\Sigma|}} \end{equation}

Here $k$ is the number of measurements (in my case 25), $\mu$ are the mean values of the measurements, and $\Sigma$ is the covariance matrix of those measurements.

The crux of my question concerns $\det \Sigma$. My data are highly correlated and so the covariance matrix has an interesting structure. It turns out that due to this structure the determinant of the covariance matrix is very nearly zero (its actual computed value is $10^{-147}$). Since it is so near to zero, the likelihood function obtains very large values relative to the prior. The result is the prior becomes irrelevant.

Basically I am trying to understand this behavior. Is this normal, or does it indicate that something is wrong? Why does the determinant dominate over everything? Is there some intuition to understand what is going on here?

$\endgroup$
6
$\begingroup$

The absolute value of the likelihood, whether it is $10^{100}$ or $10^{-200}$, has no relevance for Bayesian inference. (Think of the extreme case when the likelihood function is constant in the parameter $\boldsymbol\theta$.) What matters is how this likelihood function varies with the parameter $\theta$, since the posterior density is a probability density $$\pi(\boldsymbol\theta|\boldsymbol D,M)=\dfrac{\pi(\boldsymbol\theta)\mathcal{L}(\boldsymbol\theta | \boldsymbol D) }{\int\pi(\boldsymbol\theta)\mathcal{L}(\boldsymbol\theta | \boldsymbol D)\,\text{d}\boldsymbol\theta }$$ When \begin{equation} \mathcal{L}(\boldsymbol\theta | \boldsymbol D) = \exp\left(-\frac 1 2 ({M(\theta)}-{\boldsymbol\mu})^\mathrm{T}{\boldsymbol\Sigma}^{-1}({M(\theta)}-{\boldsymbol\mu})\right)\Big/ {\sqrt{(2\pi)^k|\boldsymbol\Sigma|}}\end{equation} \begin{align*}\pi(\boldsymbol\theta|\boldsymbol D,M)&=\dfrac{\pi(\boldsymbol\theta) \exp\left(-\frac 1 2 ({M(\theta)}-{\boldsymbol\mu})^\mathrm{T}{\boldsymbol\Sigma}^{-1}({M(\theta)}-{\boldsymbol\mu})\right)\Big/ {\sqrt{(2\pi)^k|\boldsymbol\Sigma|}}}{\int\pi(\boldsymbol\theta) \exp\left(-\frac 1 2 ({M(\theta)}-{\boldsymbol\mu})^\mathrm{T}{\boldsymbol\Sigma}^{-1}({M(\theta)}-{\boldsymbol\mu})\right)\Big/ {\sqrt{(2\pi)^k|\boldsymbol\Sigma|}} \,\text{d}\boldsymbol\theta}\\ &=\dfrac{\pi(\boldsymbol\theta) \exp\left(-\frac 1 2 ({M(\theta)}-{\boldsymbol\mu})^\mathrm{T}{\boldsymbol\Sigma}^{-1}({M(\theta)}-{\boldsymbol\mu})\right)}{\int\pi(\boldsymbol\theta) \exp\left(-\frac 1 2 ({M(\theta)}-{\boldsymbol\mu})^\mathrm{T}{\boldsymbol\Sigma}^{-1}({M(\theta)}-{\boldsymbol\mu})\right)\,\text{d}\boldsymbol\theta} \end{align*} does not depend on $|\boldsymbol\Sigma|$ at all.

$\endgroup$
  • $\begingroup$ OK, that's very interesting. So that rules out $|\Sigma|$ for being a cause of any concern. I am not really understanding how the absolute value of the likelihood has no relevance though. Obviously I am computing all of this numerically, and I find that the value of my likelihood at each iteration is enormously larger than the value of my prior. This gives the impression that changes to the likelihood is far more important than changes to the prior, no? $\endgroup$ – rhombidodecahedron Aug 25 at 10:46
  • $\begingroup$ Yes, I have understood that point about the determinant. I am just wondering more generally about the relative values of the prior and the likelihood though, since you have written "the absolute value of the likelihood has no relevance." It seems to me that if the likelihood is much larger than the prior then the prior becomes irrelevant, as even very improbable regions of the prior will contribute little compared to the very large values of the likelihood. Is that not the case? $\endgroup$ – rhombidodecahedron Aug 25 at 11:05
  • $\begingroup$ It is probably not needed to be said but I am only computing $\log \pi(\theta) + \log \mathcal L(\theta | D)$ and then inspecting these two parts. I find a log priors of about 4 and log likelihoods of about 50. $\endgroup$ – rhombidodecahedron Aug 25 at 11:07
  • 1
    $\begingroup$ Just think about it: if the likelihood is constant in $\theta$, that it is large or small relative to the values of the prior does not matter. $\endgroup$ – Xi'an Aug 25 at 11:40
  • 1
    $\begingroup$ Another way to make the point about the absolute value of the likelihood not mattering: consider a binomial sampling model with, say, a Beta prior (nice and conjugate); and then consider a negative binomial sampling model with the same Beta prior (still conjugate). Work through the math of a Bayesian update in both cases while preserving the normalizing constant of the sampling distribution for as long as possible. $\endgroup$ – Cyan Aug 29 at 13:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.