# Understanding how the determinant of the multidimensional normal likelihood can overrule the prior probability

I am doing Bayesian inference. I have a normal prior probability distribution of some theoretical parameter $$\theta$$ and I am trying to update my knowledge of $$\theta$$ using some data $$D$$ and a model $$M(\theta)$$ using Markov chain Monte Carlo.

The likelihood is given by a multidimensional Normal:

$$\begin{equation} \mathcal{L}(\boldsymbol\theta | \boldsymbol D) = \frac{\exp\left(-\frac 1 2 ({M(\theta)}-{\boldsymbol\mu})^\mathrm{T}{\boldsymbol\Sigma}^{-1}({M(\theta)}-{\boldsymbol\mu})\right)}{\sqrt{(2\pi)^k|\boldsymbol\Sigma|}} \end{equation}$$

Here $$k$$ is the number of measurements (in my case 25), $$\mu$$ are the mean values of the measurements, and $$\Sigma$$ is the covariance matrix of those measurements.

The crux of my question concerns $$\det \Sigma$$. My data are highly correlated and so the covariance matrix has an interesting structure. It turns out that due to this structure the determinant of the covariance matrix is very nearly zero (its actual computed value is $$10^{-147}$$). Since it is so near to zero, the likelihood function obtains very large values relative to the prior. The result is the prior becomes irrelevant.

Basically I am trying to understand this behavior. Is this normal, or does it indicate that something is wrong? Why does the determinant dominate over everything? Is there some intuition to understand what is going on here?

The absolute value of the likelihood, whether it is $$10^{100}$$ or $$10^{-200}$$, has no relevance for Bayesian inference. (Think of the extreme case when the likelihood function is constant in the parameter $$\boldsymbol\theta$$.) What matters is how this likelihood function varies with the parameter $$\theta$$, since the posterior density is a probability density $$\pi(\boldsymbol\theta|\boldsymbol D,M)=\dfrac{\pi(\boldsymbol\theta)\mathcal{L}(\boldsymbol\theta | \boldsymbol D) }{\int\pi(\boldsymbol\theta)\mathcal{L}(\boldsymbol\theta | \boldsymbol D)\,\text{d}\boldsymbol\theta }$$ When $$\begin{equation} \mathcal{L}(\boldsymbol\theta | \boldsymbol D) = \exp\left(-\frac 1 2 ({M(\theta)}-{\boldsymbol\mu})^\mathrm{T}{\boldsymbol\Sigma}^{-1}({M(\theta)}-{\boldsymbol\mu})\right)\Big/ {\sqrt{(2\pi)^k|\boldsymbol\Sigma|}}\end{equation}$$ \begin{align*}\pi(\boldsymbol\theta|\boldsymbol D,M)&=\dfrac{\pi(\boldsymbol\theta) \exp\left(-\frac 1 2 ({M(\theta)}-{\boldsymbol\mu})^\mathrm{T}{\boldsymbol\Sigma}^{-1}({M(\theta)}-{\boldsymbol\mu})\right)\Big/ {\sqrt{(2\pi)^k|\boldsymbol\Sigma|}}}{\int\pi(\boldsymbol\theta) \exp\left(-\frac 1 2 ({M(\theta)}-{\boldsymbol\mu})^\mathrm{T}{\boldsymbol\Sigma}^{-1}({M(\theta)}-{\boldsymbol\mu})\right)\Big/ {\sqrt{(2\pi)^k|\boldsymbol\Sigma|}} \,\text{d}\boldsymbol\theta}\\ &=\dfrac{\pi(\boldsymbol\theta) \exp\left(-\frac 1 2 ({M(\theta)}-{\boldsymbol\mu})^\mathrm{T}{\boldsymbol\Sigma}^{-1}({M(\theta)}-{\boldsymbol\mu})\right)}{\int\pi(\boldsymbol\theta) \exp\left(-\frac 1 2 ({M(\theta)}-{\boldsymbol\mu})^\mathrm{T}{\boldsymbol\Sigma}^{-1}({M(\theta)}-{\boldsymbol\mu})\right)\,\text{d}\boldsymbol\theta} \end{align*} does not depend on $$|\boldsymbol\Sigma|$$ at all.
• OK, that's very interesting. So that rules out $|\Sigma|$ for being a cause of any concern. I am not really understanding how the absolute value of the likelihood has no relevance though. Obviously I am computing all of this numerically, and I find that the value of my likelihood at each iteration is enormously larger than the value of my prior. This gives the impression that changes to the likelihood is far more important than changes to the prior, no? – rhombidodecahedron Aug 25 '19 at 10:46
• It is probably not needed to be said but I am only computing $\log \pi(\theta) + \log \mathcal L(\theta | D)$ and then inspecting these two parts. I find a log priors of about 4 and log likelihoods of about 50. – rhombidodecahedron Aug 25 '19 at 11:07
• Just think about it: if the likelihood is constant in $\theta$, that it is large or small relative to the values of the prior does not matter. – Xi'an Aug 25 '19 at 11:40