Understanding how the determinant of the multidimensional normal likelihood can overrule the prior probability

Question

I am doing Bayesian inference. I have a normal prior probability distribution of some theoretical parameter $\theta$ and I am trying to update my knowledge of $\theta$ using some data $D$ and a model $M(\theta)$ using Markov chain Monte Carlo.

The likelihood is given by a multidimensional Normal:

\begin{equation} \mathcal{L}(\boldsymbol\theta | \boldsymbol D) = \frac{\exp\left(-\frac 1 2 ({M(\theta)}-{\boldsymbol\mu})^\mathrm{T}{\boldsymbol\Sigma}^{-1}({M(\theta)}-{\boldsymbol\mu})\right)}{\sqrt{(2\pi)^k|\boldsymbol\Sigma|}} \end{equation}

Here $k$ is the number of measurements (in my case 25), $\mu$ are the mean values of the measurements, and $\Sigma$ is the covariance matrix of those measurements.

The crux of my question concerns $\det \Sigma$. My data are highly correlated and so the covariance matrix has an interesting structure. It turns out that due to this structure the determinant of the covariance matrix is very nearly zero (its actual computed value is $10^{-147}$). Since it is so near to zero, the likelihood function obtains very large values relative to the prior. The result is the prior becomes irrelevant.

Basically I am trying to understand this behavior. Is this normal, or does it indicate that something is wrong? Why does the determinant dominate over everything? Is there some intuition to understand what is going on here?

Answer 1

The absolute value of the likelihood, whether it is $10^{100}$ or $10^{-200}$, has no relevance for Bayesian inference. (Think of the extreme case when the likelihood function is constant in the parameter $\boldsymbol\theta$.) What matters is how this likelihood function varies with the parameter $\theta$, since the posterior density is a probability density $$\pi(\boldsymbol\theta|\boldsymbol D,M)=\dfrac{\pi(\boldsymbol\theta)\mathcal{L}(\boldsymbol\theta | \boldsymbol D) }{\int\pi(\boldsymbol\theta)\mathcal{L}(\boldsymbol\theta | \boldsymbol D)\,\text{d}\boldsymbol\theta }$$ When \begin{equation} \mathcal{L}(\boldsymbol\theta | \boldsymbol D) = \exp\left(-\frac 1 2 ({M(\theta)}-{\boldsymbol\mu})^\mathrm{T}{\boldsymbol\Sigma}^{-1}({M(\theta)}-{\boldsymbol\mu})\right)\Big/ {\sqrt{(2\pi)^k|\boldsymbol\Sigma|}}\end{equation} \begin{align*}\pi(\boldsymbol\theta|\boldsymbol D,M)&=\dfrac{\pi(\boldsymbol\theta) \exp\left(-\frac 1 2 ({M(\theta)}-{\boldsymbol\mu})^\mathrm{T}{\boldsymbol\Sigma}^{-1}({M(\theta)}-{\boldsymbol\mu})\right)\Big/ {\sqrt{(2\pi)^k|\boldsymbol\Sigma|}}}{\int\pi(\boldsymbol\theta) \exp\left(-\frac 1 2 ({M(\theta)}-{\boldsymbol\mu})^\mathrm{T}{\boldsymbol\Sigma}^{-1}({M(\theta)}-{\boldsymbol\mu})\right)\Big/ {\sqrt{(2\pi)^k|\boldsymbol\Sigma|}} \,\text{d}\boldsymbol\theta}\\ &=\dfrac{\pi(\boldsymbol\theta) \exp\left(-\frac 1 2 ({M(\theta)}-{\boldsymbol\mu})^\mathrm{T}{\boldsymbol\Sigma}^{-1}({M(\theta)}-{\boldsymbol\mu})\right)}{\int\pi(\boldsymbol\theta) \exp\left(-\frac 1 2 ({M(\theta)}-{\boldsymbol\mu})^\mathrm{T}{\boldsymbol\Sigma}^{-1}({M(\theta)}-{\boldsymbol\mu})\right)\,\text{d}\boldsymbol\theta} \end{align*} does not depend on $|\boldsymbol\Sigma|$ at all.