I was wondering, whether there is a possible interpretation of the KL-Divergence between sample and true distribution in terms of probabilities. E.g. given $P=\mathcal{N}\left(\mu,\Sigma\right)$ and $Q=\mathcal{N}\left(\hat{\mu},\hat{\Sigma}\right)$ we have \begin{equation} D_{KL}\left(P||Q\right)=\frac{1}{2}\left[\log\frac{|\hat{\Sigma}|}{|\Sigma|} - d + \text{tr} \{ \hat{\Sigma}^{-1}\Sigma \} + (\hat{\mu} - \mu)^T \hat{\Sigma}^{-1}(\hat{\mu} - \mu)\right] \end{equation} My question refers to the fact that the (scaled) Mahalanobis Distance (which is part of the above) \begin{equation} \frac{T(T-N)}{(T-1)N}(\hat{\mu} - \mu)^T \hat{\Sigma}^{-1}(\hat{\mu} - \mu) \end{equation} has an $F$-distribution with $(N,T)$ degrees of freedom (where $N=$ dimension, $T=$ sample size) and therefore yields a probabilistic "value" to judge how "far" $\hat{\mu}$ is away from $\mu$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.