Interpretation of covariance term in loss function

Question

We have an $N\times 1$ vector containing some experimental values $y$, an $N\times 1$ vector $\hat{y}$ containing some predicted values, and an $N\times N$ covariance matrix $V_y$ for the experimental data. As a measure of how bad $\hat{y}$ is, we compute:

$$l(\hat{y})=(y-\hat{y})^TV_y^{-1}(y-\hat{y})$$

I am not a statistician, and the only thing that I really recognize is that $l:\mathbb{R}^N\to\mathbb{R}_{\geq 0}$ is a norm which measures the distance between $y$ and $\hat{y}$. Of course, if you delete the $V_y^{-1}$ term, we are just getting the euclidean distance on the nose.

Here is my question: What does $V_y^{-1}$ do to alter the euclidean norm, and how do we interpret (and justify) its inclusion in this loss function?

Thank you for your time.

Answer 1

The expression you have here is equivalent to squared Mahalanobis distance; a hand-wavey, intuitive explanation for Mahalanobis distance is that it is the multidimensional generalization of the z-score ($\frac{x - \mu}{\sigma}$ in one dimension). When we aren't calculating this in terms of the distance to the population parameter $\mu$, Mahalanobis distance is a proxy for the dissimilarity of two vectors given that $V_y = V_{\hat{y}}$, which may or may not be a reasonable assumption, depending on your problem.

Indeed, $(y - \hat{y})^\top (y-\hat{y})$ would give you squared Euclidean distance between vectors $y$, $\hat{y}$. So let's write out, in summation form, what this covariance matrix term is actually doing. For notational simplicity I'm going to define $A = V^{-1}_y $, and $d = y-\hat{y}$ $d$ for "difference" of vectors). Then

$$(y - \hat{y})^\top V^{-1}_y (y-\hat{y}) = d^\top A d = \sum_{i=1}^n \sum_{j=1}^n d_iA_{ij}d_j.$$

Compare this to squared Euclidean distance in summation form:

$$d^\top d = \sum_{i=1}^n d_i^2.$$

We can further split the top summation: $$\sum_{i=1}^n \sum_{j=1}^n d_iA_{ij}d_j = \sum_{i=j} d_i^2A_{ii} + \sum_{i \neq j}d_iA_{ij}d_j.$$

Note that the first term in the final expression looks very similar to squared Euclidean distance. For the case of a diagonal covariance matrix, i.e. each position of the vector is uncorrelated (important: this DOES NOT necessarily mean independent), the inverse is simply the reciprocal of the diagonal elements, hence

$$\sum_{i=j} d_i^2A_{ii} + \sum_{i \neq j}d_iA_{ij}d_j = \sum_{i=j} d_i^2A_{ii} = \sum_{i=j} \frac{d_i^2}{(V_y)_{ii}};$$

that is; this is the element-wise sum of squares of differences divided by variance. So there is a direct "normalizing" effect; i.e., the contribution of particular elements to the squared Euclidean distance is divided by the variance of that element.

In the case of the non-diagonal covariance matrix, we have the additional $$\sum_{i \neq j}d_iA_{ij}d_j$$ term to worry about. Reasoning formally about the off-diagonal elements in this case is much harder. However, using the intuition about inverse covariance matrices provided here, we see that there is a similar normalization effect proportional to the covariance between $d_i, d_j$. Alternately, this answer provides a more formal treatment of the off-diagonal elements.

Tl;dr $V_y^{-1}$ has a "normalizing" effect.