# Interpretation of covariance term in loss function

We have an $$N\times 1$$ vector containing some experimental values $$y$$, an $$N\times 1$$ vector $$\hat{y}$$ containing some predicted values, and an $$N\times N$$ covariance matrix $$V_y$$ for the experimental data. As a measure of how bad $$\hat{y}$$ is, we compute:

$$l(\hat{y})=(y-\hat{y})^TV_y^{-1}(y-\hat{y})$$

I am not a statistician, and the only thing that I really recognize is that $$l:\mathbb{R}^N\to\mathbb{R}_{\geq 0}$$ is a norm which measures the distance between $$y$$ and $$\hat{y}$$. Of course, if you delete the $$V_y^{-1}$$ term, we are just getting the euclidean distance on the nose.

Here is my question: What does $$V_y^{-1}$$ do to alter the euclidean norm, and how do we interpret (and justify) its inclusion in this loss function?

• Could you explain how the predicted values depend on the experimental data, if at all?
– whuber
Jun 16 '20 at 17:01
• They don't in how I've set things up in the question. Just for the purposes of this problem, we might assume that $y$ is a physical measurement and $\hat{y}$ is a theoretically predicted value. Sometimes $\hat{y}$ can perform poorly, if not enough physics is taken into consideration. Jun 16 '20 at 17:04
• It is difficult to see what meaning this variance matrix could possibly have in the circumstances unless $\hat y$ were essentially constant. The reason is that the variance of the experimental data needn't have any relationship with the variance of the noise (except insofar as the data variance incorporates that of the noise and therefore must dominate it).
– whuber
Jun 16 '20 at 17:32

The expression you have here is equivalent to squared Mahalanobis distance; a hand-wavey, intuitive explanation for Mahalanobis distance is that it is the multidimensional generalization of the z-score ($$\frac{x - \mu}{\sigma}$$ in one dimension). When we aren't calculating this in terms of the distance to the population parameter $$\mu$$, Mahalanobis distance is a proxy for the dissimilarity of two vectors given that $$V_y = V_{\hat{y}}$$, which may or may not be a reasonable assumption, depending on your problem.

Indeed, $$(y - \hat{y})^\top (y-\hat{y})$$ would give you squared Euclidean distance between vectors $$y$$, $$\hat{y}$$. So let's write out, in summation form, what this covariance matrix term is actually doing. For notational simplicity I'm going to define $$A = V^{-1}_y$$, and $$d = y-\hat{y}$$ $$d$$ for "difference" of vectors). Then

$$(y - \hat{y})^\top V^{-1}_y (y-\hat{y}) = d^\top A d = \sum_{i=1}^n \sum_{j=1}^n d_iA_{ij}d_j.$$

Compare this to squared Euclidean distance in summation form:

$$d^\top d = \sum_{i=1}^n d_i^2.$$

We can further split the top summation: $$\sum_{i=1}^n \sum_{j=1}^n d_iA_{ij}d_j = \sum_{i=j} d_i^2A_{ii} + \sum_{i \neq j}d_iA_{ij}d_j.$$

Note that the first term in the final expression looks very similar to squared Euclidean distance. For the case of a diagonal covariance matrix, i.e. each position of the vector is uncorrelated (important: this DOES NOT necessarily mean independent), the inverse is simply the reciprocal of the diagonal elements, hence

$$\sum_{i=j} d_i^2A_{ii} + \sum_{i \neq j}d_iA_{ij}d_j = \sum_{i=j} d_i^2A_{ii} = \sum_{i=j} \frac{d_i^2}{(V_y)_{ii}};$$

that is; this is the element-wise sum of squares of differences divided by variance. So there is a direct "normalizing" effect; i.e., the contribution of particular elements to the squared Euclidean distance is divided by the variance of that element.

In the case of the non-diagonal covariance matrix, we have the additional $$\sum_{i \neq j}d_iA_{ij}d_j$$ term to worry about. Reasoning formally about the off-diagonal elements in this case is much harder. However, using the intuition about inverse covariance matrices provided here, we see that there is a similar normalization effect proportional to the covariance between $$d_i, d_j$$. Alternately, this answer provides a more formal treatment of the off-diagonal elements.

Tl;dr $$V_y^{-1}$$ has a "normalizing" effect.

• This term also appears in the Multivariate Normal log-likelihood, so there's another interpretation possible from there. Jun 16 '20 at 21:21
• Yes, indeed, the multivariate normal PDF replaces the $\frac{x - \mu}{\sigma}$ in the exponential with $(x-\mu)^\top \Sigma^{-1} (x - \mu)$; hence, $p(x)$ in a multivariate normal is a function of the Mahalanobis distance. Jun 16 '20 at 21:25
• Thank you for your thorough answer as well as the references. You expressed things very cleanly! Jun 17 '20 at 16:46