With reference to the following image from here: (can not inline it due to unsupported format) https://wikimedia.org/api/rest_v1/media/math/render/svg/9af8aa035642689bb2004047416b069a15406447

If we are interested in maximizing the gap between the data subsets associated with the same labels divided by the sum of the variance of these data subsets, why are we using the covariance matrix as opposed to the variance?

Another example where I see this apart from the wikipedia page is here, specifically this image: fisher score function

The numerator of the not-simplified score function makes sense (applying linear transformation to means and computing their distance, which we want to maximize). However, I don't understand a single part of the denominator (this includes not understanding why we use covariance instead of variance and why we multiply transpose of transformation with covariance and transformation.

Why do we use covariance instead of variance in the fisher score function, and why is the covariance surrounded by the transpose of the transformation and the transformation?


1 Answer 1


In this problem you have what is essentially paired-data, with an equal number of observations of features from two objects/classes, where values in the classes can be correlated with one another (i.e., they are not assumed to be statistically dependent). Because correlation between features can occur, the overall variance of any linear combination of the feature values will depend on their correlation. This means that we need to know the variances of the individual features, but also the covariance between them. This is the reason we use a whole covariance matrix in this case.

In this case, each random vector $\mathbf{X}_i$ has some mean vector $\boldsymbol{\mu}_i$ and some covariance matrix $\mathbf{C}_i$. The latter is a full specification of the variances of each feature in the random vector, and also the covariances between the features in the random vector (from which we get the correlations of the features). A "linear discriminant" is effectively just a measure that aggregates the features by some linear function, so that we can compare an aggregate measure across multiple objects/classes. If we use the linear discriminant $Z(\mathbf{X}_i) = \boldsymbol{\beta}^\text{T} \mathbf{X}_i = \sum_k \beta_k X_{i,k}$ then the mean and variance of this "discriminant" can be found using the ordinary rules of mean and variance of linear functions:

$$\begin{equation} \begin{aligned} \mathbb{E}(Z(\mathbf{X}_i)) &= \mathbb{E}(\boldsymbol{\beta}^\text{T} \mathbf{X}_i) = \boldsymbol{\beta}^\text{T} \mathbb{E}(\mathbf{X}_i) = \boldsymbol{\beta}^\text{T} \boldsymbol{\mu}_i = \sum_k \beta_k \mu_k, \\[8pt] \mathbb{V}(Z(\mathbf{X}_i)) &= \mathbb{V}(\boldsymbol{\beta}^\text{T} \mathbf{X}_i) = \boldsymbol{\beta}^\text{T} \mathbb{V}(\mathbf{X}_i) \boldsymbol{\beta} = \boldsymbol{\beta}^\text{T} \mathbf{C}_i \boldsymbol{\beta} = \sum_k \sum_l \beta_k \beta_l C_{i,k,l}. \end{aligned} \end{equation}$$

From these formulae we see that the variance of the discriminant $Z$ depends on the covariance matrix $\mathbf{C}_i$ and is a quadratic form. This is merely a result of the fact that when individual features are correlated, the overall variance of the "linear discriminant" is affected by these correlations. If you would like to learn more about this, I would suggest learning a bit about probability results for random vectors, and in particular, look at moment rules for linear functions. The variance of a linear combination of random variables is a quadratic form, as in the above formula. Hopefully this explanation gives you a starting point for this inquiry.

  • $\begingroup$ Is the fact that V((B_T)(X_i)) equals (b_T)V(x_I)(b) because V(c(x_i)) equals (c^2)V(x_I), where V is the variance function and c is a constnat? In addition, can you provide an intuitive explanation for "Because correlation between features can occur, the overall variance of any linear combination of the feature values will depend on their correlation." Finally, can you clarify "In this case, each random vector Xi has some mean vector μi and some covariance matrix Ci"? (despite research, I can't figure out what random vector means here, can you rephrase the sentence)? $\endgroup$ Jun 20, 2018 at 19:16
  • $\begingroup$ The way I'm interpreting your random vector sentence, you are saying each vector has an associated mean and covariance, when a set of vectors have a mean and covariance, not per vector. $\endgroup$ Jun 20, 2018 at 19:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.