Why does Fisher use covariance when only variance is needed?

Question

With reference to the following image from here: (can not inline it due to unsupported format) https://wikimedia.org/api/rest_v1/media/math/render/svg/9af8aa035642689bb2004047416b069a15406447

If we are interested in maximizing the gap between the data subsets associated with the same labels divided by the sum of the variance of these data subsets, why are we using the covariance matrix as opposed to the variance?

Another example where I see this apart from the wikipedia page is here, specifically this image:

The numerator of the not-simplified score function makes sense (applying linear transformation to means and computing their distance, which we want to maximize). However, I don't understand a single part of the denominator (this includes not understanding why we use covariance instead of variance and why we multiply transpose of transformation with covariance and transformation.

Why do we use covariance instead of variance in the fisher score function, and why is the covariance surrounded by the transpose of the transformation and the transformation?

Answer 1

In this problem you have what is essentially paired-data, with an equal number of observations of features from two objects/classes, where values in the classes can be correlated with one another (i.e., they are not assumed to be statistically dependent). Because correlation between features can occur, the overall variance of any linear combination of the feature values will depend on their correlation. This means that we need to know the variances of the individual features, but also the covariance between them. This is the reason we use a whole covariance matrix in this case.

In this case, each random vector $\mathbf{X}_i$ has some mean vector $\boldsymbol{\mu}_i$ and some covariance matrix $\mathbf{C}_i$. The latter is a full specification of the variances of each feature in the random vector, and also the covariances between the features in the random vector (from which we get the correlations of the features). A "linear discriminant" is effectively just a measure that aggregates the features by some linear function, so that we can compare an aggregate measure across multiple objects/classes. If we use the linear discriminant $Z(\mathbf{X}_i) = \boldsymbol{\beta}^\text{T} \mathbf{X}_i = \sum_k \beta_k X_{i,k}$ then the mean and variance of this "discriminant" can be found using the ordinary rules of mean and variance of linear functions:

$$\begin{equation} \begin{aligned} \mathbb{E}(Z(\mathbf{X}_i)) &= \mathbb{E}(\boldsymbol{\beta}^\text{T} \mathbf{X}_i) = \boldsymbol{\beta}^\text{T} \mathbb{E}(\mathbf{X}_i) = \boldsymbol{\beta}^\text{T} \boldsymbol{\mu}_i = \sum_k \beta_k \mu_k, \\[8pt] \mathbb{V}(Z(\mathbf{X}_i)) &= \mathbb{V}(\boldsymbol{\beta}^\text{T} \mathbf{X}_i) = \boldsymbol{\beta}^\text{T} \mathbb{V}(\mathbf{X}_i) \boldsymbol{\beta} = \boldsymbol{\beta}^\text{T} \mathbf{C}_i \boldsymbol{\beta} = \sum_k \sum_l \beta_k \beta_l C_{i,k,l}. \end{aligned} \end{equation}$$

From these formulae we see that the variance of the discriminant $Z$ depends on the covariance matrix $\mathbf{C}_i$ and is a quadratic form. This is merely a result of the fact that when individual features are correlated, the overall variance of the "linear discriminant" is affected by these correlations. If you would like to learn more about this, I would suggest learning a bit about probability results for random vectors, and in particular, look at moment rules for linear functions. The variance of a linear combination of random variables is a quadratic form, as in the above formula. Hopefully this explanation gives you a starting point for this inquiry.