# Intuition behind between-group covariance matrix from MANOVA?

Suppose that we have samples from $$m$$ different $$p$$-dimensional normal multivariate distributions, where they share a common covariance matrix $$\Sigma$$ but the mean vectors may be different for each population. That is, the distribution of the populations are $$\mathrm{N}_p(\mu_i, \Sigma)$$ , for $$i = 1,2, \ldots, m.$$ Each sample is made up of $$n_i$$ observations, depending on the group.

So, $$\mathbf{y}_{ij}$$ is observation $$j$$ (for $$j = 1,\ldots,n_i$$) coming from group $$i$$ (for $$i = 1,\ldots,m$$). Notice that $$\mathbf{y}_{ij}$$ is a $$p-$$dimensional vector that comes from the multivariate normal distribution $$\mathrm{N}_p(\mu_i, \Sigma)$$.

An estimator for the covariance matrix for all of the data (ignoring the potential difference between groups) is $$\mathbf{S}=\frac{1}{N-1} \sum_{i=1}^m \sum_{j=1}^{n_i}\left(\mathbf{y}_{i j}-\bar{\mathbf{y}}\right)\left(\mathbf{y}_{i j}-\bar{\mathbf{y}}\right)^{\prime} = \dfrac{SS_T}{N-1},$$ where $$N= \sum_{i=1}^m n_i$$ is the total number of observations and $$SS_T$$ refers to the total variation (the sum of squares). Now, the total variation $$SS_T$$ can be broken down into the within-group and between-group variation components:

\begin{align} \tag{1} SS_T= & \sum_{i=1}^m \sum_{j=1}^{n_i}\left(\mathbf{y}_{i j}-\bar{\mathbf{y}}_i\right)\left(\mathbf{y}_{i j}-\bar{\mathbf{y}}_i\right)^{\prime}+ \sum_{i=1}^m n_i\left(\bar{\mathbf{y}}_i-\bar{\mathbf{y}}\right)\left(\bar{\mathbf{y}}_i-\bar{\mathbf{y}}\right)^{\prime} =SS_W + SS_B, \end{align} where $$SS_W$$ corresponds to the within-group variation and $$SS_B$$ to the between-batch variation. From here, a weighted average matrix can be used for the within-group covariance matrix (especially useful if the mean vectors of the distributions are very different): $$\mathbf{S}_W=\frac{\sum_{i=1}^m\left(n_i-1\right) \mathbf{S}_i}{N-m}=\frac{SS_W}{N-m},\tag{2}$$ where $$\mathbf{S}_i$$ is the covariance matrix estimate from group $$i$$. This makes sense to me since it's just a weighted average of individual covariance matrices, so it will only model the within-group variation. Similarly, a between-group covariance matrix can be defined as $$\mathbf{S}_B = \dfrac{SS_B}{m-1} = \frac{\sum_{i=1}^m n_i\left(\bar{\mathbf{y}}_i-\bar{\mathbf{y}}\right)\left(\bar{\mathbf{y}}_i-\bar{\mathbf{y}}\right)^{\prime}}{m-1}. \tag{3}$$

The formulas make sense to me, at least mathematically. The total variation can be broken down into its between and within-group components as in Equation (1). Intuitively, using the within-group covariance estimator $$S_W$$ makes sense, since we are averaging individual covariance matrices, and giving a larger weight to groups with more observations. This would be a good estimate for the matrix $$\Sigma$$.

However, I don't understand the intuition behind Equation (3), because of the term $$n_i$$. If the term were not there, the expression would look like this: $$\mathbf{S}_B^* = \frac{\sum_{i=1}^m \left(\bar{\mathbf{y}}_i-\bar{\mathbf{y}}\right)\left(\bar{\mathbf{y}}_i-\bar{\mathbf{y}}\right)^{\prime}}{m-1}.$$ and I could explain it as the covariance matrix of the group means. Thus, the diagonal of $$\mathbf{S}_B^*$$ would correspond to the between-group variances for each of the $$p$$ variables or characteristics being modelled. Adding $$n_i$$ could be understood as a weighting mechanism (where groups with more observations should be considered more, I understand). But then, what is the intepretation of the resulting matrix $$\mathbf{S}_B$$? What does each entry tell us? Notice that $$\mathbf{S}$$ and $$\mathbf{S}_W$$ have $$N$$ in their denominator, while $$\mathbf{S}_B$$ does not. This means that, the bigger the groups $$n_i$$, the bigger the entries of $$\mathbf{S}_B$$. If a process has a specific between-group variation, why would making groups bigger affect those values?

When considering the computationally efficient forms, let $$L$$ $$(l=1,2,\ldots,L)$$ be the number of treatment groups, $$\mathbf{B}$$ be the variable-by-variable $$p \times p$$ between-group sum of squares matrix and $$\mathbf{W}$$ be the $$p \times p$$ within-group sum of squares matrix [1]. The $$jk$$th element of these matrices are found as $$$$\begin{split} b_{jk}&=\sum_{l=1}^{L}\frac{1}{n_{l}}T_{jl}T_{kl}-\frac{1}{n}G_j G_k\\ w_{jk}&=\sum_{l=1}^{L} \sum_{i=1}^{n_{l}} y_{ijl}y_{ikl} - \sum_{l=1}^{L} \frac{1}{n_{l}}T_{jl}T_{kl},\\ \end{split}$$$$ where $$y_{ijl}$$ is the $$i$$th value of variable $$j$$ in treatment group $$l$$, $$T_{jl}=\sum_i^{n_{l}}y_{ijl}$$ is the sum of all values of variable $$j$$ in group $$l$$, $$G_j= \sum_{l}^{L} T_{jl}$$ is the grand total of all observations on variable $$j$$, and $$n=n_1+n_2+ \cdots + n_{L}$$.

An equivalent form of the F-test ($$H_0: \boldsymbol{\mu}_1=\boldsymbol{\mu}_2= \cdots = \boldsymbol{\mu}_p$$) is based, in part, on Wilk's Lambda given as $$$$\Lambda(y_1,y_2,\ldots,y_p) = \frac{|{\bf W}|}{|{\bf B}| + |{\bf W}|} = \prod_{j=1} ^p \left( \frac{1}{1 + \lambda_j} \right),$$$$ where $$\lambda_1, \lambda_2, \ldots, \lambda_p$$ are the eigenvalues of $$\mathbf{W}^{-1}\mathbf{B}$$ [2]. Although the matrix $$\mathbf{W}^{-1}\mathbf{B}$$ is square, it is nevertheless non-symmetric such that elements in the upper triangular are not a mirror reflection of elements in the lower triangular. Since the matrix $$\mathbf{W}^{-1}\mathbf{B}$$ is not symmetric, we must resort to using, for example, the generalized eigenvalue problem.

The answer to your question about MANOVA and $$\mathbf{W}^{-1}\mathbf{B}$$ is no different from the $$F$$-statistic $$F=MST/MSE$$ for univariate ANOVA (or $$F=MSR/MSE$$ for regression) -- essentially equal to $$\mathbf{B}/\mathbf{W}$$. That is, you will obtain a greater $$F$$ statistic if the between-group variance (based on group means minus the grand mean) exceeds the within-group variance (based on within group residuals -- i.e. noise). So the diagonal values $$b_{jj}$$ are $$SST_j$$ and diagonal values $$w_{jj}$$ are $$SSE_j$$.

Lastly, whenever there's a sample size in the numerator, it denotes a pooled variance, since the vanilla-flavored variance equation is typically scaled by $$1/n$$.

References:

1. D.F. Morrison. The Multivariate Analysis of Variance, Chap. 5. In: Multivariate Statistical Methods, 3rd Ed. New York (NY), McGraw-Hill, 1990.

2. R.A. Johnson, D.W. Wichern. Comparision of Several Multivariate Means, Chap. 6. In: Applied Multivariate Statistical Methods, 4th Ed. Upper Saddle River (NJ), Prenctice-Hall, 1998.

• (+1) Thanks for the good answer and references. I understand that the sum of squares matrices $\mathbf{W}$ and $\mathbf{B}$ can be used in an F test; and I understand that $n_i$ in the numerator is used for weighting the estimates. However, my lack of understanding is regarding the intuition behind the between-group covariance matrix $\mathbf{S}_B$ that I proposed. Using your notation, $\mathbf{S}_B = \mathbf{B}/(N-K)$. While the diagonal of $\mathbf{B}$ is the "treatments sum of squares for individual response variables", what do the diagonal elements intuitively mean in $\mathbf{S}_B$? Feb 9 at 14:37
• See updated answer, with a slight change of notation for clarity, and description of what the diagonal elements represent. Feb 10 at 19:09