1
$\begingroup$

Suppose that we have samples from $m$ different $p$-dimensional normal multivariate distributions, where they share a common covariance matrix $\Sigma$ but the mean vectors may be different for each population. That is, the distribution of the populations are $\mathrm{N}_p(\mu_i, \Sigma)$ , for $i = 1,2, \ldots, m.$ Each sample is made up of $n_i$ observations, depending on the group.

So, $\mathbf{y}_{ij}$ is observation $j$ (for $j = 1,\ldots,n_i$) coming from group $i$ (for $i = 1,\ldots,m$). Notice that $\mathbf{y}_{ij}$ is a $p-$dimensional vector that comes from the multivariate normal distribution $\mathrm{N}_p(\mu_i, \Sigma)$.

An estimator for the covariance matrix for all of the data (ignoring the potential difference between groups) is $$\mathbf{S}=\frac{1}{N-1} \sum_{i=1}^m \sum_{j=1}^{n_i}\left(\mathbf{y}_{i j}-\bar{\mathbf{y}}\right)\left(\mathbf{y}_{i j}-\bar{\mathbf{y}}\right)^{\prime} = \dfrac{SS_T}{N-1},$$ where $N= \sum_{i=1}^m n_i$ is the total number of observations and $SS_T$ refers to the total variation (the sum of squares). Now, the total variation $SS_T$ can be broken down into the within-group and between-group variation components:

\begin{align} \tag{1} SS_T= & \sum_{i=1}^m \sum_{j=1}^{n_i}\left(\mathbf{y}_{i j}-\bar{\mathbf{y}}_i\right)\left(\mathbf{y}_{i j}-\bar{\mathbf{y}}_i\right)^{\prime}+ \sum_{i=1}^m n_i\left(\bar{\mathbf{y}}_i-\bar{\mathbf{y}}\right)\left(\bar{\mathbf{y}}_i-\bar{\mathbf{y}}\right)^{\prime} =SS_W + SS_B, \end{align} where $SS_W$ corresponds to the within-group variation and $SS_B$ to the between-batch variation. From here, a weighted average matrix can be used for the within-group covariance matrix (especially useful if the mean vectors of the distributions are very different): $$\mathbf{S}_W=\frac{\sum_{i=1}^m\left(n_i-1\right) \mathbf{S}_i}{N-m}=\frac{SS_W}{N-m},\tag{2}$$ where $\mathbf{S}_i$ is the covariance matrix estimate from group $i$. This makes sense to me since it's just a weighted average of individual covariance matrices, so it will only model the within-group variation. Similarly, a between-group covariance matrix can be defined as $$\mathbf{S}_B = \dfrac{SS_B}{m-1} = \frac{\sum_{i=1}^m n_i\left(\bar{\mathbf{y}}_i-\bar{\mathbf{y}}\right)\left(\bar{\mathbf{y}}_i-\bar{\mathbf{y}}\right)^{\prime}}{m-1}. \tag{3}$$

The formulas make sense to me, at least mathematically. The total variation can be broken down into its between and within-group components as in Equation (1). Intuitively, using the within-group covariance estimator $S_W$ makes sense, since we are averaging individual covariance matrices, and giving a larger weight to groups with more observations. This would be a good estimate for the matrix $\Sigma$.

However, I don't understand the intuition behind Equation (3), because of the term $n_i$. If the term were not there, the expression would look like this: $$\mathbf{S}_B^* = \frac{\sum_{i=1}^m \left(\bar{\mathbf{y}}_i-\bar{\mathbf{y}}\right)\left(\bar{\mathbf{y}}_i-\bar{\mathbf{y}}\right)^{\prime}}{m-1}. $$ and I could explain it as the covariance matrix of the group means. Thus, the diagonal of $\mathbf{S}_B^*$ would correspond to the between-group variances for each of the $p$ variables or characteristics being modelled. Adding $n_i$ could be understood as a weighting mechanism (where groups with more observations should be considered more, I understand). But then, what is the intepretation of the resulting matrix $\mathbf{S}_B$? What does each entry tell us? Notice that $\mathbf{S}$ and $\mathbf{S}_W$ have $N$ in their denominator, while $\mathbf{S}_B$ does not. This means that, the bigger the groups $n_i$, the bigger the entries of $\mathbf{S}_B$. If a process has a specific between-group variation, why would making groups bigger affect those values?

$\endgroup$

1 Answer 1

2
$\begingroup$

When considering the computationally efficient forms, let $L$ $(l=1,2,\ldots,L)$ be the number of treatment groups, $\mathbf{B}$ be the variable-by-variable $p \times p$ between-group sum of squares matrix and $\mathbf{W}$ be the $p \times p$ within-group sum of squares matrix [1]. The $jk$th element of these matrices are found as \begin{equation} \begin{split} b_{jk}&=\sum_{l=1}^{L}\frac{1}{n_{l}}T_{jl}T_{kl}-\frac{1}{n}G_j G_k\\ w_{jk}&=\sum_{l=1}^{L} \sum_{i=1}^{n_{l}} y_{ijl}y_{ikl} - \sum_{l=1}^{L} \frac{1}{n_{l}}T_{jl}T_{kl},\\ \end{split} \end{equation} where $y_{ijl}$ is the $i$th value of variable $j$ in treatment group $l$, $T_{jl}=\sum_i^{n_{l}}y_{ijl}$ is the sum of all values of variable $j$ in group $l$, $G_j= \sum_{l}^{L} T_{jl}$ is the grand total of all observations on variable $j$, and $n=n_1+n_2+ \cdots + n_{L}$.

An equivalent form of the F-test ($H_0: \boldsymbol{\mu}_1=\boldsymbol{\mu}_2= \cdots = \boldsymbol{\mu}_p$) is based, in part, on Wilk's Lambda given as \begin{equation} \Lambda(y_1,y_2,\ldots,y_p) = \frac{|{\bf W}|}{|{\bf B}| + |{\bf W}|} = \prod_{j=1} ^p \left( \frac{1}{1 + \lambda_j} \right), \end{equation} where $\lambda_1, \lambda_2, \ldots, \lambda_p $ are the eigenvalues of $\mathbf{W}^{-1}\mathbf{B}$ [2]. Although the matrix $\mathbf{W}^{-1}\mathbf{B}$ is square, it is nevertheless non-symmetric such that elements in the upper triangular are not a mirror reflection of elements in the lower triangular. Since the matrix $\mathbf{W}^{-1}\mathbf{B}$ is not symmetric, we must resort to using, for example, the generalized eigenvalue problem.

The answer to your question about MANOVA and $\mathbf{W}^{-1}\mathbf{B}$ is no different from the $F$-statistic $F=MST/MSE$ for univariate ANOVA (or $F=MSR/MSE$ for regression) -- essentially equal to $\mathbf{B}/\mathbf{W}$. That is, you will obtain a greater $F$ statistic if the between-group variance (based on group means minus the grand mean) exceeds the within-group variance (based on within group residuals -- i.e. noise). So the diagonal values $b_{jj}$ are $SST_j$ and diagonal values $w_{jj}$ are $SSE_j$.

Lastly, whenever there's a sample size in the numerator, it denotes a pooled variance, since the vanilla-flavored variance equation is typically scaled by $1/n$.

References:

  1. D.F. Morrison. The Multivariate Analysis of Variance, Chap. 5. In: Multivariate Statistical Methods, 3rd Ed. New York (NY), McGraw-Hill, 1990.

  2. R.A. Johnson, D.W. Wichern. Comparision of Several Multivariate Means, Chap. 6. In: Applied Multivariate Statistical Methods, 4th Ed. Upper Saddle River (NJ), Prenctice-Hall, 1998.

$\endgroup$
2
  • $\begingroup$ (+1) Thanks for the good answer and references. I understand that the sum of squares matrices $\mathbf{W}$ and $\mathbf{B}$ can be used in an F test; and I understand that $n_i$ in the numerator is used for weighting the estimates. However, my lack of understanding is regarding the intuition behind the between-group covariance matrix $\mathbf{S}_B$ that I proposed. Using your notation, $\mathbf{S}_B = \mathbf{B}/(N-K)$. While the diagonal of $\mathbf{B}$ is the "treatments sum of squares for individual response variables", what do the diagonal elements intuitively mean in $\mathbf{S}_B $? $\endgroup$
    – Bergson
    Feb 9 at 14:37
  • $\begingroup$ See updated answer, with a slight change of notation for clarity, and description of what the diagonal elements represent. $\endgroup$
    – wjktrs
    Feb 10 at 19:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.