3
$\begingroup$

I've been reading "A primer of multivariate statistics" by Richard J. Harris, page 546, which shows how to derive the Hotelling $T^2$ statistic, after seeing this related but different question (I have the degrees of freedom $n_1$ and $n_2$ in mine).

The following is left as an exercise for the student as it is outside the scope of a course I did last year:

Let $Y=a^TX$, where $a$ is a constant vector and $X$ is a sample member. Also $\mathcal{N(a^T\mu,a^T\Sigma a)}$

Show that

$\displaystyle\max_{\substack{a}}{\space} t^2(a)=\displaystyle\max_{\substack{a}}\space\frac{n_1n_2[a^T(\bar{x}_1-\bar{x}_2)]^2}{(n_1+n_2)a^TS_Ua}=\frac{n_1n_2}{n_1+n_2}(\bar{x}_1-\bar{x}_2)^TS_U^{-1}(\bar{x}_1-\bar{x}_2)$

which is known as the two-sample Hotelling $T^2$ statistic.

My notes say that the maximum is achieved for:

$a^*=S_U^{-1}(\bar{x}_1-\bar{x}_2)$

I can't find a proof for this despite much searching and it's not something I can work out on my own.


My thinking is that we differentiate with respect to $a$, set equal to zero and solve for $a$, and then substitute back into the original?

I can't see how one would differentiate that though and if that is how a proof is done?

$\endgroup$
  • 1
    $\begingroup$ You need to define $t(a)$, link this to $Y$, $X$, and show where the two-sample problem comes from. Also, I would suspect that $\| a \| = 1$, which would make optimization problem somewhat more complicated. $\endgroup$ – StasK Apr 13 '13 at 16:15
  • $\begingroup$ @StasK The objective is invariant with respect to scaling $a$, so it makes no difference whether $a$ is standardized to unit length or not. Clair, exactly what statement are you trying to prove? There are three equalities in your question: any one of them could be what you are referring to. $\endgroup$ – whuber Jul 22 '13 at 23:11
3
$\begingroup$

Unless I am missing something, this can be seen from using Cauchy-Schwarz & Spectral Decomposition (as shown in pages 78-80 in Applied Multivariate Statistical Analysis, by Richard A. Johnson and Dean W. Wichern, 6th edition).

First Cauchy-Schwarz: For two $p \times 1$ vectors $\mathbf{b}$ and $\mathbf{d}$, $$ (\mathbf{b}' \mathbf{d})^2 \leq (\mathbf{b}' \mathbf{b})(\mathbf{d}' \mathbf{d}),$$ with equality iff $\mathbf{b} = c \, \mathbf{d}$ for some constant $c$.

If $\mathbf{B}$ is a symmetric positive definite matrix, using spectral-decomposition, define $\mathbf{B}^{1/2}$ and $\mathbf{B}^{-1/2}$ appropriately. Then, applying the C-S inequality above to the vectors $\mathbf{B}^{1/2}\mathbf{b}$ and $\mathbf{B}^{-1/2}\mathbf{d}$, one gets a generalized version: $$ (\mathbf{b}' \mathbf{d})^2 \leq (\mathbf{b}' \mathbf{B} \mathbf{b})(\mathbf{d}' \mathbf{B}^{-1}\mathbf{d}),$$ with equality iff $\mathbf{b} = c \, \mathbf{B}^{-1}\mathbf{d}$ for some constant $c$.

Now if $\mathbf{b} \neq \mathbf{0}$, and since $\mathbf{B}$ is symmetric positive definite, we can divide both sides by $\mathbf{b}' \mathbf{B} \mathbf{b}$ to get: $$ \frac{(\mathbf{b}' \mathbf{d})^2 } {\mathbf{b}' \mathbf{B} \mathbf{b}} \leq \mathbf{d}' \mathbf{B}^{-1}\mathbf{d}$$.

Appropriate replacements by corresponding vectors and matrices should give the result stated.

HTH.

$\endgroup$
  • $\begingroup$ The differentiation approach works, but for a formal proof, the second derivative test needs to be completed. That is, finding the stationary points from setting the first derivative (as in @shabbychef's answer) equal to zero does not guarantee that the maximum is achieved there. To show the maximum, then one has to show the Hessian is negative definite. The Cauchy-Shwartz approach I pointed out (with $\mathbf{b}=\mathbf{a}$, $\mathbf{d}=\bar{\mathbf{x}}_1 - \bar{\mathbf{x}}_2$, and $\mathbf{B}=\mathbf{S}$) illustrates what is going on with the sample geometry. $\endgroup$ – thjohnSyrUni Jul 24 '13 at 22:40
  • $\begingroup$ Also, note that the equality in the inequality holds if & only if $\mathbf{a}=c\mathbf{S}^{-1}\left(\bar{\mathbf{x}}_1 - \bar{\mathbf{x}}_2\right)$. This relates to the scale invariance @whuber mentions. $\endgroup$ – thjohnSyrUni Jul 24 '13 at 22:47
  • $\begingroup$ This result assumes that the covariance matrix is positive definite when it is actually positive semi-definite. So this result should only hold almost surely? $\endgroup$ – user321627 Nov 2 '16 at 2:36
1
$\begingroup$

yes, you simply differentiate with respect to $a$, set to zero, and back-substitute. First note that the objective, $t^2(a)$, is positively scale-invariant with respect to $a$. That is, for $k>0$, $t^2(ka) = t^2(a)$. This allows you to be somewhat sloppy when computing the derivative, since any (positive) constant can be dropped.

The derivative simply follows the quotient rule of calculus. The derivative (with respect to $a$) of the numerator of $t^2$ is, up to scale, $a^{\top}\left(\bar{x}_1 - \bar{x}_2\right)$. The derivative of the denominator is, up to scale, $S_U a$. When using the 'lo-dee-hi minus hi-dee-lo etc' rule, the derivative is something like $c_1 a^{\top}\left(\bar{x}_1 - \bar{x}_2\right) - c_2 S_U a$ for positive constants $c_i$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.