How to compare multiple Two-Sample Hotelling's T-squared tests performed on different sample sizes It's my first question, so please be patient with me.
I'm working on the comparison of multiple datasets of different lengths (60 to ~10000). They share a set of 9 features which I would like to use for comparison.
I'm interested in the question whether they are sampled from the same distribution.
My approach is, to pairwise compare the datasets using the two-sample Hotelling's T² test.
I assume that the datasets are independent as they come from different sources.
I'm not only interested in whether they are sharing the same mean, but also in the fact how different the distributions are. Therefore I wanted to take the values of the T² statistic as an indicator.
While testing my setup, I observed that for two fixed datasets, if I change the sample size that I draw from the datasets (same for both), the magnitude of the T^2 statistic changes.
$$
T^2 = \mathbf{(\bar{x}_1 - \bar{x}_2)}^T\{\mathbf{S}_p(\frac{1}{n_1}+\frac{1}{n_2})\}^{-1} \mathbf{(\bar{x}_1 - \bar{x}_2)}
$$
Since the sum of the inverse of the sample sizes is used in the computation of T², I guess that this is to be expected, but it leads me to my actual question.
Is it possible to meaningfully compare T² statistics that have been obtained from tests with different sample sizes?
 A: Assuming $X_{i1} \sim \mathcal N_p\left(\mu_1,\Sigma\right),i=1,\ldots,n_1$ and $X_{j2} \sim \mathcal N_p\left(\mu_2,\Sigma\right),j=1,\ldots,n_2$, with all $p$-variate random variables $X_{11},\ldots,X_{n_11},X_{12},\ldots,X_{n_22}$ independent and $H_0\!:\mu_1=\mu_2$, we have
$$
\frac{n_1n_2\left(n_1+n_2-2\right)}{\left(n_1+n_2\right)^2}\left(\bar x_1 - \bar x_2\right)^\top S^{-1}\left(\bar x_1 - \bar x_2\right)\mathrel{=:}T^2 \overset{H_0}{\sim}T^2_{p,\ n_1+n_2-2},
$$
where $S$ is a weighted mean of $S_1$ and $S_2$ defined by
$$
\begin{align}
S&=\frac{n_1S_1+n_2S_2}{n_1+n_2},\\
S_k&=\frac{1}{n_k}\sum_{i=1}^{n_k}\left(x_{ik}-\bar x_k\right)\left(x_{ik}-\bar x_k\right)^\top,\,k=1,2.
\end{align}
$$
Therefore, the degrees of freedom of the Hotteling $T^2$-distribution of the two-sample $T^2$
statistic depend on the sum of the sample sizes $n_1$ and $n_2$.
This means that for varying $n_1$ and $n_2$ the same value of $T^2$ doesn't have the same meaning unless $n_1+n_2$ stays constant.
You could, however, compare the p-values of the observed values of the $T^2$ statistics as they are uniformly distributed on $\left[0,1\right]$ under $H_0$, regardless of the sample sizes.
Also, it could be easier to work with the equivalent $F$ statistic which is given by
$$
\frac{\left(n_1+n_2-p-1\right)}{p\left(n_1+n_2-2\right)}T^2\mathrel{=:}F \overset{H_0}{\sim} F_{p,\ n_1+n_2-p-1}.
$$
