# hypothesis testing (with many continuous variables)

I have two independent sets of subjects, $A$ and $B$. Each subject has 20 independent continuous variables, e.g., attributes of each person.

Subject  height  BMI    IQ    etc.
001      1.50    18.0   114   ...
002      1.65    28.4   97    ...
003      1.64    20.4   125   ...
...      ...     ...    ...   ...


I want to know if the sets $A$ and $B$ are statistically similar (means of each variable) - what test should I use?

While a t-test can be performed on each variable separately, the number of variables that reject the null hypothesis will be itself drawn from a Poisson distribution. It sounds clumsy to do a hypothesis test on the outcomes of hypothesis tests, so is there a particular test/method that I should be using?

For a given variable a t-test can be performed: a confidence level (e.g., 95%) is defined, which sets the arbitrary threshold for which we are willing to reject the null hypothesis that the means of $A$ and $B$ are indistinguishable.

Now take these two hypothetical scenarios:

1. if the null hypothesis were true for a given variable, and a separate t-test were performed on 20 randomly drawn samples, then one of those samples would be expected to yield a Type I error (i.e., we observe a significant difference even though we know a priori that the null hypothesis is true).
2. if the null hypothesis were true for all 20 variables, and a separate $t$-test were performed on each of the 20 variables, then one of those 20 variables would be expected to yield a Type I error.

I am more interested in point (2)

When the null hypothesis is assumed to be true, a given variable will 'pass' a t-test (fail to reject the null) with a probability equal to the confidence level that was chosen. Therefore, when running a number of t-tests on an ensemble of $n$ variables, each variable has a probability $p_\mathsf{reject}$ of rejecting the null - i.e., each t-test is essentially a Bernoulli trial. It follows that the total number of observed 'passed' tests must be drawn from a Poisson distribution such that:

$$n_\mathsf{fail,obs} = \mathsf{Pois}(np_\mathsf{reject})$$

where for $n=20$ variables and $p_\mathsf{reject}=0.05$, the expected number of 'failed' tests is $n_\mathsf{fail,obs}=1.0$. The distribution is more important: the proportion of t-tests that would yield $x$ or fewer 'fails' would be:

x   proportion of Pois <= x
0   36.8%
1   73.6%
2   92.0%
3   98.1%
4   99.6%


Therefore, I would need at least 3/20 of the variables to 'fail' in order to confidently (95%) reject the null hypothesis that the null hypothesis is true for each variable.

Is all of this necessary?

If not, how is the variability taken into account? (r.e., the number of variables that 'pass' the test)

• They're not measured outcomes, but rather attributes. My aim is to set up a control and a test cell, and to verify that they are as similar as possible across all of the subjects' 20 attributes before proceeding to executing the test. How the subjects are assigned into group $A$ or $B$ is irrelevant for now (random selection, greedy method, etc.), but what I would like advice on is: how can I confirm that the control and test groups are similar when there are so many attributes to consider? – Ben Mar 21 '18 at 9:42