I have two independent sets of subjects, $A$ and $B$. Each subject has 20 independent continuous variables, e.g., attributes of each person.

Subject  height  BMI    IQ    etc.
001      1.50    18.0   114   ...
002      1.65    28.4   97    ...
003      1.64    20.4   125   ...
...      ...     ...    ...   ...

I want to know if the sets $A$ and $B$ are statistically similar (means of each variable) - what test should I use?

While a t-test can be performed on each variable separately, the number of variables that reject the null hypothesis will be itself drawn from a Poisson distribution. It sounds clumsy to do a hypothesis test on the outcomes of hypothesis tests, so is there a particular test/method that I should be using?

For a given variable a t-test can be performed: a confidence level (e.g., 95%) is defined, which sets the arbitrary threshold for which we are willing to reject the null hypothesis that the means of $A$ and $B$ are indistinguishable.

Now take these two hypothetical scenarios:

  1. if the null hypothesis were true for a given variable, and a separate t-test were performed on 20 randomly drawn samples, then one of those samples would be expected to yield a Type I error (i.e., we observe a significant difference even though we know a priori that the null hypothesis is true).
  2. if the null hypothesis were true for all 20 variables, and a separate $t$-test were performed on each of the 20 variables, then one of those 20 variables would be expected to yield a Type I error.

I am more interested in point (2)

When the null hypothesis is assumed to be true, a given variable will 'pass' a t-test (fail to reject the null) with a probability equal to the confidence level that was chosen. Therefore, when running a number of t-tests on an ensemble of $n$ variables, each variable has a probability $p_\mathsf{reject}$ of rejecting the null - i.e., each t-test is essentially a Bernoulli trial. It follows that the total number of observed 'passed' tests must be drawn from a Poisson distribution such that:

$$ n_\mathsf{fail,obs} = \mathsf{Pois}(np_\mathsf{reject}) $$

where for $n=20$ variables and $p_\mathsf{reject}=0.05$, the expected number of 'failed' tests is $n_\mathsf{fail,obs}=1.0$. The distribution is more important: the proportion of t-tests that would yield $x$ or fewer 'fails' would be:

x   proportion of Pois <= x
0   36.8%
1   73.6%
2   92.0%
3   98.1%
4   99.6%

Therefore, I would need at least 3/20 of the variables to 'fail' in order to confidently (95%) reject the null hypothesis that the null hypothesis is true for each variable.

Is all of this necessary?

If not, how is the variability taken into account? (r.e., the number of variables that 'pass' the test)


Sounds like a one-way MANOVA: you have a single factor (independent variable) that distinguishes observations into two groups (A and B), and 20 dependent variables. MANOVA will allow you to see how the combination of your dependent outcomes distinguish your groups. The dependent variables should be related conceptually.

  • $\begingroup$ I just looked up MANOVA and I gather that this is used when there are 2 or more dependent variables. My apologies - I should have made it explicitly clear that the variables in my question are independent (they are just attributes/features of each subject in the population, e.g., age, height, IQ, etc.) $\endgroup$ – Ben Mar 20 '18 at 14:01
  • $\begingroup$ I think I see what you're saying: you're treating the group selection as the independent variable, and then treating the person's attributes as dependent on which group they're in. Is that correct? $\endgroup$ – Ben Mar 20 '18 at 14:25
  • $\begingroup$ From what I understand you have one independent variable (group status) with two levels (group A or group B) from which you have measured outcomes on 20 dependent variables. The scores on these 20 variables presumably depend on which group they are sampled. MANOVA can be used if the dependent variables are conceptually related, and if you satisfy other assumptions of the MANOVA. If you are interested in all 20 differences, this can be followed up with 20 independent group t-tests, but you risk a type 1 error due to multiple comparisons unless you apply some kind of correction to alpha. $\endgroup$ – pomodoro Mar 20 '18 at 22:53
  • $\begingroup$ They're not measured outcomes, but rather attributes. My aim is to set up a control and a test cell, and to verify that they are as similar as possible across all of the subjects' 20 attributes before proceeding to executing the test. How the subjects are assigned into group $A$ or $B$ is irrelevant for now (random selection, greedy method, etc.), but what I would like advice on is: how can I confirm that the control and test groups are similar when there are so many attributes to consider? $\endgroup$ – Ben Mar 21 '18 at 9:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.