2
$\begingroup$

I have two independent sets of subjects, $A$ and $B$. Each subject has 20 independent continuous variables, e.g., attributes of each person.

Subject  height  BMI    IQ    etc.
001      1.50    18.0   114   ...
002      1.65    28.4   97    ...
003      1.64    20.4   125   ...
...      ...     ...    ...   ...

I want to know if the sets $A$ and $B$ are statistically similar (means of each variable) - what test should I use?

While a t-test can be performed on each variable separately, the number of variables that reject the null hypothesis will be itself drawn from a Poisson distribution. It sounds clumsy to do a hypothesis test on the outcomes of hypothesis tests, so is there a particular test/method that I should be using?

For a given variable a t-test can be performed: a confidence level (e.g., 95%) is defined, which sets the arbitrary threshold for which we are willing to reject the null hypothesis that the means of $A$ and $B$ are indistinguishable.

Now take these two hypothetical scenarios:

  1. if the null hypothesis were true for a given variable, and a separate t-test were performed on 20 randomly drawn samples, then one of those samples would be expected to yield a Type I error (i.e., we observe a significant difference even though we know a priori that the null hypothesis is true).
  2. if the null hypothesis were true for all 20 variables, and a separate $t$-test were performed on each of the 20 variables, then one of those 20 variables would be expected to yield a Type I error.

I am more interested in point (2)

When the null hypothesis is assumed to be true, a given variable will 'pass' a t-test (fail to reject the null) with a probability equal to the confidence level that was chosen. Therefore, when running a number of t-tests on an ensemble of $n$ variables, each variable has a probability $p_\mathsf{reject}$ of rejecting the null - i.e., each t-test is essentially a Bernoulli trial. It follows that the total number of observed 'passed' tests must be drawn from a Poisson distribution such that:

$$ n_\mathsf{fail,obs} = \mathsf{Pois}(np_\mathsf{reject}) $$

where for $n=20$ variables and $p_\mathsf{reject}=0.05$, the expected number of 'failed' tests is $n_\mathsf{fail,obs}=1.0$. The distribution is more important: the proportion of t-tests that would yield $x$ or fewer 'fails' would be:

x   proportion of Pois <= x
0   36.8%
1   73.6%
2   92.0%
3   98.1%
4   99.6%

Therefore, I would need at least 3/20 of the variables to 'fail' in order to confidently (95%) reject the null hypothesis that the null hypothesis is true for each variable.

Is all of this necessary?

If not, how is the variability taken into account? (r.e., the number of variables that 'pass' the test)

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Sounds like a one-way MANOVA: you have a single factor (independent variable) that distinguishes observations into two groups (A and B), and 20 dependent variables. MANOVA will allow you to see how the combination of your dependent outcomes distinguish your groups. The dependent variables should be related conceptually.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I just looked up MANOVA and I gather that this is used when there are 2 or more dependent variables. My apologies - I should have made it explicitly clear that the variables in my question are independent (they are just attributes/features of each subject in the population, e.g., age, height, IQ, etc.) $\endgroup$
    – Ben
    Commented Mar 20, 2018 at 14:01
  • $\begingroup$ I think I see what you're saying: you're treating the group selection as the independent variable, and then treating the person's attributes as dependent on which group they're in. Is that correct? $\endgroup$
    – Ben
    Commented Mar 20, 2018 at 14:25
  • $\begingroup$ From what I understand you have one independent variable (group status) with two levels (group A or group B) from which you have measured outcomes on 20 dependent variables. The scores on these 20 variables presumably depend on which group they are sampled. MANOVA can be used if the dependent variables are conceptually related, and if you satisfy other assumptions of the MANOVA. If you are interested in all 20 differences, this can be followed up with 20 independent group t-tests, but you risk a type 1 error due to multiple comparisons unless you apply some kind of correction to alpha. $\endgroup$
    – pomodoro
    Commented Mar 20, 2018 at 22:53
  • $\begingroup$ They're not measured outcomes, but rather attributes. My aim is to set up a control and a test cell, and to verify that they are as similar as possible across all of the subjects' 20 attributes before proceeding to executing the test. How the subjects are assigned into group $A$ or $B$ is irrelevant for now (random selection, greedy method, etc.), but what I would like advice on is: how can I confirm that the control and test groups are similar when there are so many attributes to consider? $\endgroup$
    – Ben
    Commented Mar 21, 2018 at 9:42
  • $\begingroup$ @Ben It doesn’t matter what you call independent and dependent variables in the rest of your work. When you run the MANOVA, you use those attributes as the dependent variable. This is the same if you t-test each attribute, as you will be considering the attribute as the dependent variable of an ANOVA. // However, hypothesis testing of any kind of unlikely to help you; just because you fail to reject doesn’t mean you accept, and rejection doesn’t necessarily mean the groups aren’t similar. It might be more productive to post about what you want to do so people can be most helpful. $\endgroup$
    – Dave
    Commented Dec 25, 2022 at 22:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.