As part of our project, we have used a combination of three machine learning classifiers combined with a voting algorithm over it to obtain reasonably good results. The input data for the classifiers is a set of questions and the output for each is a probability distribution over 6 labelled classes.
Now, as part of hypothesis testing, our professor had suggested us to try multivariate tests -- in particular, the Hotelling $T^2$ test -- to find out whether the results of the three classifiers are statistically significant and to find the parameters that have provided us the maximum information.
We do not have a strong background in statistics but from what we have read, we felt that this test is usually used when you test on two different sets of samples as opposed to our case, where we have the same data used to generate the outputs for the three classifiers.
Based on this source of info, this is what we have understood:
T-tests are used to determine if two different sets of samples are represent the same kind of population
Eg. 10 patients in ward 1
13 patients in ward 2
If we are to conduct an experiment with these two sets, can we do so without the results being different due to difference in the people taken
Eg. One room had HIV patients while the other had blood cancer patients
The results might be really different if we know this. But if we don't, using use other factors like blood test results (eg. Sodium level, wbc count etc), can we say that these two groups are similar enough
This leads us to believe that maybe we are:
- Not understanding the test correctly, or
- Using the wrong test statistic
Could you help us in this regard? Thank you.