Classifier statistical significance test

Question

Hi I have a data set say of size 450. I have created 3 datasets of this dataset i.e. 150, 300 and 450.

I have used 3 classifiers on these 3 datasets and observed the performance of classifiers in terms of accuracy, Precision and recall.

How should I perform statistical significance test to compare the 3 classifiers. I want to use t-test for this.

Please let me know - should I compare on each dataset or apply t-test between 2 classifiers (paired)? Please suggest your thought.

---

Yes, I am using cross-validation for performance comparison.

I have used below steps for the significance tests using a t-test:

1. considered significance level as 0.05. Then I used the t-test (using rapidminer tool) to compute P value of the 2 classifiers (paired t -test).

2. I got the P value. Based on the P value, if its less than 0.05, I can say that there is significant difference between mean value of classifiers.

So, is this a correct approach or should I go for AUC only?

Answer 1

I would suggest that your datasets are dependent (they have repeated observations) and the direct application of statistical test can have several underlying problems. I would not recommend to divide your dataset in such a way.

Also I would recommend to introduce Train dataset for training of your classifiers or use cross validation (it is not clear from your question if you use it or not). Otherwise you will just compare the performance of the classifier on this particular sample, not on population from which it was sampled.

Area Under ROC seems appropriate measure (from the previous answer).

Answer 2

One simple statistic to compare models in Area Under ROC curve(AUC).

I think that unless the models are very much different in terms of performance, you wont find difference in their predictions to be statistically significant. But if you must do the test, notable points are:

• T-test assumes that your data shows normal distribution
• So in case you are planning to use the P(class) from each model as the input to T-test, that would be wrong, because that would be a multi-modal distribution in general (Two local peaks in case of two classes).

So you must only use P(class) for one class of people at a time to compare models. Now you'd have to perform paired T-tests between the P(class) columns for each pair of models.