Hi I have a data set say of size 450. I have created 3 datasets of this dataset i.e. 150, 300 and 450.
I have used 3 classifiers on these 3 datasets and observed the performance of classifiers in terms of accuracy, Precision and recall.
How should I perform statistical significance test to compare the 3 classifiers. I want to use t-test for this.
Please let me know - should I compare on each dataset or apply t-test between 2 classifiers (paired)? Please suggest your thought.
Yes, I am using cross-validation for performance comparison.
I have used below steps for the significance tests using a t-test:
considered significance level as 0.05. Then I used the t-test (using rapidminer tool) to compute P value of the 2 classifiers (paired t -test).
I got the P value. Based on the P value, if its less than 0.05, I can say that there is significant difference between mean value of classifiers.
So, is this a correct approach or should I go for AUC only?