In my problem I have 2 classifiers, C1 and C2. Both C1 and C2 are Naive Bayes classifiers but the difference between them is that they use different feature selection methods. Both classifiers are trained over a dataset of 10,000 instances (noisy-labeled), and tested over a different dataset of 1,000 instances (manually labeled), both datasets being balanced.
Now, I have plotted accuracy of both classifiers on increasing number of instances, and I found by visual inspection that C2 has generally better accuracy and recall than C1 . I would like to know whether such difference is statistically significant or not to assess that C2 is better than C1.
Previously, I used the same dataset for k-cross validation, got the mean and variation of the accuracies of both classifiers and computed student t-test on a specific amount of features. However, now I have 2 different datasets for training and testing. How could I perform a test in such situation? Should I get the mean of accuracies for all different feature amounts?
Thanks in advance...
Regarding the domain, I am dealing with sentiment analysis (SA), classifying text data in 3 classes : positive, negative and neutral. Regarding error cost, at this stage I suppose that all error costs are the same (although I understand that the cost of classifying negative as positive would higher than negative as neutral). Regarding the "practical significant difference" when dealing with SA I am assuming that an improvement of 1% is significant, since previous papers usually present such kind of improvements. I want to test the accuracy of C1 and C2 when trained over automatic-labeled data, and tested over manually-labeled data.