I have a text classification problem.

There are 100 articles. Each article contains from 5 up to 20 sentences (an article cannot be separated into sentences, we cannot train and test on sentence-level). I have two classifiers: classifier1 and classifier2. The macro-average F1 of classifier1 on 10-fold cross validation is higher that the macro-average F1 of classifier2.

I want to identify whether this advantage is statistical significant. What is the correct way to calculate statistical significance?

By far I found that the wilcoxon test can be compatible, however it requires at least 30 test sets (articles).

One solution could be in every fold interchange the train and the test, such that the train is the smallest set, then divide 90 test articles into 30 sets by 3 article, and for current 10 train articles calculate macro-average F1 on each bunch of 3 articles, so as result we have 30 F1 values just for the current fold.

But what to do with the other folds? As a final result should I get 300 F1 values and then check the statistical significance or calculate the average of 30 F1 values among all 10 folds and then check the statistical significance.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.