I work in a filed were there are many publications based on classifiers trained on small samples sizes (but large amount of features). In most cases the sample size can only be increased by a few instances based on the cumbersome annotation process.

I built a classifier for a multi-classification problem, that has a micro-F1 score of 0.8 based on a cross validation performed on approx. 2000 samples. There is a published study, where they also reached an micro-F1 score of 0.8, but their sample size was approx. 1000 samples.

Unfortunately the identical 1000 sample data set can not be used (it is nor public). Can I still make an argument, that my classifier might perform better on new data, since the training size was twice as much ?

Are there any studies available were performance is compared with sample size for small sample classification.

  • $\begingroup$ +1 but also the obligatory comment about proper scoring rules as opposed to the threshold-based F1 score: stats.stackexchange.com/a/469059/247274 $\endgroup$ – Dave Aug 6 at 14:02
  • $\begingroup$ I am not completely sure how the comment is related to the question, I mean, if one is bound to a certain scoring rule / metric for comparability reasons, one does not really have a choice right ? $\endgroup$ – Felix Z. Oct 22 at 8:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.