0
$\begingroup$

I have a small sample dataset (n=25) that represents the ground truth for a larger set (n=10k). I am doing a classification task and obtain, say, 3 true positives, 20 true negatives, 1 false positive, and 1 false negative. Is there a way to compute meaningfully how my classifier would perform on the larger set? Normally, I would use accuracy, precision, recall, etc. However, given the skewed data and the few observations, is that even meaningful? Are there better alternatives?

Obivously, I can calculate confidence for my accuracy measures. However, maybe there is something else.

$\endgroup$
0
$\begingroup$

I see this question is really old, but just my two cents: You could use the F1 score. I usually use it for imbalanced/skewed datasets, as it represents how well the model performs on both precision and recall - making it specifically meaningful for skewed datasets.

Whether any accuracy measure is going to be meaningful depends on how well the small sample represents the larger dataset. This is hard to know, because your data may contain complexities that cannot be represented in smaller subsamples of the larger dataset.

Obviously, another issue is the sample size itself. 25 samples is a really small sample size for any learning task. I could imagine a learning task on such a small sample size to be successful on the condition that, for instance, your prediction target classes are linearly separable (the data is of low complexity), in which case the learning task is easy, and sample size becomes less important. This is rarely the case in my experience with real world data.

In brief: You could use the F1 score, if the sample is representative (hard to know for sure), but the small sample size will almost certainly make this a very hard problem - unless the data are non-complex (e.g. output classes are linearly separable).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.