3
$\begingroup$

I am trying to evaluate how well (or bad) a semi-supervised algorithm is performing on a given dataset. The algorithms assigns one of 10 labels to each data point. The dataset is huge, and it's not possible to obtain gold standard labels to each data point. Up until now, I have only come across papers where the evaluation is done against the entire dataset (e.g. using SVM to assign 'positive' or 'negative' labels in a sentiment analysis task, where each data point is a document).

Since I can't obtain gold-standard labels for the whole dataset, I was wondering whether it is scientifically valid to evaluate the performance on a randomly selected subset of the data. If yes, then how do I ensure that the subset is indeed a true representation of the entire dataset? I want to provide a valid argument that good performance on such a subset implies good performance on the entire dataset.

I don't have a statistics background, so any reference for this will be extremely helpful.

$\endgroup$

1 Answer 1

1
$\begingroup$

If the subset is selected randomly, it will, on average, be a true representation of the data. So you evaluate performance over many random subsets, you should get an estimate of how good the classifier is.

In fact, this is how classifiers are usually evaluated: You select a random subset of the data for which you know the class, train the classifier, and predict on the rest of the data. Then you quantify how well you did.

The only limitation of this approach is that if the subset is too small, you might not be able to obtain good predictions.

$\endgroup$
3
  • $\begingroup$ Thank you for the answer. Should the random selection be biased to emulate some distribution in the underlying dataset? Also, is there a way to quantify what size of the subset is considered "too small"? $\endgroup$ Commented Apr 21, 2014 at 16:08
  • $\begingroup$ No you should not bias the random sampling in any way. There isn't really a way of quantifying what is too small, except by evaluating the performance of your classifier. BTW this sampling and evaluation process is known as cross validation - coincidentally that is where the name of this website originates from. $\endgroup$
    – Bitwise
    Commented Apr 22, 2014 at 13:39
  • $\begingroup$ Yes, I am aware of cross-validation. Unfortunately, I can't apply the standard procedure in my case because I have close to 20,000 documents and only 400 of them have gold standard labels. These 400 have been selected randomly, without any bias. I wasn't sure if this suffices ... your answer helped. $\endgroup$ Commented Apr 23, 2014 at 15:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.