# Measuring representativeness of a sample using covariates

I was provided with quite a small sample of labeled (variable of interest) observations to train a model to predict unlabeled observations. All the observations are associated with many covariates. I'm assuming that the trained model will do better in measure of how well does this small labeled sample represent the unlabeled cases. Using only the covariates is there any way to measure if an unlabeled case will be poorly predicted or not? I can imagine that if the covariates are standardized and you measure the euclidean distance (covariates are continuous) from a unlabeled point to the labeled ones and this unlabeled one tends to be "far away" from the labeled sample the prediction accuracy would drop. I'm not really sure. Does anyone have any comments on this or any pointers on what to read to assess this or if my ideas are just totally off?

By the way, the techniques I tried out for this task are Random Forests, MARS and boosted regression trees.

For a very small labeled set it would be very difficult to distinguish between the non-representativeness due to random sampling and non-reprensentativeness due to the labeled set covariates being drawn from a different population distribution than the unlabeled set.

For a "medium" sized labeled set you can try the following to assess how non-representative it is. Let the size of the labeled data set be $N$ and that of the unlabeled set be $M$. If you took the combined $N+M$ sized data set and built a classifier between the two classes labeled $L$, and unlabeled $U$, you should expect the posterior class-probabilities assigned to the labeled examples to be $N/(N+M)$ if the two distributions are the same. Otherwise some examples will have different posterior probabilities. So if you build the "best" possible classifier (in a cross-validated setting) on the $N+M$ sized data set and measured the distance of the posterior probabilities to the expected value, it would give you a measure of non-representativeness of the labeled data set. Clearly, you could also do this for the unlabeled data set.

For the classifier you can use the RandomForest classifier which can output posterior class probabilities and for the distance metric of the probabilities you might try using the KL-divergence.

• I have trying to do this for a bit. It sounds good but I am still a little skeptic. Could you maybe detail it a bit more? or point me to a more formal source? Thank you so much. – JEquihua May 31 '13 at 0:11
• This trick is more of a folk knowledge than from a formal source. There may be a paper somewhere talking about this kind of stuff but I am not aware of it. The closest I can find to a corroborating opinion is this blog post blog.smola.org/post/4110255196/… – Innuo May 31 '13 at 15:05
• Here what would the "posterior class-probabilities" mean? I label the data L and U. Then I for example run a random forest (as you adviced) for classification and choose the output to be "probability". If I have 1000 labeled cases and 100 unlabeled I should get a probability (int his case a proportion) of 100/1100 for the U cases? For each case? or the sum of the probs should be N/(N+M) – JEquihua Jun 2 '13 at 15:29

Are your unlabelled observations similar to your labelled observations with regard to independant variables? If so, why dont you run your model keeping a hold-out sample of labelled data so that you may later measure accuracy. You can relate prediction success to the distance to the center of the label points.

• Could you explain this a bit further? I didn't understand very well. – JEquihua May 6 '13 at 14:26
• Train your model with 50% of your sample and then test its accuracy with the remaining 50% hold-out sample. You can divide your hold-out sample in, say, 10 sets, according to how distant they are to the centroid of the training set and observe how the accuracy of the model changes depending on how far observations are from the training set. You could also choose your hold-out and training samples so that some of the hold-out sample are similar to the training set and some less so. – emDiaz May 6 '13 at 16:26

You can't be absolutely sure about the prediction power on the unlabeled set (if they're not an exact copy of the labeled training set).

I guess you could try out some unsupervised learners and look for similarities between the labeled and unlabeled sets.

• Like doing k-means on the labeled and unlabeled set independently and finding similar clusters? Or what would you do? – JEquihua May 6 '13 at 14:24

It will still be difficult to generalise the result from the labeled set (used to train the classifier) to the unlabeled set. I recommend to evaluate if the two sets are similar, and if not, where they differ. After this, train your model and predict the unlabeled set with it.

• How would I measure how similar they are? – JEquihua May 7 '13 at 21:50
• I recently went for very basic statistical inference. For example, you can create a confidence interval for the difference of the means of a numeric variable. If the interval overlaps 0, there's no statistically significant difference. Not 100 % sure about your situation though, an approach like this may be invalid. Have you tried running k-means, as you proposed? – Eric Paulsson May 7 '13 at 22:01