Measuring representativeness of a sample using covariates I was provided with quite a small sample of labeled (variable of interest) observations to train a model to predict unlabeled observations. All the observations are associated with many covariates. I'm assuming that the trained model will do better in measure of how well does this small labeled sample represent the unlabeled cases. Using only the covariates is there any way to measure if an unlabeled case will be poorly predicted or not? I can imagine that if the covariates are standardized and you measure the euclidean distance (covariates are continuous) from a unlabeled point to the labeled ones and this unlabeled one tends to be "far away" from the labeled sample the prediction accuracy would drop. I'm not really sure. Does anyone have any comments on this or any pointers on what to read to assess this or if my ideas are just totally off? 
By the way, the techniques I tried out for this task are Random Forests, MARS and boosted regression trees.
 A: For a very small labeled set it would be very difficult to distinguish between the non-representativeness due to random sampling and non-reprensentativeness due to the labeled set covariates being drawn from a different population distribution than the unlabeled set. 
For a "medium" sized labeled set you can try the following to assess how non-representative it is. Let the size of the labeled data set be $N$ and that of the unlabeled set be $M$. If you  took the combined $N+M$ sized data set and built a classifier between the two classes labeled $L$, and unlabeled $U$, you should expect the posterior class-probabilities assigned to the labeled examples to be $N/(N+M)$ if the two distributions are the same. Otherwise some examples will have different posterior probabilities. So if you build the "best" possible classifier (in a cross-validated setting) on the $N+M$ sized data set and measured the distance of the posterior probabilities to the expected value, it would give you a measure of non-representativeness of the labeled data set. Clearly, you could also do this for the unlabeled data set.
For the classifier you can use the RandomForest classifier which can output posterior class probabilities and for the distance metric of the probabilities you might try using the KL-divergence.
A: Are your unlabelled observations similar to your labelled observations with regard to independant variables? If so, why dont you run your model keeping a hold-out sample of labelled data so that you may later measure accuracy. You can relate prediction success to the distance to the center of the label points.
A: You can't be absolutely sure about the prediction power on the unlabeled set (if they're not an exact copy of the labeled training set).
I guess you could try out some unsupervised learners and look for similarities between the labeled and unlabeled sets.
A: It will still be difficult to generalise the result from the labeled set (used to train the classifier) to the unlabeled set. I recommend to evaluate if the two sets are similar, and if not, where they differ. After this, train your model and predict the unlabeled set with it.
