It is my first question here, so my excuses if my contribution is naïve.

I'm facing a classification problem in which I have a minority class (~100 samples) labeled as "Positive", another class (~100-1000 samples) labeled as "Likely positive", and a third one (~3000 samples) labeled as "Unknown" or "Unlabeled" or, if you wish, "Likely negative". Therefore, my classes are imbalanced and categorical, but have some sort of incremental order.

My variables are binary (presence/absence of a feature).

I'm not an expert on machine learning, but so far I've found R randomForest package most suitable for many of the problems I've addressed. Random Forests seem quite robust and useful for different situations.

Now, given a test set, in the end what I want to know, for each of the samples, is their probability of being positive. The intuition here is that the "Likely Positive" category should "help" somehow in assessing this probability.

How would you guys address this problem?

Thanks in advance!


  • 1
    $\begingroup$ Is there a reason to choose 'Likely Negative' as the correct imputation from 'Unlabeled'? Have you thought about trying some form of cluster analysis on the large unlabeled set? $\endgroup$ Apr 18, 2013 at 14:06
  • $\begingroup$ @image_doctor yes, I've adapted some ideas of "only-positive and unlabeled (PU) learning" to end up with a set of unknowns that I can pretty safely regard as "likely negatives". $\endgroup$ Apr 18, 2013 at 16:36
  • $\begingroup$ This is an interesting question. Miquel, what did you end up using and how well did it work? $\endgroup$
    – rinspy
    Jul 25, 2017 at 7:52

3 Answers 3


Image Doctor brings up a great point. You need to classify your data first. It sounds like you don't have much in the way of signal - do you really know what "negative" means?

Lets start by saying that you have enough trees, that you are randomly spreading around your information well, and that you are evaluating the most likely outcome reasonably well. It is an assumption.

Bagging, or more correctly "B-Agging", stands for Bootstrap Aggregation. It gives you estimates of the variability of an estimate given only the data, and that your resample size is not too large. If you resample too many times you falsely confound your information with the central limit theorem and results get strange.

In the RandomForest documentation the "margin" function is described as

For margin, the margin of observations from the randomForest classifier (or whatever classifier that produced the predicted probability matrix given to margin). The margin of a data point is defined as the proportion of votes for the correct class minus maximum proportion of votes for the other classes. Thus under majority votes, positive margin means correct classification, and vice versa.

This sounds like a way to reverse the "error rate" or number of times the result is not given out of the total possible outcomes. To me this sounds like a route to an estimate of the probability.

You might also consider "oob.times", "varImpPlot", and "importance".


It seems we dont have a very trustworthy response variable with which we may perform regression. What we could do is assign subjective probabilities of being positive to "positive", "likely positive" and "likely negative", say 0.95, 0.7 and 0.2 and then run 3 randomForests for three separate binary variables "positive/not positive", "likely positive/not likely positive" and "likely negative/ not likely negative". This would yield three separate predictions for each observation. You could get a probability estimate for each observation by performing a weighted average of the three predictions. Lets say a certain observation has predictions of "not positive", "likely positive" and "likely negative", the weighted average would be (0.95*0 + 0.7*1+ 0.2*1)/3 = 0.45, which would mean that based on your a priori subjective probabilities and the information available the observation is somewhere between "likely positive" and "likely negative".


I would treat "Positive" and "Likely positive" as positive, and "likely negative" as negative. You could then give greater weight to some samples than to others when evaluating your model, e.g. in cross-validation. But the samples you are more confident in are likely to be further from the decision boundary and so the classifier will probably do better on them anyway, even if the labeling of the samples on the decision boundary is less precise.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.