Positive, Likely Positives and Unknowns in Random Forests It is my first question here, so my excuses if my contribution is naïve.
I'm facing a classification problem in which I have a minority class (~100 samples) labeled as "Positive", another class (~100-1000 samples) labeled as "Likely positive", and a third one (~3000 samples) labeled as "Unknown" or "Unlabeled" or, if you wish, "Likely negative". Therefore, my classes are imbalanced and categorical, but have some sort of incremental order.
My variables are binary (presence/absence of a feature).
I'm not an expert on machine learning, but so far I've found R randomForest package most suitable for many of the problems I've addressed. Random Forests seem quite robust and useful for different situations.
Now, given a test set, in the end what I want to know, for each of the samples, is their probability of being positive. The intuition here is that the "Likely Positive" category should "help" somehow in assessing this probability.
How would you guys address this problem?
Thanks in advance!
Miquel
 A: Image Doctor brings up a great point.  You need to classify your data first.  It sounds like you don't have much in the way of signal - do you really know what "negative" means?
Lets start by saying that you have enough trees, that you are randomly spreading around your information well, and that you are evaluating the most likely outcome reasonably well.  It is an assumption.  
Bagging, or more correctly "B-Agging", stands for Bootstrap Aggregation.  It gives you estimates of the variability of an estimate given only the data, and that your resample size is not too large.  If you resample too many times you falsely confound your information with the central limit theorem and results get strange.
In the RandomForest documentation the "margin" function is described as 

For margin, the margin of observations from the randomForest classiﬁer (or whatever 
  classiﬁer that produced the predicted probability matrix given to
  margin). The margin of a data point is deﬁned as the proportion of
  votes for the correct class minus maximum proportion of votes for the
  other classes. Thus under majority votes, positive margin means
  correct classiﬁcation, and vice versa.

This sounds like a way to reverse the "error rate" or number of times the result is not given out of the total possible outcomes.  To me this sounds like a route to an estimate of the probability.
You might also consider "oob.times", "varImpPlot", and "importance".
A: It seems we dont have a very trustworthy response variable with which we may perform regression. What we could do is  assign subjective probabilities of being positive to  "positive", "likely positive" and "likely negative", say 0.95, 0.7 and 0.2 and then run 3 randomForests for three separate binary variables "positive/not positive", "likely positive/not likely positive" and "likely negative/ not likely negative". This would yield three separate predictions for each observation. You could get a probability estimate for each observation by performing a weighted average of the three predictions. Lets say a certain observation has predictions of "not positive", "likely positive" and "likely negative", the weighted average would be (0.95*0 + 0.7*1+ 0.2*1)/3 = 0.45, which would mean that based on your a priori subjective probabilities and the information available the observation is somewhere between "likely positive" and "likely negative".
A: I would treat "Positive" and "Likely positive" as positive, and "likely negative" as negative. You could then give greater weight to some samples than to others when evaluating your model, e.g. in cross-validation. But the samples you are more confident in are likely to be further from the decision boundary and so the classifier will probably do better on them anyway, even if the labeling of the samples on the decision boundary is less precise.
