How to perform unsupervised Random Forest classification using Breiman's code?

Question

I am working with Breiman's random forest code (http://stat-www.berkeley.edu/users/breiman/RandomForests/cc_manual.htm#c2) for classification of satellite data (supervised learning). I am using a training and test dataset having sample size of 2000 and variable size 10. The data is classified into two classes, A and B. In supervised learning mode, the algorithm is performing well with very low classification error (<2%). Now I want to try the unsupervised classification with no class labels in the test data set and see how the algorithm is able to predict the classes. Is there a way to implement unsupervised classification using Breiman's code? Will the error from this method will be higher than supervised classification? The data and run parameter setting in the algorithm are given below

DESCRIBE DATA 1 mdim=10,ntrain=2000,nclass=2,maxcat=1, 1 ntest=2000,labelts=1,labeltr=1,

SET RUN PARAMETERS 2 mtry0=3,ndsize=1,jbt=500,look=100,lookcls=1, 2 jclasswt=0,mdim2nd=0,mselect=0,

Answer 1

Given that your model exhibits good accuracy you can just use it to predict the class labels of records in the unlabeled dataset. However, you cannot evaluate the performances on unlabeled data.

Be careful that you should assess the quality of your model on the labeled data by cross-validation. It is not enough to check the training error rate.

If your model is not accurate enough you might think about semi-supervised learning. The unlabeled data is used in order to improve the quality of your model via inductive learning. The accuracy should always be computed by cross-validation on your labeled data.

Have a look at [ Crimisini et al. Decision Forests: A Unified Framework for Classification, Regression, Density Estimation, Manifold Learning and Semi-Supervised Learning ] Chapter 7 about semi-supervised learning and 7.4 about induction with semi-supervised learning.

Answer 2

I doubt that unsupervised will work better but it could be a cool exercise to try out. Unsupervised learning with random forest is done by constructing a joint distribution based on your independent variables that roughly describes your data. Then simulate a certain number of observations using this distribution. For example if you have 1000 observations you could simulate 1000 more. Then you label them, e.g. 1:= real observation, 0:=simulated observation. After this, you run a usual random forest classifier trying to distinguish the real observations from the simulated ones. Note that you must have the calculate the proximity option turned on. The real useful output is exactly this, a description of proximity between your observations based on what Random Forest does when trying to assign these labels. You now have a description of how "close" or "similar" your observations are from each other and you could even cluster them based on many techniques. A straightforward one would be to select thresholds for these "distances". I mean stick together observations that are closer than a certain threshold. Another easy option is to do hierarchical clustering but using this particular distance matrix. If you can work with R, most hierarchical clustering packages allow you to feed the functions custom distance matrices. You then select a cutoff point, you may visualize it as a dendrogram and so on and so forth.

This used to be a very good tutorial on Random Forest clustering and they shared some useful R functions which they wrote for this purpose but the link seems to be dead now. Maybe it will come back up later. They also wrote a very neat random glm R package (which is analogous to random forest but based on duh...glms) if you want to check that out. You could always write to the authors and ask for the material for Random Forest classification which used to be available on the dead link. I have the R code but it's too large to paste here, I can send it to you if you send me a private message.

Answer 3

If you want to use random forest in an unsupervised setting, you'll be focusing on the distance metric obtained in what Breiman calls the "proximities". This should be an NxN matrix representing the times that the samples co-occur in terminal nodes. In R's randomForest, this is obtained via (I've never used Breiman's but I'm sure it's available):

rf = randomForest( ... )
1 - rf$proximities

Now, in an unsupervised setting, random forest has no idea how many classes there should be, so that will be your job. This can be done in a variety of ways, e.g., KNN, PAM, etc., where you choose k = 2.

As you can imagine, this is quite a bit different supervised random forest, so comparing the classification accuracy between the two procedures might not be illuminating.