# How to perform unsupervised Random Forest classification using Breiman's code?

I am working with Breiman's random forest code (http://stat-www.berkeley.edu/users/breiman/RandomForests/cc_manual.htm#c2) for classification of satellite data (supervised learning). I am using a training and test dataset having sample size of 2000 and variable size 10. The data is classified into two classes, A and B. In supervised learning mode, the algorithm is performing well with very low classification error (<2%). Now I want to try the unsupervised classification with no class labels in the test data set and see how the algorithm is able to predict the classes. Is there a way to implement unsupervised classification using Breiman's code? Will the error from this method will be higher than supervised classification? The data and run parameter setting in the algorithm are given below

DESCRIBE DATA 1 mdim=10,ntrain=2000,nclass=2,maxcat=1, 1 ntest=2000,labelts=1,labeltr=1,

SET RUN PARAMETERS 2 mtry0=3,ndsize=1,jbt=500,look=100,lookcls=1, 2 jclasswt=0,mdim2nd=0,mselect=0,

• Can you elaborate further? If want to predict the classes you have to use supervised classification. – Simone Oct 9 '13 at 23:19
• Let's just say that i have two datasets with same number of variables and samples. the first data set contain the class information(A and B) while second dataset does not have any class info. Using RF code, first dataset was classified in to the two classes with very good accuracy. The run parameters given above are for that particular classification where the class info is required in the dataset. Now my question is how to classify the second datset in to two class? – Bijoy Oct 10 '13 at 16:54

Given that your model exhibits good accuracy you can just use it to predict the class labels of records in the unlabeled dataset. However, you cannot evaluate the performances on unlabeled data.

Be careful that you should assess the quality of your model on the labeled data by cross-validation. It is not enough to check the training error rate.

If your model is not accurate enough you might think about semi-supervised learning. The unlabeled data is used in order to improve the quality of your model via inductive learning. The accuracy should always be computed by cross-validation on your labeled data.

Have a look at [ Crimisini et al. Decision Forests: A Unified Framework for Classification, Regression, Density Estimation, Manifold Learning and Semi-Supervised Learning ] Chapter 7 about semi-supervised learning and 7.4 about induction with semi-supervised learning.

• Thanks Simone for the comments... As you mentioned in your comments, my objective is to predict the class labels of records in the unlabeled dataset using RF. However, i would like to know whether this is possible using the RF code of Brieman and Cutler..?? – Bijoy Oct 11 '13 at 5:48
• I am not that familiar with that code. It might also be a bit outdated. Try WEKA or R, they provide recent random forest implementations. – Simone Oct 11 '13 at 5:54

I doubt that unsupervised will work better but it could be a cool exercise to try out. Unsupervised learning with random forest is done by constructing a joint distribution based on your independent variables that roughly describes your data. Then simulate a certain number of observations using this distribution. For example if you have 1000 observations you could simulate 1000 more. Then you label them, e.g. 1:= real observation, 0:=simulated observation. After this, you run a usual random forest classifier trying to distinguish the real observations from the simulated ones. Note that you must have the calculate the proximity option turned on. The real useful output is exactly this, a description of proximity between your observations based on what Random Forest does when trying to assign these labels. You now have a description of how "close" or "similar" your observations are from each other and you could even cluster them based on many techniques. A straightforward one would be to select thresholds for these "distances". I mean stick together observations that are closer than a certain threshold. Another easy option is to do hierarchical clustering but using this particular distance matrix. If you can work with R, most hierarchical clustering packages allow you to feed the functions custom distance matrices. You then select a cutoff point, you may visualize it as a dendrogram and so on and so forth.

This used to be a very good tutorial on Random Forest clustering and they shared some useful R functions which they wrote for this purpose but the link seems to be dead now. Maybe it will come back up later. They also wrote a very neat random glm R package (which is analogous to random forest but based on duh...glms) if you want to check that out. You could always write to the authors and ask for the material for Random Forest classification which used to be available on the dead link. I have the R code but it's too large to paste here, I can send it to you if you send me a private message.

If you want to use random forest in an unsupervised setting, you'll be focusing on the distance metric obtained in what Breiman calls the "proximities". This should be an NxN matrix representing the times that the samples co-occur in terminal nodes. In R's randomForest, this is obtained via (I've never used Breiman's but I'm sure it's available):

rf = randomForest( ... )
1 - rf\$proximities


Now, in an unsupervised setting, random forest has no idea how many classes there should be, so that will be your job. This can be done in a variety of ways, e.g., KNN, PAM, etc., where you choose k = 2.

As you can imagine, this is quite a bit different supervised random forest, so comparing the classification accuracy between the two procedures might not be illuminating.