Outline of clustering technique using Random Forest

A synthetic data is created by randomly sampling from the data of interest. It is used as the base line to measure the "structureness" or "clustering" in the data of interest. The real data and synthetic data are combined and fed into the randomForest() to do classification.The distance matrix is calculated from the proximity measure of the outputted random forest and is clustering is done on the distance matrix using other clustering techniques.


1)Sampling technique

The paper by Shi et al. (http://labs.genetics.ucla.edu/horvath/RFclustering/RFclustering/RandomForestHorvath.pdf) describes two sampling techniques- (1)random sampling from the product of empirical marginal distributions of the variables of the data and (2)random sampling (uniform distribution) from the hyper rectangle containing the data.

2)No. of forests

Shi et al. reported that "RF dissimilarity can vary considerably as a function of the particular realization of the synthetic data". So a number of forests are grown and are combined to get the final result.


Which sampling technique does randomForest() function from randomForest package uses ? Also, how many forests are grown?

  • I've read that article before. It's a frustrating article that seemed to me to be misleading. Perhaps the the R and Fortran codes were different at the time of writing. They explicitly state that both methods are available in the R package randomForest but I couldn't find it. The website implies that the second sampling method is used. The only way to be certain may be to look at the underling code.
  • The issue of the number of forests is also poorly dealt with. They use some form of CV - if I remember right - to estimate the number of forests and then provide a rule of thumb based on this CV. But it (1) clearly depends on the starting sample size how stable the synthetic dataset is going to be. (2) you don't - as they admit - have to built multiple forests, but create a sufficiently large synthetic dataset. The issue is how large a synthetic dataset do you need.
  • 1
    $\begingroup$ There is no rule regarding the # of replications. Breiman used ~1,000 bootstrapped samples on a single CPU but this was in the 90s and it could take days to run. Today, people are drawing millions of samples across multi-core machines on big data to get an answer in a few hours, a workaround that wasn't available to Breiman. The limitation of methods like RFs is that, in finite data samples and no matter how many samples are drawn, it's still an approximation. It's only in the theoretical asymptotic limit with the number of samples approaching infinity that sampling bias is eliminated. $\endgroup$ – Mike Hunter Jun 11 '15 at 11:11
  • 1
    $\begingroup$ Sorry, I ran out of space before adding a reference about recent extensions of RF approaches. A Survey of Statistical Methods and Computing for Big Data by Wang, Chen, Schifano, Wu and Yan (2015) $\endgroup$ – Mike Hunter Jun 11 '15 at 11:15
  • 1
    $\begingroup$ The 'h2o' package has a "stop when you get to this level of stability in the output" button. It sounds like a stopping technique for a more conventional algorithm. $\endgroup$ – EngrStudent - Reinstate Monica May 5 '16 at 12:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.