I'm trying to apply the bag of visual words approach to make scene classification. I started to use k-means to generate my codebook, but rapidly discovered its limitations. From one codebook generation to another, I get fluctuations of 10% on the classification ratio (data dimension: 8, number of clusters: 50). In addition, k-means is slow (I'm working on Matlab).
So I did some research and found that I can do unsupervised learning with random forests (paper: "Unsupervised Learning With Random Forest Predictors" by T. Shi and S. Horvath). It looked nice, so I decided to give it a try.
What I did:
(1) I created a synthetic data set from my original data set. To generate the coordinates of my synthetic samples, I take the corresponding coordinate of one of my original samples, chosen randomly (like explained here:http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm#unsup). I also tried the Addcl2 method like explained in the paper that I mentioned above.
(2) The original data is labeled 1, the synthetic data is labeled 2. Then I train a random forest (I use Matlab's TreeBagger class) on this 2 classes. Then I compute the proximity matrix, but only for the original data.
(3) I use this proximity matrix as a distance matrix to perform unsupervised learning with hierarchical clustering.
Here is what I get for 2D data:
It works approximately well for good separated data points, while the clustering result depends from one time to another, sometimes I get the same result as k-means. For badly separated data, the result is awfull.
What am I doing wrong? I could need some advice ;-)