Can Random Forests do much better than the 2.8% test error on MNIST?

Question

I haven't found any literature on the application of Random Forests to MNIST, CIFAR, STL-10, etc. so I thought I'd try them with the permutation-invariant MNIST myself.

In R, I tried:

randomForest(train$x, factor(train$y), test$x, factor(test$y), ntree=500)

This ran for 2 hours and got a 2.8% test error.

I also tried scikit-learn, with

RandomForestClassifier(n_estimators=2000,
                       max_features="auto", 
                       max_depth=None)

After 70 minutes, I got a 2.9% test error, but with n_estimators=200 instead, I got a 2.8% test error after just 7 minutes.

With OpenCV, I tried

rf.train(images.reshape(-1, 28**2), 
         cv2.CV_ROW_SAMPLE, 
         labels.astype('int'))

This ran for 6.5 minutes, and using rf for prediction gave a test error of 15%. I don't know how many trees it trained, as their Python binding for Random Forests seems to ignore the params argument, at least in version 2.3.1. I also couldn't figure out how to make it clear to OpenCV that I want to solve a classification problem, rather than regression -- I have my doubts, because replacing astype('int') with astype('float32') gives the same result.

With neural networks, for the permutation-invariant MNIST benchmark, the state of the art is 0.8% test error, although training would probably take more than 2 hours on one CPU.

Is it possible to do much better than the 2.8% test error on MNIST using Random Forests? I thought that the general consensus was that Random Forests were usually at least as good as kernel SVMs, which I believe can get a 1.4% test error.

Answer 1

Is it possible to do much better than the 2.8% test error on MNIST using Random Forests?

Probably, yes. But that doesn't mean you'll be using the same features that you get by default. Decision trees in general don't work well on high dimensional problems like this, as you are only splitting on one feature at a time. Random Forest extends the usefulness of Decision Trees, but they still have the same issue. Beating 2.8% with RF will probably require you to do some feature pre-processing and transform the features into a more useful subset.

Neural Networks and Kernel SVMs are implicitly doing some feature transformation/engineering. So in some sense its impressive that Random Forest gets decently close without any extra work (indeed the real reason RF became popular is it was stupidly easy to get "good enough" results).

I thought that the general consensus was that Random Forests are usually at least as good as kernel SVMs

There is no such consensus. They often have similar results in terms of accuracy - but they are very different algorithms with different strengths / weaknesses. On many problems there accuracies are similar, on others SVMs win by a good margin, on some RF wins by a good margin.