# Can Random Forests do much better than the 2.8% test error on MNIST?

I haven't found any literature on the application of Random Forests to MNIST, CIFAR, STL-10, etc. so I thought I'd try them with the permutation-invariant MNIST myself.

In R, I tried:

randomForest(train$x, factor(train$y), test$x, factor(test$y), ntree=500)


This ran for 2 hours and got a 2.8% test error.

I also tried scikit-learn, with

RandomForestClassifier(n_estimators=2000,
max_features="auto",
max_depth=None)


After 70 minutes, I got a 2.9% test error, but with n_estimators=200 instead, I got a 2.8% test error after just 7 minutes.

With OpenCV, I tried

rf.train(images.reshape(-1, 28**2),
cv2.CV_ROW_SAMPLE,
labels.astype('int'))


This ran for 6.5 minutes, and using rf for prediction gave a test error of 15%. I don't know how many trees it trained, as their Python binding for Random Forests seems to ignore the params argument, at least in version 2.3.1. I also couldn't figure out how to make it clear to OpenCV that I want to solve a classification problem, rather than regression -- I have my doubts, because replacing astype('int') with astype('float32') gives the same result.

In neural networks, for the permutation-invariant MNIST benchmark, the state of the art is 0.8% test error, although training would probably take more than 2 hours on one CPU.

Is it possible to do much better than the 2.8% test error on MNIST using Random Forests? I thought that the general consensus was that Random Forests are usually at least as good as kernel SVMs, which I believe can get a 1.4% test error.

• remember that a random forest is taking a decision 1 variable ( ie pixel ) at a time. So it is not very good for image processing -raw. You are better off first using some sort of preprocessing ( eg PCA, etc ) to develop more meaningful decision variables – seanv507 Dec 6 '13 at 21:41
• Exactly what seanv507 said. OpenCV has a lot of functions for feature extraction which can detect quite useful explanatory variables for random forest to work with. – JEquihua Dec 7 '13 at 13:39
• I thought that the general consensus was that Random Forests are usually at least as good as kernel SVMs. There is no such consensus. – Marc Claesen Jan 6 '14 at 21:15