I'm new to ML and decided to start learning by having a go at number recognition using the MNIST subset on kaggle http://www.kaggle.com/c/digit-recognizer
The images are 28x28 pixel greyscale of the digits from 0 to 9, with ~34000 in the training set and 28000 in the test set. Random Forest using the raw image and 2000 trees gives a score of 0.96829.
I did some preprocessing of the images to extract more features and trained / tested on a equal sized subsets of the training data. Compared to normal random forest for subsets around 1000 - 3000 I was getting about a 3-4% improvement (e.g. 94% vs 90% for RF)
After training with the entire training set for 200 trees, I scored 0.97557 (a ~0.7% improvement on RF). However, increasing to 2000 trees I scored 0.97457 (0.1% less).
- Does this mean the features are 'saturated'? i.e. any minor performance gain / loss is just random fluctuations?
- Is there anything I can try (apart from more features) to improve the result?
- Is there any way to weight some features more than others so they are more likely to be evaluated?