Choice of test set for classification

Question

I have 50 measurements of 10 descriptors and 1 binary output variable.

I want to use a classification procedure to be able to predict the output, so I split the data into a training and a test set and I can then generate my classifier (I am using a decision tree) and test it on the test set.

Now, obviously the choice of test set is absolutely arbitrary and, whatever the result of my procedure, I cannot be certain that the result I get for that specific test set is similar to what I will get from any other test set.

So, would I have a point in repeating my classification several times, each time with a randomly chosen test/training set, then report the distribution of misclassification errors for my classifier?

I understand that this is a bit similar to what Random Forests are doing, but I am wondering if this procedure makes sense also when applied to other type of classifiers, not necessarily decision trees.

Answer 1

Basically random forests is a bagged ensemble, with something additional which makes the averaged model more independent. If you remove the random part from random forests you will end up with bagging, or bootstrap aggregating. Bagging can be applied to any type of classifier or regressor. See Wikipedia page for more details on bootstrap aggregation. That is when you want to average the results.

If you want only to estimate the error given by a single tree, (that tree is built each time, for each sample), than what you want looks like bootstrapping. See this Wikipedia bootstrapping page for starting points on these procedures. As far as I know there is plenty of knowledge on estimating with bootstrapping.

Also, you can build an empirical distribution of misclassification error, no matter which model was used to predict. When you have stronger assumptions on the distribution of the model and the distribution of the error, of course, you might be in the position to infer also, which is the name of that distribution and it's parameters. However, to build an empirical distribution you need no assumption, I think and bootstrapping will provide good results.