I have a binary classification task with 10 ordinal features and 45,000 records, 8,500 of which should be classified as positive. To evaluate the classifier I build, I also build one on (a) the same classes but no features (i.e., random guessing); (b) the same records with all features but the classes shuffled (i.e., do we not see patterns in randomness? Could we use this classifier to classify some random distribution?). The classifier would be better if it scores better than the ones built with (a) and (b), of course. I use cross validation to measure the accuracy of the classifiers.
I have done test runs using (1) all data and (2) a randomly selected subset of the negative data to give 8,500 records of both classes. In run (2), my classifier scores significantly better than (a) and (b); 0.59 vs. 0.50 with a standard deviation of 0.04 vs. 0.00/0.01. In run (1), all three classifiers classify everything as negative, giving equal scores of 0.81 (standard deviation 0.38).
What does this mean? Is my classifier bogus? Or do the settings of the classifier (see below) not work for unbalanced data? Or is there not enough information to tell, in which case: what should I find out to be able to answer these questions?
The classifier is a decision tree with depth 4 and a minimum of 40 samples per leaf.