1
$\begingroup$

Quick question: I've been always told that if a binary classifier shows a very low accuracy (~0) you can switch the predictions to get an high accuracy (~1).

It actually never happened to me before, but now I have a Random Forests classifier which gives a prediction ~0. If I invert the predictions, then I get an accuracy ~1.

I'm trying to figure it out why at a certain point it decides to classify the data in class 1 as class 0, and vice versa.

Any idea what I should look for?

The classifier is the standard TreeBagger of MATLAB.

$\endgroup$
2
  • $\begingroup$ Do you use stratification or weighting? Is the training set and test set balanced? Do you speak of out-of-bag accuracy or external test set accuracy? $\endgroup$ Commented Nov 18, 2015 at 10:17
  • $\begingroup$ @SorenHavelundWelling The training and the test sets have been artificially balanced, meaning that I doubled the samples in class 0 to get 50-50. External test set accuracy. I don't know what is the difference between stratification and weighting. $\endgroup$ Commented Nov 18, 2015 at 12:18

1 Answer 1

1
$\begingroup$

Doubling samples creates doublets, which easily becomes miss leading for a CV. Also it can potentially disturb the robustness of your ensemble. I don't use treeBagger. But I read you can set prior='Uniform'. Try that instead of doublets.

For a better performance evaluation make Treebagger output OOB-class probabilities and evaluate these against true class labels in a ROC-plot. This matlab guide should help you: http://www.mathworks.com/examples/statistics/2205-classifying-radar-returns-for-ionosphere-data-using-treebagger

$\endgroup$
1
  • $\begingroup$ yes yes, the accuracy of ~0 (or ~1 if you reverse it) is given by the ROC curve. But I try with the prior instead of doubling the 0-class data. $\endgroup$ Commented Nov 18, 2015 at 14:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.