1
$\begingroup$

Quick question: I've been always told that if a binary classifier shows a very low accuracy (~0) you can switch the predictions to get an high accuracy (~1).

It actually never happened to me before, but now I have a Random Forests classifier which gives a prediction ~0. If I invert the predictions, then I get an accuracy ~1.

I'm trying to figure it out why at a certain point it decides to classify the data in class 1 as class 0, and vice versa.

Any idea what I should look for?

The classifier is the standard TreeBagger of MATLAB.

$\endgroup$
  • $\begingroup$ Do you use stratification or weighting? Is the training set and test set balanced? Do you speak of out-of-bag accuracy or external test set accuracy? $\endgroup$ – Soren Havelund Welling Nov 18 '15 at 10:17
  • $\begingroup$ @SorenHavelundWelling The training and the test sets have been artificially balanced, meaning that I doubled the samples in class 0 to get 50-50. External test set accuracy. I don't know what is the difference between stratification and weighting. $\endgroup$ – user1384636 Nov 18 '15 at 12:18
1
$\begingroup$

Doubling samples creates doublets, which easily becomes miss leading for a CV. Also it can potentially disturb the robustness of your ensemble. I don't use treeBagger. But I read you can set prior='Uniform'. Try that instead of doublets.

For a better performance evaluation make Treebagger output OOB-class probabilities and evaluate these against true class labels in a ROC-plot. This matlab guide should help you: http://www.mathworks.com/examples/statistics/2205-classifying-radar-returns-for-ionosphere-data-using-treebagger

$\endgroup$
  • $\begingroup$ yes yes, the accuracy of ~0 (or ~1 if you reverse it) is given by the ROC curve. But I try with the prior instead of doubling the 0-class data. $\endgroup$ – user1384636 Nov 18 '15 at 14:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.