For my classification task I have two classes labeled 0 and 1. I am using Random Forests from sklearn package in python.

I have two files for different classes. So I loaded the files, combined them into one data array and trained my classifier. When I perform k fold cross validation on my input data my classifier scores at the around 98%

However, when I test the classifier on another piece of data which belongs to only one of the classes, my accuracy drops down to around 50%

Whats even more weird is that when I take two different data arrays for each class combine them. I get accuracy of around 90% with my classifier.

What's happening here? I'm confused

Note :-

Class Distribution for my training data is around 55 - 45 percent

  • $\begingroup$ It's hard to say. Can you say more about your data, etc? Is this an unbalanced classes issue? $\endgroup$ – gung Oct 16 '14 at 12:48
  • $\begingroup$ Classes are almost equally distributed $\endgroup$ – Ajit Oct 16 '14 at 12:56
  • $\begingroup$ some part of the procedure is a bit unclear: "classifier" can refer to a class of classifciation algorithms or a specific instance with estimated parameters. Based on the second use, cross-validation (what was your k?) involves a different classifier in each run, which again differs from the classifier you estimate for the ful data set. Wjhat I would do here: check sensitivity and specificity to see, wheterhyour classifier is biased in a systematic way. Hw does the data you refer to as "another piece of data" to the original data set? $\endgroup$ – jank Oct 16 '14 at 13:11
  • $\begingroup$ @jan Classifier refers to a specific instance of a classifier with estimated parameters. I have set k = 10. Data I am testing on isn't part of the original dataset. Thanks $\endgroup$ – Ajit Oct 16 '14 at 17:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.