For my classification task I have two classes labeled 0 and 1. I am using Random Forests from sklearn package in python.
I have two files for different classes. So I loaded the files, combined them into one data array and trained my classifier. When I perform k fold cross validation on my input data my classifier scores at the around 98%
However, when I test the classifier on another piece of data which belongs to only one of the classes, my accuracy drops down to around 50%
Whats even more weird is that when I take two different data arrays for each class combine them. I get accuracy of around 90% with my classifier.
What's happening here? I'm confused
Class Distribution for my training data is around 55 - 45 percent