# random forest model is making flipped prediction

I have a very strange problem when trying to use random forests. I am trying to use RF for some binary image segmentation where I am using some texture features and I am using the scikit-learn library for this.

So, since I have a bit of unbalanced classes issue (most of the image is background), I use the balanced_subsample parameter and fit the model as:

model = RandomForestClassifier(n_estimators=250,
max_depth=12,
random_state=42,
verbose=3,
class_weight='balanced_subsample')


Th thing I am seeing is that when I segment an image using this fitted model, the segmentation would be rubbish but if I just flipped the predictions, it would be good. It is almost as if the model has learnt the inverted labeling.

Could this be something that can happen with random forests? I have checked my code and verified that the labels are passed correctly. I wanted to verify if this can happen with such a setup and any advise on debugging such a scenario?