I'm testing the performance of two binary classifiers on a simulated dataset. I'm seeing that classifier 1 has a higher MSE (mean squared error / classification error) than classifier 2, but classifier 1 has greater area under ROC curve than that of classifier 2. These are seemingly contradictory, because if using MSE for selection then classifier 2 is better, but if using area under ROC then classifier 1 is better. Does anyone have any advice on how to rationalize these results?
Some more info: it seems like classifier 2 is essentially only predicting one class for all of its predictions, whereas classifier 1 is more balanced in its predictions.