Calibrating the probabilities of Ridge Classifier on imbalanced dataset

Question

I have a classification project on an imbalanced dataset (HomeCredit Kaggle dataset) and I have chosen Ridge Classifier (sklearn's implementation) as the most efficient both in terms of time and in terms of my performance indicator (ROC AUC score).

The initial dataset is widely imbalanced (0.92 TARGET = 0, 0.08 TARGET = 1) so I had to perform SMOTE oversampling to the train dataset to bring back the ratio to 50/50. There are about 450 features with 300k samples.

On a previous posts, commenters seemed dubious on the merits of oversampling. I have tried different ratios (from the intial ratio to 1 or 50/50) and there has been a linear improvement of ROC AUC when increasing the ratio and bringing it up to 1. So I think this is necessary in this case, possibly because of the high number of features.

Since I am requested to provide probabilities for my predictions, I have extended the RidgeClassifier class with softmax as such (from research on another post) :

class RidgeClassifierWithProba(RidgeClassifier):
    def predict_proba(self, X):
        d = self.decision_function(X)
        d_2d = np.c_[-d, d]
        return softmax(d_2d)

The final scores I get from my model are relatively good with a final ROC AUC score of 0.76 when taking into account those probabilities (0.70 with just the predictions). Top Kagglers have only been able to reach 0.805 during the competition so I think it is close enough.

The problem is that the histogram of probabilities show that there is no separation between the 2 classes, with an almost normally distributed density :

I tried to implement the following CalibratedClassifierCV functions to improve this distribution :

pipe_iso = CalibratedClassifierCV(final_pipe, cv=2, method="isotonic")
pipe_sig = CalibratedClassifierCV(final_pipe, cv=2, method="sigmoid")

But it make the probability distribution even worse, with all of them skewed towards the majority class :

Am I doing something wrong here? Is my model not good enough to have a separated probabilities histogram? What else can I do to calibrate my probabilities?

Edit : As requested by @Ben Reiniger, I uploaded my work on Kaggle : Link to Kagggle and am posting more code below. This is my final pipeline :

#Defining our final pipeline
final_pipe = Pipeline([
    ('cat_encode', cat_encode),
    ('imputation', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler()),
    ('var', boruta),
    ('os', ADASYN()),
    ('ridge',RidgeClassifierWithProba(alpha=6.967)),
    ], memory="./Cache/")

If it makes any difference, this is an Imbalearn pipeline and not a Sklearn pipeline since it implements oversampling.

Cat encoder is a Column Transformer that imputes categorical fields based on the number of unique categories (One Hot when nunique <= 5 and WOE when nunique >5). Boruta is a FunctionTransformer built from a Boruta Selector.

Here are my results on the test set (results on the train set are similar) :

To clarify my statement about the ROC AUC Score, when I feed only the predicted results from the base Ridge Classifier into the roc_auc_score function it returns 0.7 while when I feed it the probabilities it returns 0.76 (because it smoothes the curve).

final_pipe.fit(X_train_reduced, y_train)
train_predictions = final_pipe.predict(X_train_reduced)
proba_train = final_pipe.predict_proba(X_train_reduced)[:, 1]
test_predictions = final_pipe.predict(X_test)
proba_test = final_pipe.predict_proba(X_test)[:, 1]

roc_auc_score(y_test, test_predictions) #Returns 0.7
roc_auc_score(y_test, proba_test) #Returns 0.76

When calibrating the probablities with my isotonic calibrater, I get the following "stacked" bar chart, which looks good but actually the colors are the probabilities for each class. So unless I'm misunderstanding something it means that with a probability threshold of 0.5 this would predict that all test results are in the Majority (blue) class.

test_prob = pipe_iso.predict_proba(X_test)

plt.hist(test_prob, stacked=True)
plt.show()

For comparison this is the same chart with my uncalibrated pipeline :

Answer 1

The problem is that the histogram of probabilities show that there is no separation between the 2 classes, with an almost normally distributed density

With an AUROC of 0.76, I think this is expected. If the distribution density was bimodal (and performance not trash, so that most of the data with values in the two peaks are correctly sorted), you'd get a large number of pairs put in the correct order (AUROC having one interpretation as "ratio of positive-negative pairs that appear in the correct order by score). And just because the scores are unimodel doesn't mean that the classes aren't somewhat well separated, hence my suggestion in a comment to split the bars by class.

But it make the probability distribution even worse, with all of them skewed towards the majority class

That's not worse! The majority class does dominate, and so the probabilities should be skewed towards it.

Note that the last bin in the sigmoid plot has very little data in it, so take the very low positive rate with a grain of salt. (You could also use strategy='quantile' instead of the default 'uniform' to make the bins in the plot equal-volume rather than equal-width.)