I have about 1000 samples with 20 features and I'm using Random Forest to predict a binary class.
I'm trying to apply the probability calibration process as described on scikit using CalibratedClassifierCV.
I've noticed that even though the log-loss improves after the process, the reliability graph looks much worse after calibration. I've used make_classification to create samples for this post, and this is the reliability graph I'm getting prior to calibration:
Need to say that when I'm using my own real data, the calibrated graph looks even worse (all of the probabilities are in the range of 0.45 to 0.55).
Here is part of the code that I'm using for calibration:
from sklearn.calibration import CalibratedClassifierCV, calibration_curve from sklearn.datasets import make_classification from sklearn.model_selection import StratifiedKFold from sklearn.cross_validation import StratifiedShuffleSplit from sklearn.ensemble import RandomForestClassifier from sklearn.metrics import log_loss import matplotlib.pyplot as plt X,y = make_classification(n_samples=1000, n_features=20, n_informative=15, n_redundant=0, n_repeated=0, n_classes=2, random_state=1, shuffle=False) train, test = list(StratifiedShuffleSplit(y, 1, random_state=1)) # split the data to training and testing set X_train, X_test = X[train], X[test] y_train, y_test = y[train], y[test] rfc = RandomForestClassifier(n_estimators=300, random_state=1) cv = StratifiedKFold(n_splits=3, shuffle=False, random_state=1) sigmoid = CalibratedClassifierCV(rfc, cv=cv, method='sigmoid') sigmoid.fit(X_train, y_train) rfc.fit(X_train, y_train) probs = sigmoid.predict_proba(X_test)
I hoped you could help me understand why the calibration process doesn't seem to work so well.