Should Scikit-Learn CalibratedClassifierCV isotonic mode use bucketed rates instead of the actual targets? This is less a question about sklearn's implementation, and more theoretical. I find it weird that we'd do isotonic regression against target values in {0, 1} because that could result in very jagged results. Why not use the calibration curve to do the calibration?
So to give you an example, I had a binary classification problem which was imbalanced towards 95% probability of a positive. I trained it on rebalanced data and tried validating it on unbalanced data. Of course that didn't turn out so great, so I went for calibration via isotonic regression.
So here's the "normal" way of doing it:
def predict_via_isotonic_calibration(y_true, y_prob):
    """
    y_true is an array of binary targets
    y_prob is an array of predicted probabilities from an uncalibrated classifier
    """
    iso_reg = IsotonicRegression(out_of_bounds='clip').fit(y_prob, y_true)
    calibrated_y_prob = iso_reg.predict(y_prob)
    return calibrated_y_prob

Which gave me this (calibrated vs uncalibrated):

Whereas I think it should be more like:
def predict_via_isotonic_calibration(y_true, y_prob, n_bins=80):
    y, x = calibration_curve(y_true, y_prob, n_bins=n_bins)
    iso_reg = IsotonicRegression(out_of_bounds='clip').fit(x, y)
    calibrated_y_prob = iso_reg.predict(y_prob)
    return calibrated_y_prob

Which gave me this much nicer calibration:

So what gives? Is my idea a thing? Or is it wrong for some reason I'm overlooking.
 A: After examining the problem more I convinced myself that the official way to do it is the right way. My objection was originally that doing regression against {0, 1} could result in jagged results. But actually, that's the basis for logistics regression! Isotonic regression is not fundamentally different in that sense. Here's a bad drawing to explain why this is okay

In the same way, neither logistic regression, nor isotonic regression will have any problem with regressing to a set containing just 0s and 1s.
As for why my first graph looks "jagged"

As a commenter made me realise, it's just an artifact of uniform binning combined with the fact that the dataset is imbalanced towards 0.95. Some bins just happened to have a 1 point in them and that point was predicted wrong even after calibration.
So really, me trying to use bucketed rates instead of the actual target values is just a way of interfering with the intended function of isotonic regression. In fact, it turns out that the roc_auc score is still better for the "official" recalibration method vs my proposed one.
