4
$\begingroup$

Context

I am currently working on a heavily imbalanced classification model that predicts if someone is going to list their property in the market in 6 months or not, there are 30 million properties in the UK so potentially a dataset of 30M rows to train on times the amount of times each property has had a listing. However, the model has been trained on data from only one year worth of historical transactions with LightGBM and oversampling for the minority class to decrease the imbalance a little bit (by no means is a balanced dataset, now it is just 4% for the minority class instead of 1%). The training set is divided train/val and Finally, the model is scored and evaluated in the 30 million properties on the last 6 months.

Now, the problem is not with the modelling, this is just in case I'm missing something, the problem is with the model calibration. I have run the calibration on 10 bins/deciles for 5 different versions of the model (using different points in time) and it looks very well except for the top decile. As you can see in the picture, it is a non decreasing line except for the top decile where it drastically drops. I was wondering if you have any ideas as to why this might happen.

Calibration

Improve this question
$\endgroup$
4
  • 3
    $\begingroup$ Balancing techniques, highly discouraged in the best of situations, will ruin calibration because everything you are doing will fail to apply prospectively to non-balanced new data. And what you are doing is not a calibration curve. You are arbitrarily binning the predicted values and losing all semblance of absolute predictive accuracy. Calibration curves use raw unbinned predictions, using e.g. nonparametric smoothers on out-of-sample data or on resampling-based overfitting-corrected estimates. $\endgroup$
    – Frank Harrell
    Commented Oct 7, 2022 at 20:08
  • $\begingroup$ Hi @FrankHarrell, thanks for you answer. Can you help me understand a bit more what you mentioned? So, if I undersand correclty, you mention that techniques such as under/over-sampling will ruin the calibration because the sampled set fails to relate to a real world one, I would agree with that if the metrics were low, but the calibration seems to show the opposite (except for the last decile). What do you mean with losing all semblance of absolute predicitve accuracy? $\endgroup$
    – Felipe Araya Olea
    Commented Oct 10, 2022 at 13:41
  • $\begingroup$ I mean that in general you lose the meaning of absolute predictions when not respecting your own data to fit the model. Start over with this in mind and avoid all binning when estimating calibration curves, and let's see where that leads. $\endgroup$
    – Frank Harrell
    Commented Oct 10, 2022 at 14:17
  • 1
    $\begingroup$ Thanks for your response Frank, I understand that this calubration curve might not be the best representation of reality since it is a sampled set, but I still struggle to understand what is causing this phenomenon at the top decile? $\endgroup$
    – Felipe Araya Olea
    Commented Oct 11, 2022 at 9:00

1 Answer 1

Reset to default
0
$\begingroup$

It would help if you could tell more about the descriptors you used, and label the axes.

But what I think we are seeing is that the top scoring cases include many false positives. This is a common problem in classifiers that are designed to classify an minority class. In the PR curve, it shows up as a vertical line segment down from recall=0, precision=1.

You need to examine a sample of the high scored negatives, and try to think of some feature engineering that would help classify them better.

Given that the rest of the calibration curve looks good, you might succeed. But the risk in such work is always that the available descriptors are simply not predictive enough. Imagine that you had to predict a person's weight, based only on nationality and sex - you could make a model, but it could not be a great model.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Hello @chrishmorris, thank you for your answer. I agree with you that the problem might be caused by false negatives, but why only on the top decile? Why don't we see that in the 80-90% decile for example? $\endgroup$
    – Felipe Araya Olea
    Commented Oct 11, 2022 at 8:57
  • $\begingroup$ As I stated before the decile method is not valid. Produce a true smooth calibration curve that uses no binning. Binning covers up a lot of heterogeneity especially in outer intervals. $\endgroup$
    – Frank Harrell
    Commented Oct 11, 2022 at 16:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.