1
$\begingroup$

In the context of machine learning, for (pre-threshold) model performance metrics, I keep AUPRC and AUROC as my go-to, and then I consider things like a lift curve or cumulative gains chart as a possible handy tool for describing the model to end-users or as a help in defining a threshold.

However, I have an intuition itch that perhaps something like an "area under the lift curve" might also make sense as a performance metric (though I don't see it much discussed). My internal rebuttal initially is that a lift curve is focused on a single target class, but since its calculation includes the ratio of that class to the others, I thought perhaps that isn't a concern.

I'm left without good intuition as to whether or not (and why not if so) an area under the lift curve makes sense. What would be the justification for or against (especially in contrast to metrics like AUROC or AUPRC)?

$\endgroup$
4
  • $\begingroup$ (+1) "though I don't see it much discussed" can you please point to a reference that is discussed? I haven't seen lift-curve-AUC being used ever; I suspect you have already answered your question. It is unclear if it has a direct probabilistic interpretation; I have come across the paper ROC Curve, Lift Chart and Calibration Plot (2006) by Vuk and Curk that they give an interpretation based on the associated AUC-ROC but given people have problems interpreting AUC-ROC for practical purposes... I am bullish about the utility of AUC-Lift. $\endgroup$
    – usεr11852
    Dec 16, 2021 at 21:57
  • $\begingroup$ @usεr11852 I had seen someone else asking a tangentially related question on a forum (can't find it now) and have talked with a company that hinted they were doing something like this, but certainly nothing academic. I mainly want to check myself here as well as develop some better intuition as to why this might not work. Did you mean "bearish" on AUC-Lift btw? (I'm reading that similar to stock terminology). $\endgroup$
    – Josh
    Dec 17, 2021 at 15:30
  • $\begingroup$ (lolz. Yeah, bearish. Sorry, I probably wrote "am not bullish" decided to change it to bearish, deleted the "not" and did not change the animal.) $\endgroup$
    – usεr11852
    Dec 17, 2021 at 16:51
  • $\begingroup$ Thanks for clarifying that. As mentioned with the exception for the V&C paper linked I haven't seen something so I don't think it has huge traction because it has really low added information compared to what is already there. $\endgroup$
    – usεr11852
    Dec 18, 2021 at 13:56

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.