Skip to main content
added 124 characters in body
Source Link
Lucas Morin
  • 1.7k
  • 18
  • 32

AUC measure the area under the ROC curve, wich is built by taking (TPR,FPR) for all the possible thresholds. See here for the whole definition and construction : What does AUC stand for and what is it?

It is a good metric for simpletextbook problems, with separable classes and when your decision has no practical impact. It let you want to look at multiple thresholds at once to keep your possibilities open. In general, it is not a good metric if you want only one threshold, ie. to make a decision. It is especially bad with imbalanced data set as you have seen. See Frank Harrell answer to the question mentionned above.

There might be a bigger problem with your approach : with imbalanced data set the class are probably not separable. So it doesn't make sense to use a classifier in the ML sense. You might want to use a probabilistic approach to model your dataoutput, then use a good metric to choose how you set a threshold and you separate your classes. That metric would heavily depend on what you are trying to model and practical implication associated to your decision. See Frank Harrel blog post: https://www.fharrell.com/post/classification/

AUC measure the area under the ROC curve, wich is built by taking (TPR,FPR) for all the possible thresholds. See here for the whole definition and construction : What does AUC stand for and what is it?

It is a good metric for simple problems, when you want to look at multiple thresholds at once to keep your possibilities open. In general, it is not a good metric if you want only one threshold, ie. to make a decision. It is especially bad with imbalanced data set as you have seen. See Frank Harrell answer to the question mentionned above.

There might be a bigger problem with your approach : with imbalanced data set the class are probably not separable. So it doesn't make sense to use a classifier in the ML sense. You might want to use a probabilistic approach to model your data, then use a good metric to choose how you separate your classes. See Frank Harrel blog post: https://www.fharrell.com/post/classification/

AUC measure the area under the ROC curve, wich is built by taking (TPR,FPR) for all the possible thresholds. See here for the whole definition and construction : What does AUC stand for and what is it?

It is a good metric for textbook problems with separable classes and when your decision has no practical impact. It let you look at multiple thresholds at once. In general, it is not a good metric if you want only one threshold, ie. to make a decision. It is especially bad with imbalanced data set as you have seen. See Frank Harrell answer to the question mentionned above.

There might be a bigger problem with your approach : with imbalanced data set the class are probably not separable. So it doesn't make sense to use a classifier in the ML sense. You might want to use a probabilistic approach to model your output, then use a good metric to choose how you set a threshold and you separate your classes. That metric would heavily depend on what you are trying to model and practical implication associated to your decision. See Frank Harrel blog post: https://www.fharrell.com/post/classification/

Source Link
Lucas Morin
  • 1.7k
  • 18
  • 32

AUC measure the area under the ROC curve, wich is built by taking (TPR,FPR) for all the possible thresholds. See here for the whole definition and construction : What does AUC stand for and what is it?

It is a good metric for simple problems, when you want to look at multiple thresholds at once to keep your possibilities open. In general, it is not a good metric if you want only one threshold, ie. to make a decision. It is especially bad with imbalanced data set as you have seen. See Frank Harrell answer to the question mentionned above.

There might be a bigger problem with your approach : with imbalanced data set the class are probably not separable. So it doesn't make sense to use a classifier in the ML sense. You might want to use a probabilistic approach to model your data, then use a good metric to choose how you separate your classes. See Frank Harrel blog post: https://www.fharrell.com/post/classification/