Return to Answer

Using ROC can miss the low precision where FP > TP because ROC only look at TPR and FPR. I think what we need to ask is:

1. Will FP > TP happen with your data and the model.
2. Does FP > TP matter for your business problem.

If the answer is yes for both, then ROC only is not fit for use.

If the problem is identifying shoplifters and FP will alarm the police. Almost all the shop customers are honest, then catching more honest customers as shoplifters than real ones will cause angry customers and police. Then looking at only ROC will not be a good idea to measure the model.

If the problem is identifying potentially fatal food for toddlers, FP > TP may not be a big issue because there will be so many safe food (TN) as long as TPR is really high. Then ROC will be fit for use to measure the model.

FP: False Positive
TP: True Positive
FPR: False Positive Rate
TPR: True Positive Rate

Suppose I am running a risky loan business where many customers can be risky. If I evaluate the business performance with ROC only, I think the performance is good because TPR is high and FPR is low, although actually the business is losing money because of FP > TP. By looking at PR, it will tell the low precision and I will understand the business performance is bad as I am looking money.

Using ROC can miss the low precision where FP > TP because ROC only look at TPR and FPR. I think what we need to ask is:

1. Will FP > TP happen with your data and the model.
2. Does FP > TP matter for your business problem.

If the answer is yes for both, then ROC only is not fit for use.

If the problem is identifying shoplifters and FP will alarm the police. Almost all the shop customers are honest, then catching more honest customers as shoplifters than real ones will cause angry customers and police. Then looking at only ROC will not be a good idea to measure the model.

If the problem is identifying potentially fatal food for toddlers, FP > TP may not be a big issue because there will be so many safe food (TN) as long as TPR is really high. Then ROC will be fit for use to measure the model.

FP: False Positive
TP: True Positive
FPR: False Positive Rate
TPR: True Positive Rate

Suppose I am running a risky loan business where majority of the customers are risky. If I evaluate the business performance with ROC only, I think the performance is good because TPR is high and FPR is low, although actually the business is losing money because of FP > TP. By looking at PR, it will tell the low precision and I will understand the business performance is bad as I am looking money.

Using ROC can miss the low precision where FP > TP because ROC only look at TPR and FPR. I think what we need to ask is:

1. Will FP > TP happen with your data and the model.
2. Does FP > TP matter for your business problem.

If the answer is yes for both, then ROC only is not fit for use.

If the problem is identifying shoplifters and FP will alarm the police. Almost all the shop customers are honest, then catching more honest customers as shoplifters than real ones will cause angry customers and police. Then looking at only ROC will not be a good idea to measure the model.

If the problem is identifying potentially fatal food for toddlers, FP > TP may not be a big issue because there will be so many safe food (TN) as long as TPR is really high. Then ROC will be fit for use to measure the model.

FP: False Positive
TP: True Positive
FPR: False Positive Rate
TPR: True Positive Rate

Using ROC can miss the low precision where FP > TP because ROC only look at TPR and FPR. I think what we need to ask is:

1. Will FP > TP happen with your data and the model.
2. Does FP > TP matter for your business problem.

If the answer is yes for both, then ROC only is not fit for use.

If the problem is identifying shoplifters and FP will alarm the police. Almost all the shop customers are honest, then catching more honest customers as shoplifters than real ones will cause angry customers and police. Then looking at only ROC will not be a good idea to measure the model.

If the problem is identifying potentially fatal food for toddlers, FP > TP may not be a big issue because there will be so many safe food (TN) as long as TPR is really high. Then ROC will be fit for use to measure the model.

FP: False Positive
TP: True Positive
FPR: False Positive Rate
TPR: True Positive Rate

Suppose I am running a risky loan business where many customers can be risky. If I evaluate the business performance with ROC only, I think the performance is good because TPR is high and FPR is low, although actually the business is losing money because of FP > TP. By looking at PR, it will tell the low precision and I will understand the business performance is bad as I am looking money.

Using ROC can miss the low precision where FP > TP because ROC only look at TPR and FPR. I think what we need to ask is:

1. Will FP > TP happen with your data and the model.
2. Does FP > TP matter for your business problem.

If the answer is yes for both, then ROC only is not fit for use.

If the problem is identifying shoplifters and FP will alarm the police. Almost all the shop customers are honest, then catching more honest customers as shoplifters than real ones will cause angry customers and police. Then looking at only ROC will not be a good idea to measure the model.

If the problem is identifying potentially fatal food for toddlers, FP > TP may not be a big issue because there will be so many safe food (TN) as long as TPR is really high. Then ROC will be fit for use to measure the model.

FP: False Positive
TP: True Positive
FPR: False Positive Rate
TPR: True Positive Rate

Using ROC can miss the low precision where FP > TP because ROC only look at TPR and FPR. I think what we need to ask is:

1. Will FP > TP happen with your data and the model.
2. Does FP > TP matter for your business problem.

If the answer is yes for both, then ROC only is not fit for use.

If the problem is identifying shoplifters and FP will alarm the police. Almost all the shop customers are honest, then catching more honest customers as shoplifters than real ones will cause angry customers and police. Then looking at only ROC will not be a good idea to measure the model.

If the problem is identifying potentially fatal food for toddlers, FP > TP may not be a big issue because there will be so many safe food (TN) as long as TPR is really high. Then ROC will be fit for use to measure the model.

FP: False Positive
TP: True Positive
FPR: False Positive Rate
TPR: True Positive Rate