Skip to main content
added 1 character in body
Source Link
Marc Claesen
  • 18.7k
  • 2
  • 55
  • 76

ROC and PR curves are primarily visual tools to select a given operating point (cutoff) of existing models for the task at hand (high recall, precision, ...). Area under either curve is a commonly used, though not very intuitive, summary statistic of the performance of models over their entire operating range.

In my opinion, AUC is useful as a scoring function in hyperparameter search, or for studies that compare the performance of different learning methods (e.g. where you don't want to examine a specific operating point). That said, AUC is far less practical to finally select a model to be used for a specific task and it is entirely irrelevant to select the model's operating point. It is perfectly possible for a suitable model for a given task to have terrible AUC under either curve.


Example

Suppose I want very high precision but don't need a lot of recall and I have two models $A$ and $B$ with corresponding PR curves (precision as function of recall): $$ \begin{align} PR_A(r) &= e^{-2r} \quad \rightarrow \quad \text{PR-AUC} \approx 0.43\\ PR_B(r) &= 0.8 \quad \rightarrow \quad \text{PR-AUC} = 0.80 \end{align} $$$$ \begin{align} PR_A(r) &= e^{-2r} \quad \rightarrow \quad \text{PR-AUC} \approx 0.43\\ PR_B(r) &= 0.80 \quad \rightarrow \quad \text{PR-AUC} = 0.80 \end{align} $$ The precision of model $A$ is higher than $B$ up to about $11\%$ recall. Its AUC is far worse, though. Given the application I presented, model $A$ is probably the best choice. Examples of such applications:

  • gene prioritization: rank genes based on association with diseases; the top ranked genes are likely to become targets for (expensive) biological analyses.
  • fraud detection: rank transactions based on potential of fraudulence; top ranked transactions may lead to lawsuits, false positives lead to counterclaims.

ROC and PR curves are primarily visual tools to select a given operating point (cutoff) of existing models for the task at hand (high recall, precision, ...). Area under either curve is a commonly used, though not very intuitive, summary statistic of the performance of models over their entire operating range.

In my opinion, AUC is useful as a scoring function in hyperparameter search, or for studies that compare the performance of different learning methods (e.g. where you don't want to examine a specific operating point). That said, AUC is far less practical to finally select a model to be used for a specific task and it is entirely irrelevant to select the model's operating point. It is perfectly possible for a suitable model for a given task to have terrible AUC under either curve.


Example

Suppose I want very high precision but don't need a lot of recall and I have two models $A$ and $B$ with corresponding PR curves (precision as function of recall): $$ \begin{align} PR_A(r) &= e^{-2r} \quad \rightarrow \quad \text{PR-AUC} \approx 0.43\\ PR_B(r) &= 0.8 \quad \rightarrow \quad \text{PR-AUC} = 0.80 \end{align} $$ The precision of model $A$ is higher than $B$ up to about $11\%$ recall. Its AUC is far worse, though. Given the application I presented, model $A$ is probably the best choice. Examples of such applications:

  • gene prioritization: rank genes based on association with diseases; the top ranked genes are likely to become targets for (expensive) biological analyses.
  • fraud detection: rank transactions based on potential of fraudulence; top ranked transactions may lead to lawsuits, false positives lead to counterclaims.

ROC and PR curves are primarily visual tools to select a given operating point (cutoff) of existing models for the task at hand (high recall, precision, ...). Area under either curve is a commonly used, though not very intuitive, summary statistic of the performance of models over their entire operating range.

In my opinion, AUC is useful as a scoring function in hyperparameter search, or for studies that compare the performance of different learning methods (e.g. where you don't want to examine a specific operating point). That said, AUC is far less practical to finally select a model to be used for a specific task and it is entirely irrelevant to select the model's operating point. It is perfectly possible for a suitable model for a given task to have terrible AUC under either curve.


Example

Suppose I want very high precision but don't need a lot of recall and I have two models $A$ and $B$ with corresponding PR curves (precision as function of recall): $$ \begin{align} PR_A(r) &= e^{-2r} \quad \rightarrow \quad \text{PR-AUC} \approx 0.43\\ PR_B(r) &= 0.80 \quad \rightarrow \quad \text{PR-AUC} = 0.80 \end{align} $$ The precision of model $A$ is higher than $B$ up to about $11\%$ recall. Its AUC is far worse, though. Given the application I presented, model $A$ is probably the best choice. Examples of such applications:

  • gene prioritization: rank genes based on association with diseases; the top ranked genes are likely to become targets for (expensive) biological analyses.
  • fraud detection: rank transactions based on potential of fraudulence; top ranked transactions may lead to lawsuits, false positives lead to counterclaims.
added 210 characters in body
Source Link
Marc Claesen
  • 18.7k
  • 2
  • 55
  • 76

ROC and PR curves are primarily visual tools to select a given operating point (cutoff) of existing models for the task at hand (high recall, precision, ...). Area under either curve is a commonly used, though not very intuitive, summary statistic of the performance of models over their entire operating range.

In my opinion, AUC is useful as a scoring function in hyperparameter search, or for studies that compare the performance of different learning methods (e.g. where you don't want to examine a specific operating point). That said, AUC is far less practical to finally select a model to be used for a specific task and it is entirely irrelevant to select the model's operating point.

  It is perfectly possible for a suitable model for a given task to have terrible AUC under either curve. For instance: suppose


Example

Suppose I want very high precision but don't need a lot of recall and I have two models $A$ and $B$ with corresponding PR curves (precision as function of recall): $$ \begin{align} PR_A(r) &= e^{-2r} \quad \rightarrow \quad \text{PR-AUC} \approx 0.43\\ PR_B(r) &= 0.8 \quad \rightarrow \quad \text{PR-AUC} = 0.80 \end{align} $$ The precision of model $A$ is higher than $B$ up to about $11\%$ recall. Its AUC is far worse, though. Given the application I presented, model A$A$ is probably the best choice. An exampleExamples of such an application is gene prioritization, in which the top ranked genes are likely to become targets for (expensive) biological analyses.applications:

  • gene prioritization: rank genes based on association with diseases; the top ranked genes are likely to become targets for (expensive) biological analyses.
  • fraud detection: rank transactions based on potential of fraudulence; top ranked transactions may lead to lawsuits, false positives lead to counterclaims.

ROC and PR curves are primarily visual tools to select a given operating point (cutoff) of existing models for the task at hand (high recall, precision, ...). Area under either curve is a commonly used, though not very intuitive summary statistic of the performance of models over their entire operating range.

In my opinion, AUC is useful as a scoring function in hyperparameter search, or for studies that compare the performance of different learning methods (e.g. where you don't want to examine a specific operating point). That said, AUC is far less practical to finally select a model to be used for a specific task and it is entirely irrelevant to select the model's operating point.

  It is perfectly possible for a suitable model for a given task to have terrible AUC under either curve. For instance: suppose I want very high precision but don't need a lot of recall and I have two models $A$ and $B$ with corresponding PR curves (precision as function of recall): $$ \begin{align} PR_A(r) &= e^{-2r} \quad \rightarrow \quad \text{PR-AUC} \approx 0.43\\ PR_B(r) &= 0.8 \quad \rightarrow \quad \text{PR-AUC} = 0.80 \end{align} $$ The precision of model $A$ is higher than $B$ up to about $11\%$ recall. Its AUC far worse, though. Given the application I presented, model A is probably the best choice. An example of such an application is gene prioritization, in which the top ranked genes are likely to become targets for (expensive) biological analyses.

ROC and PR curves are primarily visual tools to select a given operating point (cutoff) of existing models for the task at hand (high recall, precision, ...). Area under either curve is a commonly used, though not very intuitive, summary statistic of the performance of models over their entire operating range.

In my opinion, AUC is useful as a scoring function in hyperparameter search, or for studies that compare the performance of different learning methods (e.g. where you don't want to examine a specific operating point). That said, AUC is far less practical to finally select a model to be used for a specific task and it is entirely irrelevant to select the model's operating point. It is perfectly possible for a suitable model for a given task to have terrible AUC under either curve.


Example

Suppose I want very high precision but don't need a lot of recall and I have two models $A$ and $B$ with corresponding PR curves (precision as function of recall): $$ \begin{align} PR_A(r) &= e^{-2r} \quad \rightarrow \quad \text{PR-AUC} \approx 0.43\\ PR_B(r) &= 0.8 \quad \rightarrow \quad \text{PR-AUC} = 0.80 \end{align} $$ The precision of model $A$ is higher than $B$ up to about $11\%$ recall. Its AUC is far worse, though. Given the application I presented, model $A$ is probably the best choice. Examples of such applications:

  • gene prioritization: rank genes based on association with diseases; the top ranked genes are likely to become targets for (expensive) biological analyses.
  • fraud detection: rank transactions based on potential of fraudulence; top ranked transactions may lead to lawsuits, false positives lead to counterclaims.
Source Link
Marc Claesen
  • 18.7k
  • 2
  • 55
  • 76

ROC and PR curves are primarily visual tools to select a given operating point (cutoff) of existing models for the task at hand (high recall, precision, ...). Area under either curve is a commonly used, though not very intuitive summary statistic of the performance of models over their entire operating range.

In my opinion, AUC is useful as a scoring function in hyperparameter search, or for studies that compare the performance of different learning methods (e.g. where you don't want to examine a specific operating point). That said, AUC is far less practical to finally select a model to be used for a specific task and it is entirely irrelevant to select the model's operating point.

It is perfectly possible for a suitable model for a given task to have terrible AUC under either curve. For instance: suppose I want very high precision but don't need a lot of recall and I have two models $A$ and $B$ with corresponding PR curves (precision as function of recall): $$ \begin{align} PR_A(r) &= e^{-2r} \quad \rightarrow \quad \text{PR-AUC} \approx 0.43\\ PR_B(r) &= 0.8 \quad \rightarrow \quad \text{PR-AUC} = 0.80 \end{align} $$ The precision of model $A$ is higher than $B$ up to about $11\%$ recall. Its AUC far worse, though. Given the application I presented, model A is probably the best choice. An example of such an application is gene prioritization, in which the top ranked genes are likely to become targets for (expensive) biological analyses.