0
$\begingroup$

I am doing research on binary classification. I have three classification models work on one data set. I have no True Positive, True Negative, False Positive, and False Negative metrics, each model produces value between 0 and 1. I calculate the four metrics: Root Mean Square Error (RMSE), Normalized Mean Square Error (NMSE), Mean Absolute Error (MAE), and Symmetric Mean Absolute Percentage Error (SMAPE) in order to find the accuracy of each model. I want now to calculate the Precision, Recall to draw the PR Curves and get the final verdict about the better classification model among the three models. Is there any way to calculate Precision and Recall, and draw PR Curve in terms of RMSE, NMSE, MAE, and SMAPE ? If not, what should I do to get True Positive, True Negative, False Positive, and False Negative values for each models? Naturally, they neither produce 0 or 1.

$\endgroup$
  • 2
    $\begingroup$ Welcome to cross validated! But please expand your acronymns to give people who are not familiar with the acronyms a chance to answer. $\endgroup$ – cbeleites Jan 29 '15 at 19:37
5
$\begingroup$

Like a receiver operating curve (ROC), the precision-recall-curve is a curve constructed by the (precision; recall) pairs you get when you vary the classification threshold that "hardens" your continous output score into class assignment.

So you don't get the curve from RMSE & Co, it is a diffent way of summarizing your validation results.


However, you can construct metrics analogue to true positives, precision, recall etc. from [0, 1] output and even for [0, 1] continuous reference.

See e.g. our paper: C. Beleites, R. Salzer and V. Sergo: Validation of Soft Classification Models using Partial Class Memberships: An Extended Concept of Sensitivity & Co. applied to Grading of Astrocytoma Tissues Chemom. Intell. Lab. Syst., 122 (2013), 12 - 22.

$\endgroup$
2
$\begingroup$

If you want a contingency table, you need to decide on a cut-off somewhere in $[0,1]$. ROC and PR curves essentially visualize all possible contingency tables your model can produce by computing tables at every cut-off. RMSE, MAE and co are all regression metrics, don't use those to assess classification models. It's quite easy to make perfect classifiers whose decision values would yield poor RMSE.

Note that deciding which model is best based on PR curves is not trivial. Typically you will see that the curves of different models cross (e.g. one is better at low recall while another is better at high recall). Which one is best then?

You can think of PR and ROC curves is exploratory tools, rather than decision criteria. Some obvious deduced metrics are the area under the PR/ROC curve, which quantify how well models works over all thresholds; but whether or not AUC is really what you want depends entirely on the application.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.