I am doing research on binary classification. I have three classification models work on one data set. I have no True Positive, True Negative, False Positive, and False Negative metrics, each model produces value between 0 and 1. I calculate the four metrics: Root Mean Square Error (RMSE), Normalized Mean Square Error (NMSE), Mean Absolute Error (MAE), and Symmetric Mean Absolute Percentage Error (SMAPE) in order to find the accuracy of each model. I want now to calculate the Precision, Recall to draw the PR Curves and get the final verdict about the better classification model among the three models. Is there any way to calculate Precision and Recall, and draw PR Curve in terms of RMSE, NMSE, MAE, and SMAPE ? If not, what should I do to get True Positive, True Negative, False Positive, and False Negative values for each models? Naturally, they neither produce 0 or 1.
Like a receiver operating curve (ROC), the precision-recall-curve is a curve constructed by the (precision; recall) pairs you get when you vary the classification threshold that "hardens" your continous output score into class assignment.
So you don't get the curve from RMSE & Co, it is a diffent way of summarizing your validation results.
However, you can construct metrics analogue to true positives, precision, recall etc. from [0, 1] output and even for [0, 1] continuous reference.
See e.g. our paper: C. Beleites, R. Salzer and V. Sergo: Validation of Soft Classification Models using Partial Class Memberships: An Extended Concept of Sensitivity & Co. applied to Grading of Astrocytoma Tissues Chemom. Intell. Lab. Syst., 122 (2013), 12 - 22.
If you want a contingency table, you need to decide on a cut-off somewhere in $[0,1]$. ROC and PR curves essentially visualize all possible contingency tables your model can produce by computing tables at every cut-off. RMSE, MAE and co are all regression metrics, don't use those to assess classification models. It's quite easy to make perfect classifiers whose decision values would yield poor RMSE.
Note that deciding which model is best based on PR curves is not trivial. Typically you will see that the curves of different models cross (e.g. one is better at low recall while another is better at high recall). Which one is best then?
You can think of PR and ROC curves is exploratory tools, rather than decision criteria. Some obvious deduced metrics are the area under the PR/ROC curve, which quantify how well models works over all thresholds; but whether or not AUC is really what you want depends entirely on the application.