While writing a paper, I was trying to motivate the use of PR curves over ROC curves and read The Relationship Between Precision Recall and ROC Curves, which was very helpful, but raised a methodological question that I haven't been able to answer. The article specifically addresses methodological soundness as follows (quoting from page 5 (237 in print)):
An important methodological issue must be addressed when building a convex hull in ROC space or an achievable curve in PR space. When constructing a ROC curve (or PR curve) from an algorithm that outputs a probability, the following approach is usually taken: (...). Thus each point in ROC space or PR space represents a specific classifier, with a threshold for calling an example positive. Building the convex hull can be seen as constructing a new classifier, as one picks the best points. Therefore it would be methodologically incorrect to construct a convex hull or achievable PR curve by looking at performance on the test data and then constructing a convex hull. To combat this problem, the convex hull must be constructed using a tuning set as follows: First, use the method described above to find a candidate set of thresholds on the tuning data. Then, build a convex hull over the tuning data. Finally use the thresholds selected on the tuning data, when building an ROC or PR curve for the test data. While this test-data curve is not guaranteed to be a convex hull, it preserves the split between training data and testing data.
I do not understand what is meant by creating a new classifier through the convex hull. My understanding of the training/test split is that its' purpose is selecting parameters of specific algorithms. Basically, I take a bunch of data and one or more algorithms, initialize each with a different threshold and observe their output given the training set. So, I would have one PR/ROC curve for each algorithm, spanning all thresholds. I use this information to choose a specific (optimal for this application) threshold and deploy the system with this threshold. The test set would then be used to decide which of the analyzed algorithms performs best. However, fixing a threshold clearly does not generate a PR/ROC curve for the test set, but only individual points.
My question is: is it methodologically sound to use the entire dataset (test+training) to generate PR curves, assuming I want to reason how the threshold of an algorithm influences its' performance on this set? Is it similarly sound to reason about the relative performance of algorithms with those curves?
I'm not sure whether it is relevant, but I'm aiming for PR curves as opposed to ROC, as the consensus appears to be that their graphical representation is more informative, especially for highly imbalanced sets.