7
$\begingroup$

Due to my background as a pure biologist, I've been struggling with the comment acquired from a reviewer about the accuracy test used in my regression study. While I stick to MSE, MAE and R2 as the parameters to determine accuracy of my regression model (Support Vector Regression and Simple Linear Regression), one reviewer asks me to perform F1 score, PR or ROC curve with the data.

The reviewer has noted that "F1 score, PR or ROC curve are not specific to classification models only." With my limited knowledge, I cannot find any evidences of applying such parameters with regression study.

It would be very kind of you if anyone could provide me the source of such application. Either R or python packages for applying such test with regression study would be really appreciated.

Best regards, Kaj

$\endgroup$
7
  • $\begingroup$ I assume that the reviewer must think you're performing a logistic regression. If you include the term "logistic" in your google search, you'll find more relevant references. $\endgroup$
    – JimB
    Commented Nov 18, 2020 at 4:00
  • $\begingroup$ Thank you very much for your response!! The fact is that I've performed Support Vector Regression, and Simple Linear Regression. I do not have any clue about such parameters in these models. I would improve my question accordingly. $\endgroup$
    – Tofu King
    Commented Nov 18, 2020 at 4:05
  • $\begingroup$ I hadn’t considered that, but @JimB raises a reasonable point. Are you doing a logistic regression? Even the Wikipedia article on ROC gives some references to extensions to continuous responses, so it isn’t that obscure, but it would make more sense to ask for ROC if the regression is a logistic regression. $\endgroup$
    – Dave
    Commented Nov 18, 2020 at 4:05
  • $\begingroup$ I'm so sorry for my unclear question. I've just edited the question to include the regression models I'm considering. $\endgroup$
    – Tofu King
    Commented Nov 18, 2020 at 4:10
  • 2
    $\begingroup$ Maybe the reviewer expects you to bin predictions and true outputs, thus producing a "classes", and applying classification metrics to this. Perhaps they want it so that they can see interpretable results like "90% observations with values between 0 and 1 were predicted correctly". $\endgroup$
    – Euphe
    Commented Nov 18, 2020 at 7:42

1 Answer 1

9
$\begingroup$

F1 score, PR or ROC curve are not specific to classification models only.

I have never seen the F1 score or ROC used to evaluate a numerical prediction. I am unfamiliar with "PR".

The definition of the F1 score crucially relies on precision and recall, or positive/negative predictive value, and I do not see how it can reasonably be generalized to a numerical forecast.

The ROC curve plots the true positive rate against the false positive rate as a threshold varies. Again, it relies on a notion of "true positive" and "false positive", and I don't see how these can be applied to numerical predictions.

All that is not to say that efforts have not been made to apply these concepts to numerical forecasts.

It would feel a lot like hammering square pegs into round holes to me, though. I would say that there is a reason why I (we?) haven't seen this a lot: it's unintuitive, and it does not provide the information that standard error measures like the MAE or the MSE do. Honestly, if I got a paper for review that used F1/ROC to evaluate numerical predictions, I would recommend that they throw these out and use more standard error measures.

My recommendation: ask the editor to communicate to the reviewer that you need more information on applying F1 and ROC in your case. Maybe the reviewer can provide a reference or two? You may want to provide a link to this CV thread as an indication that you did do your homework and asked statistical experts (cough), and that the experts were similarly bewildered.

The best possible outcome would be if your reviewer posted their thoughts here.

$\endgroup$
6
  • 1
    $\begingroup$ just a guess, but "PR" could simply be short for Precision & Recall $\endgroup$
    – nope
    Commented Nov 18, 2020 at 6:43
  • $\begingroup$ I politely implied the reviewer for something like 'provision of evidence'. It seemed to make the reviewer really angry. But the reviewer did not provide me any source as well. Sadly, I've already got rejection with this in my mind. After reading your answer, I would like to thank you for making me feel normal again. $\endgroup$
    – Tofu King
    Commented Nov 18, 2020 at 7:03
  • 3
    $\begingroup$ @TofuKing you need to be careful with the wording. Don't "imply" and don't invoke principles such as "provision of evidence". That's not helpful, and likely to annoy the reviewer. Show that you've done your homework, state the facts ("We know ROC/PR/F1 are defined on binary classification tasks, have been extended to multiclass classification, but we're not aware of their application in regression") and ask the reviewer for clarification ("Could the reviewer please point you to the literature they have in mind?"). $\endgroup$
    – Calimo
    Commented Nov 18, 2020 at 7:17
  • 2
    $\begingroup$ @TofuKing I am with Calimo for being polite in your skepticism of the reviewer. We have an academia Stack if you want to post there about how to handle a reviewer who appears to be dead wrong. I do wonder if it would help your paper to point out that support vector regression is different from the support vector machine classifier, hence the use of the performance metrics you have chosen. The reviewer might be wrong and even vindictive about having that pointed out, but if a professional is confused, some of that is on you to clarify in the article. $\endgroup$
    – Dave
    Commented Nov 18, 2020 at 11:07
  • 2
    $\begingroup$ That said, good for you for having the courage to (politely!) tell the reviewer, “I think you’re wrong.” $\endgroup$
    – Dave
    Commented Nov 18, 2020 at 11:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.