I'm calculating the F-Score for a sandbox dataset: 100 medical patients, 20 of which have cancer. Our classifier mis-classifies 20 healthy patients as having cancer, and 5 patients with cancer as healthy, the rest it gets right.

We compute True Positives; True Negatives; False Positives; and False Negatives.

We ran into a debate about which class comes first, those that test "Positive" for cancer, or the majority class, e.g. those that are "Healthy".

Explicit Question: What is the correct true-positive rate in this dataset? Is it:

  1. # of predicted healthy patients over # of actual healthy patients
  2. # of predicted cancer patients over # of actual cancer patients

Bonus points if you can reference some literature that supports one supposition or the other.

Note, I've skimmed through a few texts on f-scores but haven't seen an explicit discussion of this point:

https://en.wikipedia.org/wiki/F1_score http://rali.iro.umontreal.ca/rali/sites/default/files/publis/SokolovaLapalme-JIPM09.pdf

Wikipedias text on precision and recall seem to suggest that "true positive" be defined by whatever "test" is being performed, and thus in this case defined as the minority class because the "test" is for cancer. However I don't find the discussion rigorous enough to convince me. If I simply describe the test in terms of testing for "healthy" patients I change the f-score, but this was just a semantic change. I would expect the f-score to have a mathematically rigorous definition.


  • $\begingroup$ I didn't clarify in an attempt to simplify, but I oversimplified. We have a classifier which mis-classifies 20 of the healthy patients as having cancer, and 5 of the cancer patients as healthy, the rest it gets correct. I've updated the question with that detail. I'm just making these numbers up for the sake of understanding what the correct convention is with regard to computing the F-Score. Fundamentally it's just a skewed dataset. I can compute the F-Score two ways, considering the true-postiive as a measure of the healthy, majority class, or as the unhealthy, minority class. $\endgroup$ Commented Jan 20, 2016 at 20:48
  • $\begingroup$ I never gave this much thought before. In reference to your "explicit question" above, in the case when one is looking for someone with cancer, I would think a 1 in the training set would mean they have cancer. Meaning you are trying to increase the TP rate in your model. As mentioned below, F1-score may not be the best metric to use, but may still have some value depending on your use case. If you use it, i would look a this article for some insights on this metric: hpl.hp.com/techreports/2009/HPL-2009-359.pdf $\endgroup$
    – Donald S
    Commented Jun 22, 2020 at 7:27

3 Answers 3


I think you've discovered that the F-score is not a very good way to evaluate a classification scheme. From the Wikipedia page you linked, there is a simplification of the formula for the F-score:

$$ {F1} = \frac {2 {TP}} {2 {TP} + {FP} + {FN}} $$

where $TP,FP,FN$ are numbers of true positives, false positives, and false negatives, respectively.

You will note that the number of true negative cases (equivalently, the total number of cases) is not considered at all in the formula. Thus you can have the same F-score whether you have a very high or a very low number of true negatives in your classification results. If you take your case 1, "# of predicted healthy patients over # of actual healthy patients", the "true negatives" are those who were correctly classified as having cancer yet that success in identifying patients with cancer doesn't enter into the F-score. If you take case 2, "# of predicted cancer patients over # of actual cancer patients," then the number of patients correctly classified as not having cancer is ignored. Neither seems like a good choice in this situation.

If you look at any of my favorite easily accessible references on classification and regression, An Introduction to Statistical Learning, Elements of Statistical Learning, or Frank Harrell's Regression Modeling Strategies and associated course notes, you won't find much if any discussion of F-scores. What you will often find is a caution against evaluating classification procedures based simply on $TP,FP,FN,$ and $TN$ values. You are much better off focusing on an accurate assessment of likely disease status with an approach like logistic regression, which in this case would relate the probability of having cancer to the values of the predictors that you included in your classification scheme. Then, as Harrell says on page 258 of Regression Modeling Strategies, 2nd edition:

If you make a classification rule from a probability model, you are being presumptuous. Suppose that a model is developed to assist physicians in diagnosing a disease. Physicians sometimes profess to desiring a binary decision model, but if given a probability they will rightfully apply different thresholds for treating different patients or for ordering other diagnostic tests.

A good model of the probability of being a member of a class, in this case of having cancer, is thus much more useful than any particular classification scheme.

  • $\begingroup$ Thanks for the references and explanation of why TP/FP/FN/TN might be bad measures for this kind of case (and in turn the F score which depends on those measures). That's the little slap in the face I needed. :) $\endgroup$ Commented Jan 27, 2016 at 17:57
  • $\begingroup$ @DavidParks glad to help. $\endgroup$
    – EdM
    Commented Jan 27, 2016 at 18:28
  • $\begingroup$ An excellent addition to this Q/A: stats.stackexchange.com/questions/76776/… $\endgroup$ Commented Apr 8, 2016 at 4:55
  • $\begingroup$ I never did like the idea of ignoring the TN result. You can unintentionally make your model perform better or worse by choosing the 1 vs 0 label accordingly. $\endgroup$
    – Donald S
    Commented Jun 22, 2020 at 7:19

Precision is what fraction actually has cancer out of the total number that you predict positive,

precision = ( number of true positives ) / (number of positives predicted by your classifier)

Recall (or true positive rate) is, what fraction of all predicted by your classifier were accurately identified.

true positive rate = true positives / ( True positive + False negative)

Coming to F-score, it is a measure of trade-off between precision and recall. Lets assume you set the thresh-hold for predicting a positive as very high. Say predicting positive if h(x) >= 0.8, and negative if h(x) < 0.8 you have huge precision but low recall. You have a precision of (15)/(15+20) = 42.8% (15 is the number of true positives 20 total cancerous, subtracted 5 which are wrongly predicted)

If you want to have a high recall [or true positive rate], it means you want to avoid missing positive cases, so you predict a positive more easily. Predict positive if h(x) >= 0.3 else predict negative. Basically having a high recall means you are avoiding a lot of false negatives. Here your true positive rate is ( 15 / (15+5) )= 75%

Having a high recall for cancer classifiers can be a good thing, you totally need to avoid false negatives here. But of course this comes at the cost of precision.

F-score measures this trade-off between precise prediction vs avoiding false negatives. Its definition can be arbitrary depending upon your classifier, lets assume it is defined as the average between precision and true positive rate.

This is not a very good F-score measure because you can have huge recall value, and very low precision [eg predicting all cases positive] and you will still end up with an F-score which is same that when your precision and recall are well balanced.

Define F score as :

              2 * (Precision * Recall) / (Precision + Recall) 

Why? If you have very low precision or recall or both, your F-score falls; and you'll know that something is wrong.

I would advise you to calculate F-score, precision and recall, for the case in which your classifier predicts all negatives, and then with the actual algorithm. If it is a skewed set you might want more training data.

Also note that it is a good idea to measure F score on the cross-validation set. It is also known as F1-score.



  • $\begingroup$ If you are using the F1-score and cross-validation, I came across some good points in this article: hpl.hp.com/techreports/2009/HPL-2009-359.pdf $\endgroup$
    – Donald S
    Commented Jun 22, 2020 at 7:16
  • $\begingroup$ Specifically, it mentions that how you calculate the F1-score is important to consider. For example, if you take the mean of the F1-scores over all the CV runs, you will get a different value than if you add up the tp,tn,fp,fn values first and then calculate the F1 score from the raw data, you will get a different (and better according to the paper) value. I tend to agree with this calculation as the other is more like taking an average of an average (something to avoid). $\endgroup$
    – Donald S
    Commented Jun 22, 2020 at 7:16
  • $\begingroup$ I believe where it says " you can have huge recall value, and very low precision [eg predicting all cases positive] " it meant to say low specificity instead of low precision, maybe? predicting all positive will indeed lead to the same f1score while having low (0) specificity $\endgroup$ Commented May 17, 2023 at 20:19

In addition to the former answers, note that the F1 score can also be solved as being:

$$ F_1 score = \frac{2}{\frac{1}{P}+\frac{1}{R}}$$

Where P = precision and R = recall = true positive rate (TPR).

This offers the advantage of referencing P and R a single time each when solving for the F1 score.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.