I was confused about the differences between the F1 score, Dice score and IoU (intersection over union). By now I found out that F1 and Dice mean the same thing (right?) and IoU has a very similar formula to the other two.

  • F1 / Dice: $$\frac{2TP}{2TP+FP+FN}$$
  • IoU / Jaccard: $$\frac{TP}{TP+FP+FN}$$

Are there any practical differences or other things worth noting except that F1 weights the true-positives higher? Is there a situation where I'd use one but not the other?

  • Apparently the Jaccard coefficient is also the same as IoU – pietz Apr 15 '17 at 13:06
  • I'd be especially interested if some of these (now 4) measurements are only meant for binary data. – pietz Apr 18 '17 at 7:46
up vote 17 down vote accepted

You're on the right track.

So a few things right off the bat. From the definition of the two metrics, we have that IoU and F score are always within a factor of 2 of each other: $$ F/2 \leq IoU \leq F $$ and also that they meet at the extremes of one and zero under the conditions that you would expect (perfect match and completely disjoint).

Note also that the ratio between them can be related explicitly to the IoU: $$ IoU/F = 1/2 + IoU/2 $$ so that the ratio approaches 1/2 as both metrics approach zero.

But there's a stronger statement that can be made for the typical application of classification a la machine learning. For any fixed "ground truth", the two metrics are always positively correlated. That is to say that if classifier A is better than B under one metric, it is also better than classifier B under the other metric.

It is tempting then to conclude that the two metrics are functionally equivalent so the choice between them is arbitrary, but not so fast! The problem comes when taking the average score over a set of inferences. Then the difference emerges when quantifying how much worse classifier B is than A for any given case.

In general, the IoU metric tends to penalize single instances of bad classification more than the F score quantitatively even when they can both agree that this one instance is bad. Similarly to how L2 can penalize the largest mistakes more than L1, the IoU metric tends to have a "squaring" effect on the errors relative to the F score. So the F score tends to measure something closer to average performance, while the IoU score measures something closer to the worst case performance.

Suppose for example that the vast majority of the inferences are moderately better with classifier A than B, but some of them of them are significantly worse using classifier A. It may be the case then that the F metric favors classifier A while the IoU metric favors classifier B.

To be sure, both of these metrics are much more alike than they are different. But both of them suffer from another disadvantage from the standpoint of taking averages of these scores over many inferences: they both overstate the importance of sets with little-to-no actual ground truth positive sets. In the common example of image segmentation, if an image only has a single pixel of some detectable class, and the classifier detects that pixel and one other pixel, its F score is a lowly 2/3 and the IoU is even worse at 1/2. Trivial mistakes like these can seriously dominate the average score taken over a set of images. In short, it weights each pixel error inversely proportionally to the size of the selected/relevant set rather than treating them equally.

There is a far simpler metric that avoids this problem. Simply use the total error: FN + FP (e.g. 5% of the image's pixels were miscategorized). In the case where one is more important than the other, a weighted average may be used: $c_0$FP + $c_1$FN.

  • willem, i couldn't have asked for a better answer. thank you very much for taking the time. – pietz Apr 28 '17 at 7:01
  • I tried your total error approach and just wanted to add that it doesn't work well with constant imbalances between positives and negatives. Imagine an entire dataset of images where only one pixel makes up the ground truth segmentation. Neural networks might learn fairly quickly that an empty prediction is always 99.9% accurate using the total error. By going with IoU or DSC we pressure the network into finding a segmentation due to the same reasons you mentioned above. So, in the end it's very problem depend. – pietz Aug 10 '17 at 8:43
  • Can someone help me reconcile the following two statements?: 1: "That is to say that if classifier A is better than B under one metric, it is also better than classifier B under the other metric." and 2: "It may be the case then that the F metric favors classifier A while the IoU metric favors classifier B." – Matt Kleinsmith Nov 12 '17 at 19:58
  • The former refers to a score of a single inference, and the latter refers to an average score over a set of inferences (e.g. a set of images). – willem Nov 13 '17 at 21:31

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.