# F1/Dice-Score vs IoU

I was confused about the differences between the F1 score, Dice score and IoU (intersection over union). By now I found out that F1 and Dice mean the same thing (right?) and IoU has a very similar formula to the other two.

• F1 / Dice: $$\frac{2TP}{2TP+FP+FN}$$
• IoU / Jaccard: $$\frac{TP}{TP+FP+FN}$$

Are there any practical differences or other things worth noting except that F1 weights the true-positives higher? Is there a situation where I'd use one but not the other?

• Apparently the Jaccard coefficient is also the same as IoU Apr 15, 2017 at 13:06
• I'd be especially interested if some of these (now 4) measurements are only meant for binary data. Apr 18, 2017 at 7:46
• I made some contour plots for IoU and F1 for intuitive understanding inspired by one of the answers above. I posted my plots on tomkwok.com/posts/iou-vs-f1. Jan 19, 2021 at 10:50
• stats.stackexchange.com/a/55802/3277 answers why we need that 2 multiplier in Dice. Nov 8, 2021 at 9:14
• @Tom Kwok: Your link is not working Mar 13, 2023 at 14:54

You're on the right track.

So a few things right off the bat. From the definition of the two metrics, we have that IoU and F score are always within a factor of 2 of each other: $$F/2 \leq IoU \leq F$$ and also that they meet at the extremes of one and zero under the conditions that you would expect (perfect match and completely disjoint).

Note also that the ratio between them can be related explicitly to the IoU: $$IoU/F = 1/2 + IoU/2$$ so that the ratio approaches 1/2 as both metrics approach zero.

But there's a stronger statement that can be made for the typical application of classification a la machine learning. For any fixed "ground truth", the two metrics are always positively correlated. That is to say that if classifier A is better than B under one metric, it is also better than classifier B under the other metric.

It is tempting then to conclude that the two metrics are functionally equivalent so the choice between them is arbitrary, but not so fast! The problem comes when taking the average score over a set of inferences. Then the difference emerges when quantifying how much worse classifier B is than A for any given case.

In general, the IoU metric tends to penalize single instances of bad classification more than the F score quantitatively even when they can both agree that this one instance is bad. Similarly to how L2 can penalize the largest mistakes more than L1, the IoU metric tends to have a "squaring" effect on the errors relative to the F score. So the F score tends to measure something closer to average performance, while the IoU score measures something closer to the worst case performance.

Suppose for example that the vast majority of the inferences are moderately better with classifier A than B, but some of them of them are significantly worse using classifier A. It may be the case then that the F metric favors classifier A while the IoU metric favors classifier B.

To be sure, both of these metrics are much more alike than they are different. But both of them suffer from another disadvantage from the standpoint of taking averages of these scores over many inferences: they both overstate the importance of sets with little-to-no actual ground truth positive sets. In the common example of image segmentation, if an image only has a single pixel of some detectable class, and the classifier detects that pixel and one other pixel, its F score is a lowly 2/3 and the IoU is even worse at 1/2. Trivial mistakes like these can seriously dominate the average score taken over a set of images. In short, it weights each pixel error inversely proportionally to the size of the selected/relevant set rather than treating them equally.

There is a far simpler metric that avoids this problem. Simply use the total error: FN + FP (e.g. 5% of the image's pixels were miscategorized). In the case where one is more important than the other, a weighted average may be used: $c_0$FP + $c_1$FN.

• willem, i couldn't have asked for a better answer. thank you very much for taking the time. Apr 28, 2017 at 7:01
• I tried your total error approach and just wanted to add that it doesn't work well with constant imbalances between positives and negatives. Imagine an entire dataset of images where only one pixel makes up the ground truth segmentation. Neural networks might learn fairly quickly that an empty prediction is always 99.9% accurate using the total error. By going with IoU or DSC we pressure the network into finding a segmentation due to the same reasons you mentioned above. So, in the end it's very problem depend. Aug 10, 2017 at 8:43
• Can someone help me reconcile the following two statements?: 1: "That is to say that if classifier A is better than B under one metric, it is also better than classifier B under the other metric." and 2: "It may be the case then that the F metric favors classifier A while the IoU metric favors classifier B." Nov 12, 2017 at 19:58
• The former refers to a score of a single inference, and the latter refers to an average score over a set of inferences (e.g. a set of images). Nov 13, 2017 at 21:31

Yes, they indeed represent different things and have different meaning when looking at the formulas. However, when you use them as a evaluation measure to compare the performance of different model, you only need to choose one of them.

The reason can be explained by the following evidence:

First, let $$a = TP,\quad b=TP+FP+FN$$

Then, we have $$IoU = \frac{TP}{TP+FP+FN} = \frac{a}{b}$$ $$Dice = \frac{TP+TP}{TP+TP+FP+FN} = \frac{2a}{a+b}$$

Hence, $$Dice = \frac{\frac{2a}{b}}{\frac{a+b}{b}}= \frac{2 \cdot \frac{a}{b}}{\frac{a}{b}+1} = \frac{2 \cdot IoU}{IoU + 1}$$

Considering the line plot of $$y=2x/(x+1)$$ in the range of [0,1], we find out that Dice has a monotonic increasing relation to IoU. Then the following situation will not happen: $$Dice_1 < Dice_2$$ while $$IoU_1 > IoU_2$$ (the subscript represents different model). That is, Dice score is just similar representation of IoU under the numerical sense. It's enough to only using one of them for model comparison.

For Nico's answer above, I'm wondering shouldn't IoU be TP/(TP+FP+FN) instead of TP/(TP+FP+TN)? Also shouldn't the Dice score be (TP+TP)/(TP+TP+FP+FN)?

• Welcome to CV. This does not really answer the question. Please submit a new question (and make a reference to this question if needed) when you are in this situation. The site works better this way and you increase your chance to get an answer. Thank you. Nov 8, 2021 at 6:34
• This does not provide an answer to the question. Once you have sufficient reputation you will be able to comment on any post; instead, provide answers that don't require clarification from the asker. - From Review Nov 8, 2021 at 7:41

You can look at the ratio between DICE and IoU, it may be interesting:

$$IoU = \frac{|A \cap B|}{|A \cup B|}$$

$$DICE = \frac{2|A \cap B|}{|A| + |B|}$$

To compute the ratio, we would like to divide DICE by IoU, but need to be careful not to divide by zero, which will be the case when $$|A \cap B| = 0$$. In such case, by just looking at the metric definitions, we can notice that they both will be identically 0.

Recall that $$|A \cup B| = |A| + |B| - |A \cap B| \Longrightarrow |A \cup B| + |A \cap B| = |A| + |B|$$

So, suppose that $$|A \cap B| > 0$$, we get:

$$\frac{DICE}{IoU} = \frac{2|A \cap B| |A \cup B|}{(|A| + |B|)|A \cap B|} = \frac{2|A \cup B|}{|A| + |B|} = \frac{2|A \cup B|}{|A \cup B| + |A \cap B|}$$.

Now, we know that $$|A \cap B| \le |A \cup B|$$, which means that:

• When the $$A \cap B = \emptyset$$, both metrics are identical and equal to 0 and the ratio cannot be computed.
• When $$A = B \Longrightarrow |A \cap B| = |A \cup B|$$, the ratio will be 1, meaning that the two metrics are again identical, and both equal to 1.
• In all other cases, since $$|A \cap B| < |A \cup B|$$, the ratio will be in $$[1,2)$$, meaning that DICE will be up to 2x larger that IoU.

To conclude,

• DICE and IoU will be identical when $$A = B$$ or when $$A \cap B = \emptyset$$
• When the intersection rate is very small but not identically 0, DICE will tend to be up to 2x larger than IoU, but never identically 2x.
• The higher the intersection rate, the closer DICE and IoU will be, while DICE will remain somewhat larger than IoU, as long as the two sets aren't identical.

Here you can see the relationship between them visually. The union size is set to 1000 and the DICE/IoU ratio is plotted where the intersection size runs from 1 to 1000...

Here is the difference between them:

Here the two metrics are plotted individually, under the same conditions: