F1/Dice-Score vs IoU I was confused about the differences between the F1 score, Dice score and IoU (intersection over union). By now I found out that F1 and Dice mean the same thing (right?) and IoU has a very similar formula to the other two.


*

*F1 / Dice: $$\frac{2TP}{2TP+FP+FN}$$

*IoU / Jaccard: $$\frac{TP}{TP+FP+FN}$$


Are there any practical differences or other things worth noting except that F1 weights the true-positives higher? Is there a situation where I'd use one but not the other?
 A: Yes, they indeed represent different things and have different meaning when looking at the formulas.
However, when you use them as a evaluation measure to compare the performance of different model, you only need to choose one of them.
The reason can be explained by the following evidence:
First, let $$ a = TP,\quad b=TP+FP+TN $$
Then, we have
$$ IoU = \frac{TP}{TP+FP+TN} = \frac{a}{b} $$
$$ Dice = \frac{TP+TP}{TP+TP+FP+TN} = \frac{2a}{a+b} $$
Hence,
$$ Dice = \frac{\frac{2a}{b}}{\frac{a+b}{b}}= \frac{2 \cdot \frac{a}{b}}{\frac{a}{b}+1} = \frac{2 \cdot IoU}{IoU + 1} $$
Considering the line plot of $y=2x/(x+1)$ in the range of [0,1], we find out that Dice has a monotonic increasing relation to IoU. Then the following situation will not happen: $Dice_1 < Dice_2$ while $IoU_1 > IoU_2$ (the subscript represents different model).
That is, Dice score is just similar representation of IoU under the numerical sense. It's enough to only using one of them for model comparison.
A: You're on the right track.
So a few things right off the bat. From the definition of the two metrics, we have that IoU and F score are always within a factor of 2 of each other:
$$ F/2 \leq IoU \leq F $$
and also that they meet at the extremes of one and zero under the conditions that you would expect (perfect match and completely disjoint).
Note also that the ratio between them can be related explicitly to the IoU:
$$ IoU/F = 1/2 + IoU/2 $$
so that the ratio approaches 1/2 as both metrics approach zero.
But there's a stronger statement that can be made for the typical application of classification a la machine learning. For any fixed "ground truth", the two metrics are always positively correlated. That is to say that if classifier A is better than B under one metric, it is also better than classifier B under the other metric.
It is tempting then to conclude that the two metrics are functionally equivalent so the choice between them is arbitrary, but not so fast! The problem comes when taking the average score over a set of inferences. Then the difference emerges when quantifying how much worse classifier B is than A for any given case.
In general, the IoU metric tends to penalize single instances of bad classification more than the F score quantitatively even when they can both agree that this one instance is bad. Similarly to how L2 can penalize the largest mistakes more than L1, the IoU metric tends to have a "squaring" effect on the errors relative to the F score. So the F score tends to measure something closer to average performance, while the IoU score measures something closer to the worst case performance.
Suppose for example that the vast majority of the inferences are moderately better with classifier A than B, but some of them of them are significantly worse using classifier A. It may be the case then that the F metric favors classifier A while the IoU metric favors classifier B.
To be sure, both of these metrics are much more alike than they are different. But both of them suffer from another disadvantage from the standpoint of taking averages of these scores over many inferences: they both overstate the importance of sets with little-to-no actual ground truth positive sets. In the common example of image segmentation, if an image only has a single pixel of some detectable class, and the classifier detects that pixel and one other pixel, its F score is a lowly 2/3 and the IoU is even worse at 1/2. Trivial mistakes like these can seriously dominate the average score taken over a set of images. In short, it weights each pixel error inversely proportionally to the size of the selected/relevant set rather than treating them equally.
There is a far simpler metric that avoids this problem. Simply use the total error: FN + FP (e.g. 5% of the image's pixels were miscategorized). In the case where one is more important than the other, a weighted average may be used: $c_0$FP + $c_1$FN.
A: For Nico's answer above, I'm wondering shouldn't IoU be TP/(TP+FP+FN) instead of TP/(TP+FP+TN)? Also shouldn't the Dice score be (TP+TP)/(TP+TP+FP+FN)?
