Is there an official procedure to compute mIoU (mean intersection over union)? Although it sounds silly, I'm not finding an official source to compute mean intersection over union (mIoU).
I'm realizing a semantic segmentation task, and I want to compute the mIoU over a dataset. My doubt is, should I compute the mIoU of each image and average the results in the end, or should I build a giant confusion matrix of all image results and compute the mIoU from there?
APPENDIX:
In case, anyone finds it later, it really seems to have multiple implementations that will differ on results for mIoU. On this issue is pointed out some divergences. It's also noted differences on the implementation of fastai and Cityscape.
 A: In general, when a two step procedure loses information, the best order of procedural steps is that which performs the least information reduction first. Indeed, the difference can be quite striking.
Having said that much, there are exceptions, but they tend to be trivial and apply to procedures that not different and also obey associative and/or commutative laws. For example,
$$(1+2+3)+(4+5+6)=(1+2)+(3+4+5+6)$$
because addition obeys the associative law. Notice that when we add numbers we have lost information, for example, given the number 6 alone, we do not know how many of which numbers were added to obtain that number. And both steps lose information. However, taking a median would not follow that law as
$$\text{median}\big[\big\{\text{median}\{1,2,3\},\text{median}\{4,5,6\}\big\}\big]=\frac{7}{2}\;\;,$$
whereas
$$\text{median}\big[\big\{\text{median}\{1,2\},\text{median}\{3,4,5,6\}\big\}\big]=3\;\;.$$
However, this is still too simple for a good example as
$$\text{median}\big[\big\{\text{median}\{1,2\},\text{median}\{3,4,5,6\}\big\}\big]=3=\text{median}\big[\big\{\text{median}\{3,4,5,6\},\text{median}\{1,2\}\big\}\big],$$
as taking a median obeys the commutative law. So, let us give an example for which it does make a big difference, with the downside being that you would have to try it yourself to see the improvement, which although quite striking visually, is tedious to prove.
That is, suppose we wish to add images to improve their resolution starting with a time series of images. What should one do first? Take two 64X64 pixel images and add them then expand the resulting image by interpolation to be 128X128. Or, should we expand each 64X64 image by interpolation to be 128X128 and then add those?
It turns out that greater image resolution results from the second procedure than the first, and the same is most likely true for the OP's question. Notice that we do not lose information by interpolative expansion of an image, but we lose information by image addition, so we do the expansion first before the addition to get the better result.
