# Context

For simplicity let us pretend we are performing semantic segmentation on a series of one pixel high images of width w with three channels (r, g, b) with n label classes.

In other words, a single image might look like:

img = [
[r1, r2, ..., rw], # channel r
[g1, g2, ..., gw], # channel g
[b1, b2, ..., bw], # channel b
]


and have dimensions [3, w].

then for a given image with w=10 and n=3 its labels ground truth might be:

# ground "truth"
target = np.array([
#0     1     2     3     4     5     6     7     8     9      # position
[0,    1,    1,    1,    0,    0,    1,    1,    1,    1],    # class 1
[0,    0,    0,    0,    1,    1,    1,    1,    0,    0],    # class 2
[1,    0,    0,    0,    0,    0,    0,    0,    0,    0],    # class 3
])


and our model might predict as output:

# prediction
output = np.array([
#0     1     2     3     4     5     6     7     8     9      # position
[0.11, 0.71, 0.98, 0.95, 0.20, 0.15, 0.81, 0.82, 0.95, 0.86], # class 1
[0.13, 0.17, 0.05, 0.42, 0.92, 0.89, 0.93, 0.93, 0.67, 0.21], # class 2
[0.99, 0.33, 0.20, 0.12, 0.15, 0.15, 0.20, 0.01, 0.02, 0.13], # class 3
])


for further simplicity, let us transform our model's output by binarizing it with a cutoff of 0.9

# binary mask with cutoff 0.9
#0     1     2     3     4     5     6     7     8     9      # position
[0,    0,    1,    1,    0,    0,    0,    0,    1,    0],    # class 1
[0,    0,    0,    0,    1,    0,    1,    1,    0,    0],    # class 2
[1,    0,    0,    0,    0,    0,    0,    0,    0,    0],    # class 3
])


Then if we were to look at the "objects" of each class the bounding boxes (or in this case just boundaries i.e. [start, stop] pixels) our predicted objects from the binary mask "introduce" an object:

# "detected" objects
p_obj = [
[[2, 3], [8, 8]],  # class 1
[[4, 4], [6, 7]],  # class 2
[[0, 0]]           # class 3
]


compared to the objects of the ground truth:

# true objects
t_obj = [
[[1, 3], [6, 9]],  # class 1
[[4, 7]],          # class 2
[[0, 0]]           # class 3
]


# Question

If I wanted a metric to describe the accuracy of the boundaries, on average, per object, what would be the appropriate metric?

I understand IOU in the context of training a model which predicts bounding boxes, e.g. it is an object to object comparison, but what should one do when one object might be fragmented into several?

# Goal

I would like a metric that, per class, gives me something like this:

class 1: [-1, 2]  # bounding boxes for class one, on average start one
# pixel before they should and end two pixels after
# they should

class 2: [ 0, 3]  # bounding boxes for class two, on average start
# exactly where they should and end three pixels
# after they should

class 3: [ 3, -1] # bounding boxes for class three, on average start
# three pixels after where they begin and end one
# pixels too soon


but I am not sure how to best approach this when a single object is fragmented into several...