regarding the output format for semantic segmentation

Question

While reading the semantic segmentation papers as well as their corresponding implementations, I found that some approaches use softmax while others use sigmoid for the pixel-level labeling.

For instance, with respect to u-net paper, the output is a feature map with two channels.

I have seen some implementations using softmax over these two channel outputs. I am not sure whether my following understanding is correct or not?

For illustration purposes, the masked portion belongs to class 1 and the other part belongs to class 2. I only assume two class: masked or non-masked.

I use xy to represent the output map with shape (1, image_row,image_col,2). Then,xy[1,0,0,0] will represent the probability of pixel at (0,0) belonging to class 1 while xy[1,0,0,1] will represent the probability of pixel (0,0) belonging to class 2. In other words, xy[1,row,col,0]+xy[1,row,col,1]=1

Is my understanding correct?

Answer 1

Semantic segmentation is just extended classification, where you perform classification of each pixel into the n_classes.

Let's say your input is an RGB image with size (cols,rows,3), you pass a batch of such images sized (batch_size, cols, rows, 3) to the CNN.

After performing computations in the network graph, you will end up with a choice to have the last convolutional layer to have n_outputs.

Binary segmentation (pixelwise yes / no )

---

Then you have can have n_outputs = 1 and the output shape will be (batch_size, cols, rows, 1). You later take the sigmoid activation use binary_crossentropy loss. Note that this only works for binary segmentation.

MultiClass segmentation (pixelwise probability vector)

---

Then you have n_outputs = n_classes and the output shape will be (batch_size, cols, rows, n_classes). Now comes the tricky part. You need to apply softmax to each pixel probability vector which generally involves permuting dimensions depending on the deep learning framework you are using. In this case you use categorical_crossentropy as it

In Keras you can

final_conv_out = Convolution2D(n_classes, 1, 1)(conv9)

x = Reshape((n_classes, rows*cols))(final_conv_out)
x = Permute((2,1))(x)

# seg is a pixelwise probability vector sized (batch_size, rows*cols, n_classes)
seg = Activation("softmax")(x)