I am trying to understand better how the image semantic/ instance segmentation work.

I understand going from the concept of the perceptron that Deep Neural Networks have one or both of the following:

  • a CNN for features leaning and outputs feature maps and/or
  • a FCN that flattens its inputed feature maps and may or may not be dense with final output layer having n nodes where n is the number of class that we wish to classify.

While training the network, we usually design a loss function that minimizes the error between the true outputs y_true (Ground Truth or Label), and the predicted output y_pred that the network has generated.

We can then update the weight with our obtained minima often via backpropagation.

1) How does this work when it comes to image segmentation networks?

2) How are the masks that come with the dataset used in getting the correct weights to output the correct predicted masks for each input image?

3) Is the loss function here minimizing the error in each pixel? that seems to be a lot of calculations.

4) if that is the case, how do we ensure that after the weights have been trained to properly predict the mask of a given image, those same weights will also perform well for a different image?

Thank you very much.


1 Answer 1


In Deep Learning only 3 things are required,

  1. Data
  2. Model
  3. Loss Function

For Image Segmentation we have input data and output data of same size for height and width. The output 3rd dimension varies depending on number of classes you have. for example input 3 channel RGB image and output mask of 5 classes have dimentions 3 x H x W to 5 x H x W, So we need a to have model whose input and outputs are matching dimensions so we usually use U-Net for such tasks And this is per pixel classification task. So any classification loss function can be used. Other things are same e.g. calculating Loss, BackPropagation, Optimizers, etc.

Therefore answers to your questions are, 1,2) Image segmentation is per pixel classification task where your output size is same as input size (approx.) You have labels for per pixel as a class index e.g. 1 is cat, 2 is dog, etc. For labels to visualize we usually color code them i.e. same class have same color. Per pixel loss is calculated and backpropagation is done.

3) If we talk about calculations then as compared to classification then lot of calculations are required at the second half of the network.

4) If you trained your model well then model learns the object's details so well that can recognize objects in an unseen image. For ensuring that we fed lot of unseen image to the model calculate accuracy, precision, recall using matrices such as Intersection Over Union threshold.

For more information you can view this excellent lecture - https://course.fast.ai/videos/?lesson=3

  • $\begingroup$ Thank you so much, if I understand correctly, let's say we are training on the handwritten MNIST data set for example for digit recognition. In this case, our input is a W x H image and our output is and 10 x 1 vector where 1 indicates the true value/ possible predicted value and 0 other. For Segmentation, however, our input is 3 x W x H, and our output is n x W x H for n class prediction. $\endgroup$
    – site
    Commented May 20, 2020 at 8:49
  • $\begingroup$ Yes! for more information you can tensorflow.org/tutorials/images/segmentation $\endgroup$ Commented May 20, 2020 at 8:54
  • $\begingroup$ for the 10 x 1 vector, our loss function will be applied to the sum of the error in each of the 10 classes for the MNIST data. For the segmentation problem, if your input image is 3 x W x H, and we want to predict 5 classes, does that mean that the loss function will be applied to the each pixel predicted for each class ie f [((W x H pixels)_pred_class1 - (W x H pixels)_true_class1)+ ((W x H pixels)_pred_class2-(W x H pixels)_true_class1)+ .... + ((W x H pixels)_pred_class5]- (W x H pixels)_true_class5)? $\endgroup$
    – site
    Commented May 20, 2020 at 8:59
  • $\begingroup$ Also, when we look at how Mask RCNN works, the paper said that it outputs a 28x28 mask for a given class. It was not clear to me how the final mask for a given class is created/predicted as the network outputs a 28 x 28 mask. I am assuming that after getting the 28 x 28 mask, they warp it back to the correct size in the original image and map it to the correct location in the W x H mask using the coordinates of the bounding box that was detected in the traditional RCNN $\endgroup$
    – site
    Commented May 20, 2020 at 9:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.