Conceptual question on Image semantic/ instance segmentation networks

Question

I am trying to understand better how the image semantic/ instance segmentation work.

I understand going from the concept of the perceptron that Deep Neural Networks have one or both of the following:

• a CNN for features leaning and outputs feature maps and/or
• a FCN that flattens its inputed feature maps and may or may not be dense with final output layer having n nodes where n is the number of class that we wish to classify.

While training the network, we usually design a loss function that minimizes the error between the true outputs y_true (Ground Truth or Label), and the predicted output y_pred that the network has generated.

We can then update the weight with our obtained minima often via backpropagation.

1) How does this work when it comes to image segmentation networks?

2) How are the masks that come with the dataset used in getting the correct weights to output the correct predicted masks for each input image?

3) Is the loss function here minimizing the error in each pixel? that seems to be a lot of calculations.

4) if that is the case, how do we ensure that after the weights have been trained to properly predict the mask of a given image, those same weights will also perform well for a different image?

Thank you very much.

Answer 1

In Deep Learning only 3 things are required,

1. Data
2. Model
3. Loss Function

For Image Segmentation we have input data and output data of same size for height and width. The output 3rd dimension varies depending on number of classes you have. for example input 3 channel RGB image and output mask of 5 classes have dimentions 3 x H x W to 5 x H x W, So we need a to have model whose input and outputs are matching dimensions so we usually use U-Net for such tasks And this is per pixel classification task. So any classification loss function can be used. Other things are same e.g. calculating Loss, BackPropagation, Optimizers, etc.

Therefore answers to your questions are, 1,2) Image segmentation is per pixel classification task where your output size is same as input size (approx.) You have labels for per pixel as a class index e.g. 1 is cat, 2 is dog, etc. For labels to visualize we usually color code them i.e. same class have same color. Per pixel loss is calculated and backpropagation is done.

3) If we talk about calculations then as compared to classification then lot of calculations are required at the second half of the network.

4) If you trained your model well then model learns the object's details so well that can recognize objects in an unseen image. For ensuring that we fed lot of unseen image to the model calculate accuracy, precision, recall using matrices such as Intersection Over Union threshold.

For more information you can view this excellent lecture - https://course.fast.ai/videos/?lesson=3