Can we use the semantic segmented images directly to perform image classification using CNN model?

Updating the question: I am trying to classify images as below:
a. Input : Images taken from camera fitted on a vehicle.
b. Output : Check if there is an vehicle in front of my vehicle (from where image is taken)

I am currently using the RGB images to perform classification. However in this mis-classification would happen if there is an building (or any other obstacle other than vehicle) in the image.

Could you please clarify if the below approach would improve the accuracy in prediction
Since the semantic segmented images are colored according to the type, will it help in improving the model?


2 Answers 2


Semantic segmentation of an image (at least in context of CNN based methods) can simply be viewed as a pixel-wise classification problem, that is to say that you assign a class to every pixel of the image. So in your case if you are able to successfully perform a semantic segmentation of your camera feed then the problem of classification of cars/no cars is trivial or rather non-existent since you were able to solve a more difficult problem of classifying every pixel. Just check for cars in your pixel-wise classification output.

However, if your objective is only to identify whether or not there is a car in frame then honestly semantic segmentation is a overkill and I don't see any reason why you can't train a near perfect binary classifier for cars/no cars problem which would be robust to various background artifacts(buildings etc.) even with limited data. Try transfer learning on ILSVRC pre-trained resnet 34 or smaller variants. Fine tune the final few layers first and then when you hit saturation fine tune the entire network with these updated weights. You can also try cyclical learning rates (https://arxiv.org/abs/1506.01186).

  • $\begingroup$ Thanks for the input. It was really helpful. I have trained binary classification model using Transfer learning (ResNet50, VggNet), however the model is not robust because of below reasons: 1. Color of the obstacle (Car) can be different. 2. Shadow effect 3. Background effect (Building, Road boundary etc) $\endgroup$
    – deepguy
    Commented Mar 15, 2019 at 4:05

Yes, having ground truth segmentation would make classification tasks quite easy, perhaps even trivial.

A small nitpick is that a semantic segmentation map is not an image "colored according to type". The colored image is merely a visualization of the $K$ channel segmentation map, which when $K > 3$, cannot be directly visualized.

  • $\begingroup$ I am bit confused in the statement The colored image is merely a visualization of the 𝐾 channel segmentation map, which when 𝐾>3, cannot be directly visualized. $\endgroup$
    – deepguy
    Commented Mar 15, 2019 at 4:09
  • $\begingroup$ @KK2491 well a segmentation map is a $H$ by $W$ by $K$ array. A computer screen with only 3 channels cannot display this easily, so we assign each segmentation class to an arbitrary RGB value. Therefore there is an important difference between the visualization of a segmentation map (what you refer to as the "segmented image") and the segmentation map itself. It would be pretty weird to feed the visualization and not the map into a network. $\endgroup$
    – shimao
    Commented Mar 15, 2019 at 4:14
  • $\begingroup$ Thanks for the explanation. Now I understood. I was thinking to consider only roads by extracting the ROI from segmented images (visualization of segmentation map). I thought that this will reduce the effect of background on model. $\endgroup$
    – deepguy
    Commented Mar 15, 2019 at 4:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.