1
$\begingroup$

I am working on project where main task is semantic segmentation of land cover and another objects in Sentinel 2 multi-spectral images. Currently I posses dataset of 50 000 single-labeled images trimmed from Sentinel 2 tiles using OpenStreetMap polygons i.e. every image contains one class e.g. water-body. There is 7 classes in those dataset. Meant for image classification task using convolutional neural networks.

Can I use those single-labeled images for training fully convolutional neural network and then use it for semantic segmentation of (larger) images containing more classes e.g. 7 ?

Or need I to create new dataset of images, each containing those 7 classes ?

Any suggestions or ideas are appreciated !

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

In general, if you don't have a dataset that contains those 7 classes, you wouldn't be able to get a neural network which performs your task of semantic segmentation.

My understanding of your question is the following:

  • you own a dataset, each image labelled with a single class label
  • you are able to train a CNN or similar structure with these images
  • you want that neural network to perform semantic segmentation for you at inference

This task seems to be a "zero-shot learning task" for image segmentation trying to leverage related image filters for the task.

You could try training a neural network for your classification task and then replacing the last layers with randomly initialised layers to provide the segmentation. However, that would hardly provide any useful performance for you.

Most likely, an already trained image segmentation model would be the best way to proceed in your case. The main problem is that you have in my opinion is that you won't be able to get away with segmenting at least part of the masks if you have completely novel segmentation classes, otherwise you won't be able to claim what method works for your dataset in a quantitative way.

If you really have to do the problem zero shot, I would try to look at zero shot segmentation papers. For example, this work might be interesting for your case. In their work, they exploit relations between semantic word embeddings to alleviate annotation needs where generalisation to new classes are required.

Improve this answer
$\endgroup$
2
  • $\begingroup$ You understand it correctly, each image in dataset is labelled with single class label ( because it contains only one class per image). I am not sure if it is zero-shot task because I won't use that to segment previously unseen classes. In other words, network would see every time only image with one class at a time not image containing all the classes at a time. It seems to be not a usual way of training net for multi-class segmetation but I hoped that there is a way to deal with that and save some time for generating new dataset of images each containing all the classes at a time. $\endgroup$
    – Many
    Commented Dec 31, 2020 at 23:05
  • $\begingroup$ It sounds like you want to get from classification to segmentation without having segmented maps for the training, which is an unusual thing as far as I know. If you are interested in what parts of the image the neural net uses to classify into class A, you could look into explainable ML techniques too, like GradCAM, LRP and attention. With GradCAM, I think you might get something similar to segmentation if you run it for each class. $\endgroup$
    – boomkin
    Commented Dec 31, 2020 at 23:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.