I am new to deep learning and have already got my hands dirty with running deep network on MNIST and CIFAR datasets.

Is there a way to directly use deep learning for semantic segmentation of image?

Suppose, the training images consist of RGB images and images of their corresponding groundtruth (i.e. pixel level annotation). If I want the network to directly classify a pixel as one of the N classes (say), what should be the network architecture ?

  • 2
    $\begingroup$ Read Semantic Segmentation section of Awesome Deep Vision repo. $\endgroup$
    – yobibyte
    Commented Jul 4, 2016 at 10:34
  • $\begingroup$ When I think about this link I wonder about sub-pixel classification. What is your interpolating function? Each pixel is a time and space average of incident light from (typically 3) spectral window(s). It approximates a system where the physics are nearly continuously varying, but the sampling isn't. (If we can abandon the regular grid, and we can use more physically valid interpolating functions than cosine, then we might do better there). What can you share about the physics that drive pixel values? $\endgroup$ Commented Dec 24, 2017 at 14:15
  • $\begingroup$ Maybe you can have a look at "U-nets", a comparably simple, yet powerful architecture for segmentation. Input and outputs are images and the net connects them by a U-shaped sequence of (up-)convolutions. You will find many such nets on kaggle or github, e.g. made with keras. $\endgroup$
    – Michael M
    Commented Sep 2, 2019 at 15:17

1 Answer 1


While this approach looses one of the main advantages of DCNN's; it can be done utilizing sequence classifiers, see Kera's Sequence tutorial or Jason Brownlee's http://machinelearningmastery.com/tutorial-first-neural-network-python-keras/

note: you'll likely have just as much accuracy as if you did a Bayesian classifier, which is going to be easier to visualize and troubleshoot. A note that you want to ensure the classes are separable and not bi-modal; so you do want to plot your training space to ensure you have enough classes. If one class is bi-modal, just figure out how to break it into two classes.

Also note that the old-style Neural nets typically never out-perform a Baye's classifier. It's only been the recent advances in DNN that do.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.