7
$\begingroup$

The neural network known as a "U-Net" ( Ronneberger, Fischer, and Brox 2015) was a prominent technique in Kaggle's recent Ultrasound Nerve Segmentation contest, where high scores were awarded to algorithms that created pixel masks with a high degree of overlap with the hand drawn regions.

Nerve with brachial plexus outline (Photo from Christopher Hefele)

If one proceeds to classify every pixel (perhaps from a down-sampled image), there must be many ways to incorporate the prior knowledge that neighboring pixels will tend to have the same class, and furthermore that all the positive classifications must reside in a single spatial region. However, I can't figure out how these U-Nets are doing it. They do classify every pixel, albeit by way of a maze of convolutional and pooling operators: The U-Net

There are separation borders involved, but the paper notes that they are "computed using morphological operations" which I take to mean entirely separate from the U-Net itself. Those borders are only used to modify the weights so that more emphasis is placed on the pixels at the border. They do not appear to fundamentally change the classification task.

Separation border

In classifying every pixel, how does this deep convolutional neural network, called a "U-Net," incorporate the prior knowledge that the predicted region will be a single spatial region?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

It incorporates "prior knowledge" by training the network over a training dataset which will update the weights of the convolution filters. This is how most neural networks are trained with standard backprop. Where the loss to be backproped is based on the segmenation loss in this case.

Here's a link to better show a deconvolution visualization viz. It doesn't show how it is trained because that is the same as how regular convolution is trained and there are other resources for that such as here backprop.

Improve this answer
$\endgroup$
5
  • $\begingroup$ So think about the structure of LSTMs, for instance. The architecture itself allows for stability over time via a "cell state." I didn't see anything like that for U-nets in the spatial realm. However, since asking this question, I learned a bit more. Now I think pixels near each other tend to have the same predicted class because the inputs are almost the same, due to the filters and the upconvolution operations. $\endgroup$
    – Ben Ogorek
    Aug 2, 2017 at 2:27
  • $\begingroup$ What you stated is incorrect. The architecture of an LSTM does not inherently allow for stability (even over time). Instead what the LSTM does is perform a nonlinear combination of a hidden state (previous inputs) with its current input. The nonlinear combination need not be stable at all. $\endgroup$
    – Steven
    Aug 2, 2017 at 17:15
  • $\begingroup$ This is more similar to image classification. How does VGG, or Resnet for example classify an image as a cat or dog etc. It builds up some nonlinear representations of the pixels that it can then use to classify the image. In this case the U-Net architecture builds up nonlinear combinations of the pixels in larger and larger spatial resolution by downsampling then it will perform upsampling but learned upsampling that will prioritize some features over others in the original image. You are training the weights that perform both the downsampling and upsampling to better segment an image. $\endgroup$
    – Steven
    Aug 2, 2017 at 17:19
  • $\begingroup$ Your last two sentences are more of what I was hoping to see in the original answer. There's not a lot of material in general on up-sampling (up-convolution?) and the types of features that can be learned. Can you expand on that in your original answer? $\endgroup$
    – Ben Ogorek
    Aug 3, 2017 at 13:21
  • $\begingroup$ Really it is the same as convolution. How are the filters learned in any of the processes are by back propagation. I've included a link to the another post that better highlights what upconvolution is doing. The grey block of squares are a filter that is learned and applied to the padded blue input. Let me know if this helps or there's still confusion. $\endgroup$
    – Steven
    Aug 10, 2017 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.