3
$\begingroup$

I am trying to train a fully convolutional neural network for 3D medical image segmentation, I have started from the architecture of this paper with the differences being that I have images of varying sizes so I train the network one image at a time (no batching) and I use relus instead of prelus as the non-linearities.

The problem I am having is that the outputs of the model before the softmax/sigmoid are too large (around 1e32 each logit) and when calculating the cross entropy loss the calculation blows up and returns infinity or nan.

At first I thought this might be due to exploding gradients so I tried gradient clipping and the problem remained. After this I just took the outputs and divided them by a large number (1e32) and I started to get real values for the loss function.

My question is, what it the correct (certainly more elegant way) of achieving reasonable values for the logits , perhaps some sort of local normalisation at the end of each convolution layer?

Improve this question
$\endgroup$
3
  • $\begingroup$ What package are you using to construct the model? Also, how are you initializing your weights? $\endgroup$
    – Alex R.
    Commented Jul 12, 2017 at 20:15
  • $\begingroup$ I am using tensorflow 1.2 and initialising the weights with truncated_normal. I want to try the xavier initialiser but am still trying to figure out how it translates to 3D convolutions (tensorflow does have it for 2D convolutions). Also, all my biases start at the value of 1.0. $\endgroup$
    – Miguel
    Commented Jul 13, 2017 at 8:30
  • $\begingroup$ P.S I have made a post regarding Xavier initialisation for 3D convolutions. $\endgroup$
    – Miguel
    Commented Jul 13, 2017 at 9:03

1 Answer 1

Reset to default
3
$\begingroup$

Try either removing some layers or reducing the learning rate. If explosion happens before calculating the first or second loss, reducing the LR won't help.

I had the same problem and now I'm stuck with LR=0.001. Tell me if you found something better, so I can try it too.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I did find a solution afterwards, initialising the weights to smaller values. Before I sampled the initial weights from uniform distribution with zero mean and unit variance. Now I sample them from a distribution with zero mean and variance calculated with the Xavier Glorot method. I can use whatever learning rate I want, it doesn't explode. $\endgroup$
    – Miguel
    Commented Aug 5, 2017 at 17:06
  • $\begingroup$ reducing the learning rate worked for me $\endgroup$
    – liang
    Commented Apr 16, 2018 at 6:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.