At first, this question is less about programming itself but about some logic behind the CNN architecture. I do understand how every layer works but my only question is: Does is make sense to separate the ReLU and Convolution-Layer? I mean, can a ConvLayer exist and work and update its weights by using backpropagation without having a ReLU behind it?

I thought so. This is why I created the following independent layers:

  1. ConvLayer
  2. ReLU
  3. Fully Connected
  4. Pooling
  5. Transformation (transform the 3D output into one dimension) for ConvLayer -> Fully Connected.

I am thinking about merging Layer 1 and 2 into one. What should I go for?


It makes sense. ReLU activation is not always following convolution layer. For example, when you're using residual networks, you may want to have ReLU before convolution, not after. The reason is that there are good results obtained by making residual step (addition) on bare convolution output (as described in http://florianmuellerklein.github.io/wRN_vs_pRN/).

However, in most cases, schema will look like you described (Conv + ReLU) and there is nothing wrong with merging code for these two.

  • $\begingroup$ Thank you for your help :) Just one more tiny question... Are the weights in the convolution layer updated like this: weight += -eta * error_signal * previousLayer.output where the weight is the one that connects the previousLayer.output with the neuron with the given error_signal? $\endgroup$ – Luecx May 23 '17 at 13:50
  • $\begingroup$ It depends on how do you optimize a cost function. In general it looks like you wrote, but sometimes people add weight decay, momentum... $\endgroup$ – zlenyk May 23 '17 at 18:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.