0
$\begingroup$

This question already has an answer here:

I know that when dealing with artificial neural networks, RELU yields a value based on the weighted sum of the inputs plus a bias term. However, this logic does not seem to apply to convolutional neural networks.

Looking at the ResNet architecture, the outputs of the convolutional neural nets (what I believe to be feature maps), is added to the input x, and then RELU is applied onto it. What exactly does the RELU function do in this case? Do the convolution layers output feature maps, or something else?

$\endgroup$

marked as duplicate by Sycorax, mdewey, kjetil b halvorsen, gunes, Siong Thye Goh Apr 26 at 14:20

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

migrated from softwareengineering.stackexchange.com Apr 25 at 14:59

This question came from our site for professionals, academics, and students working within the systems development life cycle.

1
$\begingroup$

It's no different from using ReLU in any other context. In a feedforward network, a standard usage is $\text{ReLU}(Ax+b)$. In a CNN, a standard usage is $\text{ReLU}(\text{convolution}(y))$: all you do is apply the convolution operation and then the ReLU operation.

It's not clear what you mean by "feature maps." The learned parameters of a convolution layer are sometimes called "feature maps" or "kernels". However, these are not the same thing as the output of a convolution operation applied to an image. The output of a convolution layer is just an image with a filter passed over it; this is what the ReLU operation is applied to.

$\endgroup$
0
$\begingroup$

A Relu activated neuron is not a linear function which simply returns the weighted sum plus a bias term. Such a neuron returns 0 until the weighted sum of inputs surpasses a certain value. enter image description here

$\endgroup$