2
$\begingroup$

I know that when dealing with artificial neural networks, RELU yields a value based on the weighted sum of the inputs plus a bias term. However, this logic does not seem to apply to convolutional neural networks.

Looking at the ResNet architecture, the outputs of the convolutional neural nets (what I believe to be feature maps), is added to the input x, and then RELU is applied onto it. What exactly does the RELU function do in this case? Do the convolution layers output feature maps, or something else?

Improve this question
$\endgroup$
2

2 Answers 2

Reset to default
2
$\begingroup$

It's no different from using ReLU in any other context. In a feedforward network, a standard usage is $\text{ReLU}(Ax+b)$. In a CNN, a standard usage is $\text{ReLU}(\text{convolution}(y))$: all you do is apply the convolution operation and then the ReLU operation.

It's not clear what you mean by "feature maps." The learned parameters of a convolution layer are sometimes called "feature maps" or "kernels". However, these are not the same thing as the output of a convolution operation applied to an image. The output of a convolution layer is just an image with a filter passed over it; this is what the ReLU operation is applied to.

Improve this answer
$\endgroup$
0
0
$\begingroup$

A Relu activated neuron is not a linear function which simply returns the weighted sum plus a bias term. Such a neuron returns 0 until the weighted sum of inputs surpasses a certain value. enter image description here

Improve this answer
$\endgroup$