Questions tagged [relu]
The relu tag has no usage guidance.
13
questions
0
votes
0
answers
12
views
Activation function for performing counting, modified version of leaky ReLU
Is there any known activation function that is like a leaky ReLU but also has its growth capped at the top once it reaches one through linear growth? I mean something like this.
I need this for ...
1
vote
0
answers
26
views
When or why would we want to use one of the smooth approximation functions for the ReLU? [duplicate]
Learning about the ReLU, I keep finding variants or approximations that attempt to smooth out the function (eg. squarreplus).
Why (and when) is smoothing desirable?
- 129
2
votes
1
answer
36
views
ReLU Variance in Practice Disagrees with Theory
Let $X$ be a normally distributed random variable with $\mu=0$ and $\sigma=1$. Theory tells us that $\mathbb{V}[\text{ReLU}(X)] = \frac{1}{2}\mathbb{V}[X] = \frac{1}{2}$. Thus, we should have that $\...
- 123
0
votes
1
answer
23
views
How are groups created in maxout units when dividing the set of inputs 𝑧 into groups of 𝑘 values?
I don't get $G^(i)$the set of indices into the inputs for group $i$, $\{(i −1)k+ 1, \ldots , ik\}$ when creating a maxout units/function, these thing that outputs the maximum element of groups: $$g(z)...
1
vote
0
answers
31
views
Is ReLU activation function unsuitable for input layer if the input data has high inter-example correlation?
After making a neural network using ReLU as the activation function throughout, I had a look at the input layer activations and noticed that about 10% of the neurons are dead on initialization (never ...
- 375
0
votes
0
answers
19
views
Why do we choose activation functions (like the RELU et al) which strictly suppress negative values? [duplicate]
I understand the need to introduce non-linearity into a neural network, and the requirements for differentiability etc, but why do we focus on activation functions that essentially kill off negative ...
- 1,180
0
votes
0
answers
123
views
With 2 ReLU activated layers, if the 2nd layer has all weights initialized to < 0, the network is always stillborn?
I've built on my own neural network library with keras-like syntax. I noticed that when using 2 consecutive ReLU activated layers, and the 2nd of those layers has its weights initialized to negative ...
0
votes
1
answer
81
views
Autoencoder accuracy with standardized data
I want to make an autoencoder over the data that I originally standardized (that is, the data is now normally distributed ~ N(0,1)). The activation functions I use in the linear autoencoder is ReLu.
...
- 51
0
votes
1
answer
392
views
Derivate of Neural Network respect to input
I have a neural network like this
$x=\text{input}$
$z_1=W_{1x}\cdot x+b_1$
$h_1=\text{relu}(z_1)$
$z_2=W_2\cdot h_1+W_{2x}\cdot x+b_2$
$h_2=\text{relu}(z_2)$
$y=W_3\cdot h_2+W_{3x}\cdot x+b_3$
input ...
2
votes
0
answers
29
views
Can predicting negative regression values increase the chance of dying ReLUs?
Since ReLUs have zero gradient around negative values, we know that if a neuron outputs a negative value, the corresponding ReLU activation will cause it to die. Therefore, if I use a neural network ...
- 500
2
votes
0
answers
24
views
How is ReLU used in neural networks if it doesn't squish the weighted sum into the interval of (0, 1)? [duplicate]
So as far as I understand from watching 3Blue1Brown's video on neural networks, all neurons operate on numbers ranging from 0 to 1. Since a weighted sum goes larger than that, a sigmoid function is ...
- 21
0
votes
1
answer
31
views
If all computed features ( or components ) in Neural Network nodes are positive numbers , does using Relu meaningful?
I am trying to understand the following issue. The reason we use activation functions such as sigmoid,tanh or relu in neural networks is to obtain a nonlinear combination of input features ( x's). My ...
- 145
10
votes
5
answers
8k
views
Can a neural network work with negative and zero inputs?
As the title suggests, I have several features which have values of either -1, 0 or 1. If I feed this data into a neural network where I use ReLu as the activation ...
- 340