Questions tagged [relu]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers

Activation function for performing counting, modified version of leaky ReLU

Is there any known activation function that is like a leaky ReLU but also has its growth capped at the top once it reaches one through linear growth? I mean something like this. I need this for ...
Jose Manuel de Frutos's user avatar
1 vote
0 answers

When or why would we want to use one of the smooth approximation functions for the ReLU? [duplicate]

Learning about the ReLU, I keep finding variants or approximations that attempt to smooth out the function (eg. squarreplus). Why (and when) is smoothing desirable?
blueberryfields's user avatar
2 votes
1 answer

ReLU Variance in Practice Disagrees with Theory

Let $X$ be a normally distributed random variable with $\mu=0$ and $\sigma=1$. Theory tells us that $\mathbb{V}[\text{ReLU}(X)] = \frac{1}{2}\mathbb{V}[X] = \frac{1}{2}$. Thus, we should have that $\...
krc's user avatar
  • 123
0 votes
1 answer

How are groups created in maxout units when dividing the set of inputs 𝑧 into groups of 𝑘 values?

I don't get $G^(i)$the set of indices into the inputs for group $i$, $\{(i −1)k+ 1, \ldots , ik\}$ when creating a maxout units/function, these thing that outputs the maximum element of groups: $$g(z)...
Revolucion for Monica's user avatar
1 vote
0 answers

Is ReLU activation function unsuitable for input layer if the input data has high inter-example correlation?

After making a neural network using ReLU as the activation function throughout, I had a look at the input layer activations and noticed that about 10% of the neurons are dead on initialization (never ...
Museful's user avatar
  • 375
0 votes
0 answers

Why do we choose activation functions (like the RELU et al) which strictly suppress negative values? [duplicate]

I understand the need to introduce non-linearity into a neural network, and the requirements for differentiability etc, but why do we focus on activation functions that essentially kill off negative ...
Vishal's user avatar
  • 1,180
0 votes
0 answers

With 2 ReLU activated layers, if the 2nd layer has all weights initialized to < 0, the network is always stillborn?

I've built on my own neural network library with keras-like syntax. I noticed that when using 2 consecutive ReLU activated layers, and the 2nd of those layers has its weights initialized to negative ...
Tim de Jong's user avatar
0 votes
1 answer

Autoencoder accuracy with standardized data

I want to make an autoencoder over the data that I originally standardized (that is, the data is now normally distributed ~ N(0,1)). The activation functions I use in the linear autoencoder is ReLu. ...
josf's user avatar
  • 51
0 votes
1 answer

Derivate of Neural Network respect to input

I have a neural network like this $x=\text{input}$ $z_1=W_{1x}\cdot x+b_1$ $h_1=\text{relu}(z_1)$ $z_2=W_2\cdot h_1+W_{2x}\cdot x+b_2$ $h_2=\text{relu}(z_2)$ $y=W_3\cdot h_2+W_{3x}\cdot x+b_3$ input ...
Gaweiliex's user avatar
2 votes
0 answers

Can predicting negative regression values increase the chance of dying ReLUs?

Since ReLUs have zero gradient around negative values, we know that if a neuron outputs a negative value, the corresponding ReLU activation will cause it to die. Therefore, if I use a neural network ...
desert_ranger's user avatar
2 votes
0 answers

How is ReLU used in neural networks if it doesn't squish the weighted sum into the interval of (0, 1)? [duplicate]

So as far as I understand from watching 3Blue1Brown's video on neural networks, all neurons operate on numbers ranging from 0 to 1. Since a weighted sum goes larger than that, a sigmoid function is ...
Yeepsta's user avatar
  • 21
0 votes
1 answer

If all computed features ( or components ) in Neural Network nodes are positive numbers , does using Relu meaningful?

I am trying to understand the following issue. The reason we use activation functions such as sigmoid,tanh or relu in neural networks is to obtain a nonlinear combination of input features ( x's). My ...
levitatmas's user avatar
10 votes
5 answers

Can a neural network work with negative and zero inputs?

As the title suggests, I have several features which have values of either -1, 0 or 1. If I feed this data into a neural network where I use ReLu as the activation ...
spectre's user avatar
  • 340