82
votes
What are the advantages of ReLU over sigmoid function in deep neural networks?
Just complementing the other answers:
Vanishing Gradients
The other answers are right to point out that the bigger the input (in absolute value) the smaller the gradient of the sigmoid function. ...
- 1,002
60
votes
Is Wikipedia's page on the sigmoid function incorrect?
The unsatisfying answer is "It depends who you ask." "Sigmoid", if you break it into parts, just means "S-shaped".
The logistic sigmoid function is so prevalent that ...
- 9,001
50
votes
tanh activation function vs sigmoid activation function
Thanks a lot @jpmuc ! Inspired by your answer, I calculated and plotted the derivative of the tanh function and the standard sigmoid function seperately. I'd like to share with you all. Here is what I ...
- 601
45
votes
Accepted
Why is tanh almost always better than sigmoid as an activation function?
Yan LeCun and others argue in Efficient BackProp that
Convergence is usually faster if the average of each input variable over the training set is close to zero. To see this, consider the extreme ...
- 1,611
23
votes
Why is tanh almost always better than sigmoid as an activation function?
It's not that it is necessarily better than $\text{sigmoid}$. In other words, it's not the center of an activation fuction that makes it better. And the idea behind both functions is the same, and ...
- 399
16
votes
What are the advantages of ReLU over sigmoid function in deep neural networks?
An advantage to ReLU other than avoiding vanishing gradients problem is that it has much lower run time. max(0,a) runs much faster than any sigmoid function (logistic function for example = 1/(1+e^(-a)...
- 161
16
votes
Accepted
When is logit function preferred over sigmoid?
It would not make sense to use the logit in place of the sigmoid in classification problems.
The sigmoid (*) function is used because it maps the interval $[-\infty, \infty]$ monotonically onto $[0, ...
- 36.3k
15
votes
sklearn logistic regression converging to unexpected coefficient for a simple case
As Demetri suggested, we need to add penalty='none' for the code to give expected results.
The revised code is as follows:
...
- 393
14
votes
Accepted
Finding the slope at different points in a sigmoid curve
Your question is very broad. There are many ways to do this, even without assuming a specific function. For the following I assume that you have a good reason to use the Gompertz model.
First let's ...
- 7,076
14
votes
Relu vs Sigmoid vs Softmax as hidden layer neurons
In addition to @Bhagyesh_Vikani:
Relu behaves close to a linear unit
Relu is like a switch for linearity. If you don't need it, you "switch" it off. If you need it, you "switch" it on. Thus, we get ...
- 7,351
13
votes
What are the advantages of ReLU over sigmoid function in deep neural networks?
The main reason why ReLu is used is because it is simple, fast, and empirically it seems to work well.
Empirically, early papers observed that training a deep network with ReLu tended to converge ...
- 6,738
12
votes
Logistic function with a slope but no asymptotes?
Initially I was thinking you did want the horizontal asymptotes at $0$ still; I moved my original answer to the end. If you instead want $\lim_{x\to\pm \infty} f(x) = \pm\infty$ then would something ...
- 20.8k
12
votes
Accepted
Logistic function with a slope but no asymptotes?
You could just add a term to a logistic function:
$$
f(x; a, b, c, d, e)=\frac{a}{1+b\exp(-cx)} + dx + e
$$
The asymptotes will have slopes $d$.
Here is an example with $a=10, b = 1, c = 2, d = \...
- 31.2k
12
votes
What is a sigmoid function and what does it give as output?
Your description is correct. The proper name of the function is logistic function, as "sigmoid" is ambiguous and may be applied to different S-shaped functions. It takes as input some value $...
- 141k
10
votes
Why do we use the natural exponential in logistic regression?
Because base $e$ is convenient, and it doesn't matter if you can freely scale your coefficient estimate.
Would using a functional form of $\frac{a^\mathbf{x\cdot b}}{1 + a^\mathbf{x\cdot b} }$ change ...
- 23k
10
votes
Accepted
sklearn logistic regression converging to unexpected coefficient for a simple case
I will add my own answer to this question in order to shine some light on why a penalty is added by default. I'm also posting for posterity as you are not the first person to get caught by this and ...
- 39.6k
10
votes
Is Wikipedia's page on the sigmoid function incorrect?
As Arya said, it depends who you ask, but this is not specific to Machine Learning, and even in Machine Learning the situation is not consistent (or not consistently bad). Bishop, for example, uses ...
- 9,668
10
votes
Accepted
How does logistic growth rate coincide with the slope of the line in the exponential phase of the growth?
Let's do the calculations to see what the answers are.
By changing the units of measurement of $x$ to the origin $x_0$ we may assume $x_0=0$ (to simplify the work and the notation) and--therefore--the ...
- 334k
9
votes
Why do we use the natural exponential in logistic regression?
In binary regression, one can use any cdf to relate the probability $\mathbb{P}(Y=1|\mathbf{x})$ and $\mathbf{x}$ in a generalised linear way
$$\mathbb{P}(Y=1|\mathbf{x})=\Phi(\mathbf{x}^\text{T}\beta)...
- 108k
9
votes
Why is tanh almost always better than sigmoid as an activation function?
It all essentially depends on the derivatives of the activation function, the main problem with the sigmoid function is that the max value of its derivative is 0.25, this means that the update of the ...
9
votes
What are the advantages of ReLU over sigmoid function in deep neural networks?
Main benefit is that the derivative of ReLu is either 0 or 1, so multiplying by it won't cause weights that are further away from the end result of the loss function to suffer from the vanishing ...
- 3,670
8
votes
Do you do linear regression in logistic regression?
The simplest way of likening logistic regression to standard linear regression is using the latent variable interpretation. The logistic regression model can be described by considering the ...
- 133k
8
votes
Accepted
Why $1/(1+e^{-x}) = e^x/(1+e^x)$
It is easy.
$ \dfrac{1}{1+ e^{-x}} = \dfrac{e^{x}}{1+e^{x}} $
Consider lhs
$ \dfrac{1}{1+ \frac{1}{e^{x}}} $
which is equal to
$ \dfrac{e^{x}}{e^{x}+ 1} $
- 387
7
votes
Relu vs Sigmoid vs Softmax as hidden layer neurons
Relu have its own pros and cons:
Pros:
1. Does not saturate (in +ve region)
2. Computationally, it is very efficient
3. Generally models with relu neurons converge much faster than neurons ...
- 118
7
votes
Why do we use the natural exponential in logistic regression?
For a Bernoulli likelihood, the variance is a function of the mean such that:
$$\text{var}(Y) = E(Y)(1-E(Y))$$
It turns out that a sigmoid function, also called the "inverse link" (for a logistic ...
- 64.8k
7
votes
Logistic function with a slope but no asymptotes?
I will go ahead and turn the comment into an answer. I suggest
$$
f(x) = \operatorname{sign}(x)\log{\left(1 + |x|\right)},
$$
which has slope tending towards zero, but is unbounded.
edit by popular ...
- 563
7
votes
Do you do linear regression in logistic regression?
No, it is not done like this. Quoting my other answer
Logistic regression can be described as a linear combination
$$ \eta = \beta_0 + \beta_1 X_1 + ... + \beta_k X_k $$
that is passed through the ...
- 141k
7
votes
Is Wikipedia's page on the sigmoid function incorrect?
I believe one more answer, specifically addressing your points as they currently stand (Revision 11) and comments is warranted.
Is Wikipedia's page on the sigmoid function incorrect?
No. In some ...
- 9,668
7
votes
Accepted
Where does the Logistic Distribution get its name?
The cumulative distribution function of the logistic distribution is the logistic function
$$
F(x) = \frac{1}{1+e^{-(x-\mu)/s}}
$$
For an explanation of where the logistic function got its name, check ...
- 141k
7
votes
Example where the initial random state of a logistic regression matters?
Both linear and logistic regression are convex optimisation problems and have same behaviour.
If the 2nd derivative of the objective at the minimum is positive definite, then the minimum is unique, ...
- 7,262
Only top scored, non community-wiki answers of a minimum length are eligible