181 votes
Accepted

What are the advantages of ReLU over sigmoid function in deep neural networks?

Two additional major benefits of ReLUs are sparsity and a reduced likelihood of vanishing gradient. But first recall the definition of a ReLU is $h = \max(0, a)$ where $a = Wx + b$. One major benefit ...
user avatar
  • 2,627
93 votes

What are the advantages of ReLU over sigmoid function in deep neural networks?

Advantage: Sigmoid: not blowing up activation Relu : not vanishing gradient Relu : More computationally efficient to compute than Sigmoid like functions since Relu just needs to pick ...
user avatar
70 votes

What are the advantages of ReLU over sigmoid function in deep neural networks?

Just complementing the other answers: Vanishing Gradients The other answers are right to point out that the bigger the input (in absolute value) the smaller the gradient of the sigmoid function. ...
user avatar
60 votes

Is Wikipedia's page on the sigmoid function incorrect?

The unsatisfying answer is "It depends who you ask." "Sigmoid", if you break it into parts, just means "S-shaped". The logistic sigmoid function is so prevalent that ...
user avatar
37 votes
Accepted

Why is tanh almost always better than sigmoid as an activation function?

Yan LeCun and others argue in Efficient BackProp that Convergence is usually faster if the average of each input variable over the training set is close to zero. To see this, consider the extreme ...
user avatar
22 votes

Why is tanh almost always better than sigmoid as an activation function?

It's not that it is necessarily better than $\text{sigmoid}$. In other words, it's not the center of an activation fuction that makes it better. And the idea behind both functions is the same, and ...
user avatar
  • 371
15 votes

sklearn logistic regression converging to unexpected coefficient for a simple case

As Demetri suggested, we need to add penalty='none' for the code to give expected results. The revised code is as follows: ...
user avatar
  • 393
14 votes

Relu vs Sigmoid vs Softmax as hidden layer neurons

In addition to @Bhagyesh_Vikani: Relu behaves close to a linear unit Relu is like a switch for linearity. If you don't need it, you "switch" it off. If you need it, you "switch" it on. Thus, we get ...
user avatar
  • 6,800
14 votes

What are the advantages of ReLU over sigmoid function in deep neural networks?

An advantage to ReLU other than avoiding vanishing gradients problem is that it has much lower run time. max(0,a) runs much faster than any sigmoid function (logistic function for example = 1/(1+e^(-a)...
user avatar
  • 141
14 votes
Accepted

Finding the slope at different points in a sigmoid curve

Your question is very broad. There are many ways to do this, even without assuming a specific function. For the following I assume that you have a good reason to use the Gompertz model. First let's ...
user avatar
  • 5,808
12 votes
Accepted

Logistic function with a slope but no asymptotes?

You could just add a term to a logistic function: $$ f(x; a, b, c, d, e)=\frac{a}{1+b\exp(-cx)} + dx + e $$ The asymptotes will have slopes $d$. Here is an example with $a=10, b = 1, c = 2, d = \...
user avatar
  • 25.9k
11 votes
Accepted

When is logit function preferred over sigmoid?

It would not make sense to use the logit in place of the sigmoid in classification problems. The sigmoid (*) function is used because it maps the interval $[-\infty, \infty]$ monotonically onto $[0, ...
user avatar
11 votes

Logistic function with a slope but no asymptotes?

Initially I was thinking you did want the horizontal asymptotes at $0$ still; I moved my original answer to the end. If you instead want $\lim_{x\to\pm \infty} f(x) = \pm\infty$ then would something ...
user avatar
  • 18.6k
10 votes

What are the advantages of ReLU over sigmoid function in deep neural networks?

The main reason why ReLu is used is because it is simple, fast, and empirically it seems to work well. Empirically, early papers observed that training a deep network with ReLu tended to converge ...
user avatar
  • 5,953
10 votes
Accepted

sklearn logistic regression converging to unexpected coefficient for a simple case

I will add my own answer to this question in order to shine some light on why a penalty is added by default. I'm also posting for posterity as you are not the first person to get caught by this and ...
user avatar
10 votes

Is Wikipedia's page on the sigmoid function incorrect?

As Arya said, it depends who you ask, but this is not specific to Machine Learning, and even in Machine Learning the situation is not consistent (or not consistently bad). Bishop, for example, uses ...
user avatar
  • 6,283
10 votes

What is a sigmoid function and what does it give as output?

Your description is correct. The proper name of the function is logistic function, as "sigmoid" is ambiguous and may be applied to different S-shaped functions. It takes as input some value $...
user avatar
  • 113k
9 votes

Why do we use the natural exponential in logistic regression?

Because base $e$ is convenient, and it doesn't matter if you can freely scale your coefficient estimate. Would using a functional form of $\frac{a^\mathbf{x\cdot b}}{1 + a^\mathbf{x\cdot b} }$ change ...
user avatar
  • 20.8k
8 votes

Why do we use the natural exponential in logistic regression?

In binary regression, one can use any cdf to relate the probability $\mathbb{P}(Y=1|\mathbf{x})$ and $\mathbf{x}$ in a generalised linear way $$\mathbb{P}(Y=1|\mathbf{x})=\Phi(\mathbf{x}^\text{T}\beta)...
user avatar
  • 91.8k
8 votes

Do you do linear regression in logistic regression?

The simplest way of likening logistic regression to standard linear regression is using the latent variable interpretation. The logistic regression model can be described by considering the ...
user avatar
  • 94.4k
8 votes
Accepted

Why $1/(1+e^{-x}) = e^x/(1+e^x)$

It is easy. $ \dfrac{1}{1+ e^{-x}} = \dfrac{e^{x}}{1+e^{x}} $ Consider lhs $ \dfrac{1}{1+ \frac{1}{e^{x}}} $ which is equal to $ \dfrac{e^{x}}{e^{x}+ 1} $
user avatar
  • 437
7 votes

Relu vs Sigmoid vs Softmax as hidden layer neurons

Relu have its own pros and cons: Pros: 1. Does not saturate (in +ve region) 2. Computationally, it is very efficient 3. Generally models with relu neurons converge much faster than neurons ...
user avatar
7 votes

Logistic function with a slope but no asymptotes?

I will go ahead and turn the comment into an answer. I suggest $$ f(x) = \operatorname{sign}(x)\log{\left(1 + |x|\right)}, $$ which has slope tending towards zero, but is unbounded. edit by popular ...
user avatar
7 votes

Do you do linear regression in logistic regression?

No, it is not done like this. Quoting my other answer Logistic regression can be described as a linear combination $$ \eta = \beta_0 + \beta_1 X_1 + ... + \beta_k X_k $$ that is passed through the ...
user avatar
  • 113k
7 votes

Is Wikipedia's page on the sigmoid function incorrect?

I believe one more answer, specifically addressing your points as they currently stand (Revision 11) and comments is warranted. Is Wikipedia's page on the sigmoid function incorrect? No. In some ...
user avatar
  • 6,283
7 votes
Accepted

How does logistic growth rate coincide with the slope of the line in the exponential phase of the growth?

Let's do the calculations to see what the answers are. By changing the units of measurement of $x$ to the origin $x_0$ we may assume $x_0=0$ (to simplify the work and the notation) and--therefore--the ...
user avatar
  • 287k
6 votes
Accepted

Difference between logistic regression models for classification problems

They are equivalent, so both are correct. It can be proved simply by checking the probabilities for $Y= +/- 1$ In the first model: $$P(Y=1 |x,w) = \frac{1}{1+\exp(-w^Tx)} = \frac{\exp(w^Tx)}{1+\exp(...
user avatar
  • 1,026
6 votes
Accepted

How to model positive S-shaped-function?

The sigmoid, S-shaped or ogive curve shown in your plot is ubiquitous in nature. Geoffrey West's recent book Scale: The Universal Laws of Growth, Innovation, Sustainability, and the Pace of Life in ...
user avatar
  • 9,812
6 votes
Accepted

Looking for function to fit sigmoid-like curve

I think smoothing splines with small degrees of freedom would do the trick. Here's an example in R: The R code: ...
user avatar
  • 18.6k
6 votes

Looking for function to fit sigmoid-like curve

To fit a sigmoid-like function in a nonparametric way, we could use a monotone spline. This is implemented in the R package (all R packages here referenced are on CRAN) ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible