# Tag Info

Accepted

### What are the advantages of ReLU over sigmoid function in deep neural networks?

Two additional major benefits of ReLUs are sparsity and a reduced likelihood of vanishing gradient. But first recall the definition of a ReLU is $h = \max(0, a)$ where $a = Wx + b$. One major benefit ...
• 2,627

### What are the advantages of ReLU over sigmoid function in deep neural networks?

Advantage: Sigmoid: not blowing up activation Relu : not vanishing gradient Relu : More computationally efficient to compute than Sigmoid like functions since Relu just needs to pick ...

### What are the advantages of ReLU over sigmoid function in deep neural networks?

Just complementing the other answers: Vanishing Gradients The other answers are right to point out that the bigger the input (in absolute value) the smaller the gradient of the sigmoid function. ...

### Is Wikipedia's page on the sigmoid function incorrect?

The unsatisfying answer is "It depends who you ask." "Sigmoid", if you break it into parts, just means "S-shaped". The logistic sigmoid function is so prevalent that ...
• 6,797
Accepted

### Why is tanh almost always better than sigmoid as an activation function?

Yan LeCun and others argue in Efficient BackProp that Convergence is usually faster if the average of each input variable over the training set is close to zero. To see this, consider the extreme ...

### Why is tanh almost always better than sigmoid as an activation function?

It's not that it is necessarily better than $\text{sigmoid}$. In other words, it's not the center of an activation fuction that makes it better. And the idea behind both functions is the same, and ...
• 371

### sklearn logistic regression converging to unexpected coefficient for a simple case

As Demetri suggested, we need to add penalty='none' for the code to give expected results. The revised code is as follows: ...
• 393

### Relu vs Sigmoid vs Softmax as hidden layer neurons

In addition to @Bhagyesh_Vikani: Relu behaves close to a linear unit Relu is like a switch for linearity. If you don't need it, you "switch" it off. If you need it, you "switch" it on. Thus, we get ...
• 6,800

### What are the advantages of ReLU over sigmoid function in deep neural networks?

An advantage to ReLU other than avoiding vanishing gradients problem is that it has much lower run time. max(0,a) runs much faster than any sigmoid function (logistic function for example = 1/(1+e^(-a)...
• 141
Accepted

### Finding the slope at different points in a sigmoid curve

Your question is very broad. There are many ways to do this, even without assuming a specific function. For the following I assume that you have a good reason to use the Gompertz model. First let's ...
• 5,808
Accepted

• 33.6k

### Logistic function with a slope but no asymptotes?

Initially I was thinking you did want the horizontal asymptotes at $0$ still; I moved my original answer to the end. If you instead want $\lim_{x\to\pm \infty} f(x) = \pm\infty$ then would something ...
• 18.6k

### What are the advantages of ReLU over sigmoid function in deep neural networks?

The main reason why ReLu is used is because it is simple, fast, and empirically it seems to work well. Empirically, early papers observed that training a deep network with ReLu tended to converge ...
• 5,953
Accepted

### sklearn logistic regression converging to unexpected coefficient for a simple case

I will add my own answer to this question in order to shine some light on why a penalty is added by default. I'm also posting for posterity as you are not the first person to get caught by this and ...
• 25.8k

### Is Wikipedia's page on the sigmoid function incorrect?

As Arya said, it depends who you ask, but this is not specific to Machine Learning, and even in Machine Learning the situation is not consistent (or not consistently bad). Bishop, for example, uses ...
• 6,283

• 1,026
Accepted

### How to model positive S-shaped-function?

The sigmoid, S-shaped or ogive curve shown in your plot is ubiquitous in nature. Geoffrey West's recent book Scale: The Universal Laws of Growth, Innovation, Sustainability, and the Pace of Life in ...
• 9,812
Accepted

### Looking for function to fit sigmoid-like curve

I think smoothing splines with small degrees of freedom would do the trick. Here's an example in R: The R code: ...
• 18.6k