Linked Questions

295
votes
10answers
283k views

Difference between logit and probit models

What is the difference between Logit and Probit model? I'm more interested here in knowing when to use logistic regression, and when to use Probit. If there is any literature which defines it using ...
127
votes
5answers
56k views

What should I do when my neural network doesn't learn?

I'm training a neural network but the training loss doesn't decrease. How can I fix this? I'm not asking about overfitting or regularization. I'm asking about how to solve the problem where my ...
79
votes
2answers
87k views

tanh activation function vs sigmoid activation function

The tanh activation function is: $$tanh \left( x \right) = 2 \cdot \sigma \left( 2 x \right) - 1$$ Where $\sigma(x)$, the sigmoid function, is defined as: $$\sigma(x) = \frac{e^x}{1 + e^x}$$. ...
38
votes
4answers
27k views

Why sigmoid function instead of anything else?

Why is the de-facto standard sigmoid function, $\frac{1}{1+e^{-x}}$, so popular in (non-deep) neural-networks and logistic regression? Why don't we use many of the other derivable functions, with ...
17
votes
2answers
3k views

Backpropagation algorithm

I got a slight confusion on the backpropagation algorithm used in multilayer perceptron (MLP). The error is adjusted by the cost function. In backpropagation, we are trying to adjust the weight of ...
24
votes
1answer
10k views

Why are non zero-centered activation functions a problem in backpropagation?

I read here the following: Sigmoid outputs are not zero-centered. This is undesirable since neurons in later layers of processing in a Neural Network (more on this soon) would be receiving ...
6
votes
1answer
1k views

Exact definition of Maxout

I've been trying to figure out what exactly it meant by the "Maxout" activation function in neural networks. There is this question, this paper, and even in the Deep Learning book by Bengio et al., ...
4
votes
1answer
784 views

ReLUs and Gradient Descent for Deep Neural Nets

I understand that ReLUs are used in Neural Nets generally instead of sigmoid activation functions for the hidden layer. However, many commonly used ReLUs are not differentiable at zero. Gradient ...
2
votes
1answer
1k views

activation functions of elman recurrent neural network

From the Book"fundamentals of neural network", the input layer of a feedforward neural network has linear activation function. Elman recurrent NN is the same as a feedforward except that it has ...
5
votes
2answers
711 views

Sigmoid type functions for logistic regression

I am trying to find sigmoid function alternatives for logistic regression. I am curious that if I can replace sigmoid function by any cumulative distribution function, and what will be the best?
3
votes
1answer
165 views

The reason that the larger gradient flowing through an ReLU neuron can cause it to die

In this link about different neuron types, there is an introduction on the disadvantage of ReLU, (-) Unfortunately, ReLU units can be fragile during training and can "die". For example, a large ...
2
votes
1answer
268 views

ANN Handling Nonlinear Data

One of the main reasons that ANN performs better is due to its influential feature in handling nonlinear data (Wu et al, 2008). Can anyone explain to me what is this meaning?
2
votes
1answer
135 views

Transfer Function on Neural Network

Do different Transfer Function produce different prediction in neural network model? How do we know which transfer function suitable for the data we used?