279k views

### Difference between logit and probit models

What is the difference between Logit and Probit model? I'm more interested here in knowing when to use logistic regression, and when to use Probit. If there is any literature which defines it using ...
37k views

### What should I do when my neural network doesn't learn?

I'm training a neural network but the training loss doesn't decrease. How can I fix this? I'm not asking about overfitting or regularization. I'm asking about how to solve the problem where my ...
82k views

### tanh activation function vs sigmoid activation function

The tanh activation function is: $$tanh \left( x \right) = 2 \cdot \sigma \left( 2 x \right) - 1$$ Where $\sigma(x)$, the sigmoid function, is defined as: $$\sigma(x) = \frac{e^x}{1 + e^x}$$. ...
25k views

### Why sigmoid function instead of anything else?

Why is the de-facto standard sigmoid function, $\frac{1}{1+e^{-x}}$, so popular in (non-deep) neural-networks and logistic regression? Why don't we use many of the other derivable functions, with ...
3k views

### Backpropagation algorithm

I got a slight confusion on the backpropagation algorithm used in multilayer perceptron (MLP). The error is adjusted by the cost function. In backpropagation, we are trying to adjust the weight of ...
9k views

### Why are non zero-centered activation functions a problem in backpropagation?

I read here the following: Sigmoid outputs are not zero-centered. This is undesirable since neurons in later layers of processing in a Neural Network (more on this soon) would be receiving ...
1k views

### Exact definition of Maxout

I've been trying to figure out what exactly it meant by the "Maxout" activation function in neural networks. There is this question, this paper, and even in the Deep Learning book by Bengio et al., ...
758 views

### ReLUs and Gradient Descent for Deep Neural Nets

I understand that ReLUs are used in Neural Nets generally instead of sigmoid activation functions for the hidden layer. However, many commonly used ReLUs are not differentiable at zero. Gradient ...
1k views

### activation functions of elman recurrent neural network

From the Book"fundamentals of neural network", the input layer of a feedforward neural network has linear activation function. Elman recurrent NN is the same as a feedforward except that it has ...
684 views

### Sigmoid type functions for logistic regression

I am trying to find sigmoid function alternatives for logistic regression. I am curious that if I can replace sigmoid function by any cumulative distribution function, and what will be the best?
138 views

### The reason that the larger gradient flowing through an ReLU neuron can cause it to die

In this link about different neuron types, there is an introduction on the disadvantage of ReLU, (-) Unfortunately, ReLU units can be fragile during training and can "die". For example, a large ...