Linked Questions
11 questions linked to/from What are good initial weights in a neural network?
120
votes
3
answers
149k
views
tanh activation function vs sigmoid activation function
The tanh activation function is:
$$tanh \left( x \right) = 2 \cdot \sigma \left( 2 x \right) - 1$$
Where $\sigma(x)$, the sigmoid function, is defined as:
$$\sigma(x) = \frac{e^x}{1 + e^x}$$.
...
- 1,353
22
votes
0
answers
15k
views
When should I use the Normal distribution or the Uniform distribution when using Xavier initialization?
Xavier initialization seems to be used quite widely now to initialize connection weights in neural networks, especially deep ones (see What are good initial weights in a neural network?).
The ...
- 2,140
16
votes
3
answers
23k
views
MNIST digit recognition: what is the best we can get with a fully connected NN only? (no CNN)
To fully understand how it works internally, I'm re-writing a neural network from scratch in Python + numpy only. (As it's for learning purposes, performance is not an issue).
Before moving to ...
- 385
8
votes
2
answers
5k
views
What values should initial weights for a ReLU network be?
For a standard feed-forward Neural Network, what range should my initial weights fall under if I'm planning to use Rectified Linear Unit as an activation function? A mathematical justification for the ...
- 201
5
votes
4
answers
5k
views
Deep Neural Network weight initialization [duplicate]
Given difficult learning task (e.g high dimensionality, inherent data complexity) Deep Neural Networks become hard to train.
To ease many of the problems one might:
Normalize && handpick ...
- 518
5
votes
2
answers
914
views
Why does pre-training help avoid the vanishing gradient problem?
I read that a problem with the Classic approach to deep NN is the vanishing gradient, which is caused by the derivative of the logistic activation function - broadly speaking, the update flowing down ...
- 677
2
votes
1
answer
3k
views
Feed forward Neural Network and MSE issues
I've been implementing a Feed-forward Neural Network in C++ and CUDA. It is a basic Multi-layered Feed Forward ANN, using various activation functions (sigmoid bipolar, tanh, tanh scaled, and soft-...
- 123
1
vote
1
answer
122
views
Initialising the weights of a neural network in regards to the Var{y}
So for a neural network, I'm trying to find the best weight $a$ to form the weight interval $[-a, a]$, given the function for the output of a single layer neural network
$y = \sigma(Wx)$
where $x \...
- 11
0
votes
1
answer
85
views
Neural net: Cost goes down but performance on train isnt! [duplicate]
So, While building a simpel neural net (MLP) for recognizing digits, I ordered my function to print me both the mean cost overall off the train dataset and %currect answers, also over the train ...
- 223
0
votes
1
answer
561
views
Why my weight Initialization saturates the activation function?
I am trying to initialize the Hidden Layers of my Convolutional Neural Network with 7 hidden layers, all with ELU Activation with Sigmoid Activation at the end for binary classification.
For ...
- 201
0
votes
0
answers
3k
views
Choice of initial value for weights in Tensorflow MNIST Tutorial [duplicate]
I was tweaking the Tensorflow MNIST Tutorial, and I am not clear why the weights are initialized using tf.zeros. If I switch to using tf.random_normal, or tf.truncated_normal, I see a very significant ...
- 121