Andrej Karpathy in his blog post "A Recipe for Training Neural Networks" states that initialization is important for convergence. I get that but when he says:

init well. Initialize the final layer weights correctly. E.g. if you are regressing some values that have a mean of 50 then initialize the final bias to 50. If you have an imbalanced dataset of a ratio 1:10 of positives:negatives, set the bias on your logits such that your network predicts probability of 0.1 at initialization. Setting these correctly will speed up convergence and eliminate “hockey stick” loss curves where in the first few iteration your network is basically just learning the bias.

Why an incorrect initialization causes the network to learn the bias only during the first iterations? How is this related to bias?


1 Answer 1


The bias of a neural network is the response to zero input, which is very similar to the response of a randomly initialized network. So, intuitively, if you initialize your biases to suboptimal values, the optimization with respect to the loss can gain most by first adapting the bias and thus setting the proper range for the output.

Hence, setting the biases correctly according to your prior knowledge of the output makes for the weight being learned earlier on in the process. Note that another way to achieve the same behavior would be to normalize (zero mean, unit variance) the targets of the network.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.