5
$\begingroup$

From a blog post from Andrej Karpathy on training neural networks:

Initialize the final layer weights correctly. E.g. if you are regressing some values that have a mean of 50 then initialize the final bias to 50. If you have an imbalanced dataset of a ratio 1:10 of positives:negatives, set the bias on your logits such that your network predicts probability of 0.1 at initialization.

I don't understand the bolded sentence at all. What does predict 0.1 mean? Is it that all the outputs should be 0.1? How would one go about doing that?

$\endgroup$

1 Answer 1

4
$\begingroup$

He is referring to a problem where you have a one output, a sigmoid neuron. When you initialize the weights of the network you can set bias to approximately $-2.3$.

Why? The last layers bias is going to be dependant only on the statistics of the data, since it doesnt have any connection to the input. Meaning that if you have an imbalanced dataset, where you have 10 times as much negative examples as positive, and the network would consist only of this bias, then it would output on average the number 1/11. Output of 1/11 corresponds to input to sigmoid layer equal to $-log(10)$, which is $-2.3$

And as a comment, the unbolded sentence is bullshit. It's much better to normalize the targets in the data. Don't take someone's word for granted just because they are a big name in the field.

$\endgroup$
1
  • 1
    $\begingroup$ Yup, a bias of 50 would not do anything to put the spread of the data (signified by the standard deviation) on the same scale. $\endgroup$
    – grofte
    Jul 6, 2020 at 15:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.