This article says

At times, you might see that your loss drops steeply after a short period of training, before stabilizing. This is a strong indication that your initial weight allocation is inadequate.

I am wondering why steep loss reduction is an indication of inadequate initial weight allocation. Thanks!


Well, it's actually quite the opposite.

Given a classification network with N outputs and a balanced dataset your initial loss ALWAYS will be around $-\ln(\frac{1}{N})$. It means nothing more than that you will guess the output correctly 1 out of N times on average. It makes sense, since you are initializing the weights randomly from some given distributions.

But contrary to what's written in the article, a steep loss reduction means you gave the network a good starting point which it can utilise to quickly descend towards a minimum.

Think of it that way: your goal is to climb Mt. Everest. You can start right at the base of the hill and quickly ascend to the top. Or you can start somewhere far off, probably by some other mountain, wonder around and slowly ascend. Which do you think is the better starting position?

(and keep in mind you can only look under your feet, since this is how gradient descent works :D)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.