I watched this (with time marker to important moment https://youtu.be/TCH_1BHY58I?t=3040) video of Andrej Karpathy explaining some language model, and in ~50:40, he explains how to pick good initial learning rate. He does 1000 consecutive training steps while gradualy increasing learning rate. Then he plots LR versus loss (or in second version the exponent of LR versus loss). He concludes, that lr<0.1 is too low (why tho?), and lr>0.1 makes the training explode (this part is straightforward). I dont undersand why lower rate on this chart is proven to be too small. This (allegedly too small) learning rate is actually quickly lowering the loss. Can someone explain this?

  • $\begingroup$ It feels like perhaps he's forgotten that he's not re-initializing the network between changing learning rates, and treating the loss function as that for a fresh fit? There are several comments on the video asking the same thing. $\endgroup$ Commented Mar 15, 2023 at 16:52


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.