I'm playing with different learning rates and batch sizes for a neural network. What I want to understand is a more technical definition of what learning rate is.
I understand that it is, in some way, a "step size" for the adjustments made to the network parameters based on the data it sees. However, I'd like to understand to a bit more depth.
If the learning rate is 0.001, what exactly does that mean? And is this value "applied" over batches, or epochs?
Thank you!
Gradient descent updates the model parameters $\theta$ using the rule
$$ \theta^{(t+1)} = \theta^{(t)}-\eta \nabla \mathcal{L}(f(X,\theta^{(t)}),y) $$ where $\eta$ is the learning rate, $\mathcal L$ is the loss function, $f$ is the model, $X$ stores the features and $y$ stores the lables.
Viewed in this way, the learning rate is used to rescale the gradient $\nabla \mathcal L$. This is the sense in which it is a "step size": if we draw a ray from $\theta^{(t)}$ in the direction of $\nabla \mathcal L$, the step size tells us where on that line $\theta^{(t+1)}$ is.
The model parameters are updated whenever you update them. Usually, this is once per batch, but some more complex models can do more complex things like store some (but not all) updates in a buffer and apply them at a later time (such as once per epoch).
