I found in https://dl.acm.org/doi/abs/10.1145/3320060 (section 3) this graph that illustrates influence of batch size. enter image description here

Below there is an explanation:

We can show the existence of region C by combining SGD with the descent lemma for a function f with L-Lipschitz gradient: $$ \mathbb{E}_z[f(w^{(t+1)}] \leq f(w^{(t)}) - \eta_t \Vert \nabla f(w^{(t)}) \Vert^2 + \eta_t^2 \frac{L}{2} \mathbb{E}_z [\Vert \nabla f(w^{(t)}) \Vert^2] $$ where $z∼D$ and $∇f_z$ is the stochastic subgradient for $z$. This indicates that a large minibatch(with adjusted learning rate) can increase the convergence rate (negative term), but along with it the gradient variance and learning rate, which causes the last term to hinder convergence

Which I don't understand because in mini-batch SGD we take batch to estimate a gradient instead of calculating it for the whole set. So increasing a batch size should inherently improve quality of the model when using the same number of iterations, and the loss of accuracy in large batch size training comes from decreased number of iterations and is epoch-wise.

Another question is why some of terms in the equation are expectation values and some not.

Who is right here? Do you know any formal explanation for region C?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.