I am trying to implement "from scratch" SGD and Mini Batch Gradient Descent in Matlab.

I have to minimize a function like $f(x)= \sum f_i (X_i, y_i)$ where $(X_i, y_i)$ is a data point (features and class label one hot encoded respectively)

However, I am a bit confused about how to choose the learning rate. When implementing gradient descent, I used backtracking line search in order to find a learning rate that always guarantees a descent direction.

However, in SGD (or even in MBGD), one choose randomly some $k$ data points ($k=1$ for SGD). Let's call them $x_{i_{l_1}},..., x_{i_{l_k}}$. Let's call those indexes $l_1,...l_k$ the subset $L$. The gradient is now only taken on $\sum_{i\in L} f_i (X_i, y_i)$. The direction given by the gradient of this "partial" function is not necessarily a descent direction, thus how to find a "good" learning rate ?

I guess that backtracking taking into account the overall objective function is not that useful, as there is actually no guarantee to obtain a learning rate that could guarantee the decrease of the original $f(x)$... So I was thinking about making backtracking but only on the batch of points $L$ used for computing the gradient (i.e. find a good learning rate that minimizes $\sum_{i\in L} f_i (X_i, y_i)$ rather than the overall function $f(x)$).

Does it make sense ?

Hoping to get an answer soon !


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.