2
$\begingroup$

We are already aware that in case the data is quite bulky, mini-batch gradient descent based approaches may be applied. These approaches load a mini-batch of data, compute the loss on this batch, and optimize the loss function by updating function parameters by a set of gradient update rules.

Now, my question is that can this concept of mini-batches be applied to subgradient rules as well? All we will be doing is that we will be changing the update rule to include sub-gradients instead of gradients computed on that mini-batch.

I am using the method of sub-gradients since my loss function is convex but not differentiable.

$\endgroup$
1
$\begingroup$

Not sure what kind of models you are using exactly, but in deep learning it often happens that the gradients are not defined everywhere (e.g. ReLU activation function).

Also, Adagrad, a well-known optimisation strategy in deep learning, explicitly takes subgradients into account.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.