can we perform sub-gradient updates in mini-batches

Question

We are already aware that in case the data is quite bulky, mini-batch gradient descent based approaches may be applied. These approaches load a mini-batch of data, compute the loss on this batch, and optimize the loss function by updating function parameters by a set of gradient update rules.

Now, my question is that can this concept of mini-batches be applied to subgradient rules as well? All we will be doing is that we will be changing the update rule to include sub-gradients instead of gradients computed on that mini-batch.

I am using the method of sub-gradients since my loss function is convex but not differentiable.

Answer 1

Not sure what kind of models you are using exactly, but in deep learning it often happens that the gradients are not defined everywhere (e.g. ReLU activation function).

Also, Adagrad, a well-known optimisation strategy in deep learning, explicitly takes subgradients into account.