I'm doing stochastic gradient descent on a non-convex optimization problem. Gradient corresponds to an intractable expectation which I approximate via Monte Carlo averaging. I'm trying to infer the values of a vector (corresponding to my parameters). One of the dimensions of the vector has much higher variance compared to the rest. I'm not converging to a good local minimum. Are there any efficient variance reducing gradient descent methods I could use instead? I'm aware of SVRG however not familiar at the moment with the state of the art in this area.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.