I've recently read that SGD (Stochastic Gradient Descent) is one of the most popular techniques for training Machine Learning algorithms, including DNNs (deep neural networks).

However, my understanding is that when one performs SGD, the optimization procedure picks at random a training example and then updates the weights of the DNN according to the gradient corresponding to that training example. Are there some theoretical results that show that the resulting values of the weights at the end, or at least the performance on the test set is independent of this random order in which SGD goes through the data? More precisely, if we start with two copies of the network and training data with the same (suitably initialized) weights, do we expect to reach the same performance if we train the two copies independently using SGD?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.