I have been thinking of proving the following:

Prove that the gradient calculated on a random subset of a training set on average is proportional to the true gradient.

However, proving is not my strongest competence and I am trying to get better. Intuitively it makes sense. How would I go on proving this?

  • $\begingroup$ What have you tried? $\endgroup$
    – Tim
    Sep 28, 2019 at 15:47
  • $\begingroup$ I was thinking of looking at the subset S of training set T, $S \in T$, then find the gradient of the subset by some loss function $\nabla L(w_S)$ where $L(w_S) = \sum_{i \in S}||y(x_i)-t_i||^2_2$ and then show $E(\nabla L(w_S)) \propto \nabla L(w)$ where $\nabla L(w)$ is the true gradient of the training set. Something like that. $\endgroup$
    – Chris
    Sep 28, 2019 at 17:40


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.