0
$\begingroup$

Say we are doing mini-batch gradient descent, the common way of doing so is by calculating the overall gradient of several samples and then update the parameter. Can we average the samples passed in first and then conducting the gradient update on this "averaged data"?

$\endgroup$
1
$\begingroup$

In general, no.

Consider a function $f[x]$ applied to some data $x_1,x_2,\ldots,x_n$, where $n$ is the size of the mini-batch.

Let $g[x]=f'[x]$ be the derivative of $f$. Then the average derivative over the data is $\langle{g}\rangle=\frac{1}{n}\sum_ig[x_i]$, while the derivative evaluated at the average data is $g[\langle{x}\rangle]$, where $\langle{x}\rangle=\frac{1}{n}\sum_ix_i$.

The two expressions are equal only if $g[x]=f'[x]$ is a linear map $$g[a_1x_1+a_2x_2]=a_1g[x_1]+a_2g[x_2]$$ This will only be true in the special case where $f[x]$ is a quadratic form $$f[x]=\tfrac{1}{2}x^TAx \implies \frac{\partial{f}}{\partial{x}}=Ax$$ for some matrix $A$ that does not depend on $x$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.