Can we average the data first before updating its gradient while doing mini-batch gradient descent?

Question

Say we are doing mini-batch gradient descent, the common way of doing so is by calculating the overall gradient of several samples and then update the parameter. Can we average the samples passed in first and then conducting the gradient update on this "averaged data"?

Answer 1

In general, no.

Consider a function $f[x]$ applied to some data $x_1,x_2,\ldots,x_n$, where $n$ is the size of the mini-batch.

Let $g[x]=f'[x]$ be the derivative of $f$. Then the average derivative over the data is $\langle{g}\rangle=\frac{1}{n}\sum_ig[x_i]$, while the derivative evaluated at the average data is $g[\langle{x}\rangle]$, where $\langle{x}\rangle=\frac{1}{n}\sum_ix_i$.

The two expressions are equal only if $g[x]=f'[x]$ is a linear map $$g[a_1x_1+a_2x_2]=a_1g[x_1]+a_2g[x_2]$$ This will only be true in the special case where $f[x]$ is a quadratic form $$f[x]=\tfrac{1}{2}x^TAx \implies \frac{\partial{f}}{\partial{x}}=Ax$$ for some matrix $A$ that does not depend on $x$.