Is stochastic gradient descent biased?

Question

In the paper Mutual Information Neural Estimation, the authors derive the following gradient for the network $$ \nabla_\theta\mathcal V(\theta)=\mathbb E\left[\nabla_\theta T_\theta\right]-{\mathbb E\left[e^{T_\theta}\nabla_\theta T_\theta\right]\over \mathbb E\left[e^{T_\theta}\right]} $$ and say it's biased because, in the second term, the expectations are over the sample of a minibatch.

However, based on my knowledge, SGD is high variance indeed but it shouldn't introduce any bias. Is my previous understanding wrong?

Furthermore, the authors say that by replacing the expectation in the denominator with an exponential moving average, the bias can be reduced. Why does that make sense?

Answer 1

For a typical loss function $L = E_{x_i \sim \text{D}}[f(x_i)]$ and true gradient $\nabla L = E[\nabla f(x_i)]$, the expectation of the SGD gradient is $E[\nabla f(x')]$ where $x'$ is the datapoint in our batch of size this is 1. This is clearly unbiased.

The loss function in the paper takes the form $L = \log E[e^{f(x)}]$ and has the gradient $$\nabla L = \frac{1}{E[e^{f(x)}]} E[\nabla e^{f(x)}] = \frac{E[\nabla f(x) e^{f(x)}]}{E[e^{f(x)}]} $$

Note that the SGD gradient $\frac{\nabla f(x') e^{f(x')}}{e^{f(x')}} = \nabla f(x')$ is biased.

However, what we we only "did SGD" for the numerator and computed the exact expectation for the denominator? This pseudo-SGD gradient $\frac{\nabla f(x') e^{f(x')}}{E[e^{f(x)}]}$ is indeed unbiased.

Although it is too expensive to recompute the denominator at every SGD step, if we assume that the parameters of $f$ do not change too rapidly (and therefore $f(x)$ also does not change rapidly), one way to estimate the denominator is to use an exponentially weighted moving average. This would lead us to a relatively unbiased estimate.