2
$\begingroup$

By the law of large numbers, given a continuous random vector $\mathbf{x}$, then:

$$ \mathbb{E}[\mathbf{x}] \approx \frac{1}{N} \sum_{i=1}^{N} \mathbf{x}_i $$

Where $\mathbf{x}_1,\mathbf{x}_2,...,\mathbf{x}_N$ are sampled from $p(\mathbf{x})$. By the law of the unconscious statistician:

$$ \mathbb{E}[f(\mathbf{x})] \approx \frac{1}{N} \sum_{i=1}^{N} f(\mathbf{x}_i) $$

Now let $\theta$ be a vector of non-random parameters, such that $g(\mathbf{x};\theta)$ is some function of the random vector $\mathbf{x}$ and the parameters in $\theta$. Can I then approximate the following expectation:

$$ \mathbb{E}\left[\frac{\partial g(\mathbf{x};\theta)}{\partial \theta}\right] $$

Like this?

$$ \mathbb{E}\left[\frac{\partial g(\mathbf{x};\theta)}{\partial \theta}\right] \approx \frac{1}{N} \sum_{i=1}^{N} \frac{\partial g(\mathbf{x}_i;\theta)}{\partial \theta} $$

If I can, then is it accurate to say that:

$$ \mathbb{E}\left[\frac{\partial g(\mathbf{x};\theta)}{\partial \theta}\right] \approx \frac{\partial}{\partial \theta}\left(\frac{1}{N} \sum_{i=1}^{N} g(\mathbf{x}_i;\theta)\right) $$

In other words, the sample mean of a gradient is equal to the gradient of the sample mean?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

The derivative and expectation have the associative property (you can exchange the order) by Leibniz integral rule (computing the expectation is just some sort of integration)

$$\frac{\partial}{\partial \theta} \int_{a}^{b} f(x,\theta) d\,x = \int_a^b \frac{\partial}{\partial \theta} f(x,\theta) d\,x$$

(Note that not every operation has this property and you can have $f(E(X)) \neq E(f(X))$)

Improve this answer
$\endgroup$
2
  • $\begingroup$ +1. So this holds even before the approximation, i.e. $E[\triangledown_\theta(g(\cdot))]=\triangledown_\theta(E[g(\cdot)])$, where $\triangledown$ is the gradient. So then the approximation inside $\triangledown()$ follows directly from the LLN, assuming $\textrm x$ is $iid$. I think of $g(x,\theta)$ as a random variable itself (since it depends on a rv), hence for the LLN, LOTUS is not even needed here. Also, an example for $f(E[\cdot]) \ne E[f(\cdot)]$ (which holds most of the time, equality is an exception) is with $f(x)=x^2$. $\endgroup$
    – PaulG
    Commented Dec 4, 2020 at 12:15
  • $\begingroup$ Ah, now I see that you were thinking about the continuous mapping theorem. That is indeed not necessary for this case. But that theorem works for every function. $\endgroup$
    – Sextus Empiricus
    Commented Dec 4, 2020 at 12:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.