I was reading this post which enlightened me about the technicalities of the reparametrisation trick, but I only get the intuition of this equivalent transform and I'm not sure why it is true: $$𝐸_𝑞[x^2]=𝐸_𝑝[(𝜃+𝜖)^2]$$

The intuition is clear - we have introduced another random variable which produces our previous random variable but our $ \epsilon $ is sampled from a new random variable, which has different distribution.

The definitions are: $$ q_{\theta}(x) = \mathcal{N}(\theta,1) $$ $$ x = \theta + \epsilon $$ $$ \epsilon \sim \mathcal{N}(0,1) $$

My attempt: I think p in this context means $ p(\epsilon) = \mathcal{N}(0,1) $

So then the exercise is to prove:

$$ E_q [ x^2] = \int_{X} x^2 q_{\theta}(x) dx = \int_{\eta} (\theta + \epsilon)^2 p(\epsilon) d\epsilon $$

I assume it is possible then to argue that for a given $ x_0 = \theta + \epsilon_0 $ the equation will only agree if we make a shift, but for me this is still too intuitive. Could we express this more rigorously? (feel free to throw Measure Theory at me if needed)

  • $\begingroup$ $\epsilon$ is the random variable that follows a normal distirbution with mean $0$ and variance $1$. What do you mean by $p(\epsilon)$? $\endgroup$ – nbro Jun 13 at 15:32
  • 1
    $\begingroup$ There's no exercise to be done, the equality holds above basically by definition since this is just a change of variables from x to $\epsilon$ en.wikipedia.org/wiki/… $\endgroup$ – aleshing Jun 13 at 15:51

This is merely a change of variable. The expectation operator uses the density of the random variable(s) inside the expression. However, if you just look from a calculus perspective: $$x=\theta+\epsilon\rightarrow dx=d\epsilon, \ \ \epsilon=x-\theta$$ $$q_\theta(x)=\frac{1}{\sqrt{2\pi}}\exp\left(\frac{-(x-\theta)^2}{2}\right)=\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{\epsilon^2}{2}\right)=p(\epsilon)$$ And, the limits of $x$, i.e. $(-\infty,\infty)$ are shifted by $\theta$, but, since infinity, they're the same. Or, you could leave them as in your notation, $X\rightarrow\eta$. Then, it follows that $E_q[x^2]=E_p[(x+\theta)^2]$

  • $\begingroup$ Yes, I believe this is what i was looking at. The interpretation with calculus combined with the change of variable for probability densities makes me feel more comfortable about this. $\endgroup$ – boomkin Jun 14 at 10:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.