0
$\begingroup$

Problem

Suppose I have a random variable $z$ following a distribution $p(z)$. Suppose I have a transformation

$$ f(z) = x $$

that transforms the random variable $z$ into a new random variable $x$ with distribution $p(x)$. I have often seen the following result

$$ \mathbb{E}_{p(x)}[g(x)] = \mathbb{E}_{p(z)}[g(f(z))]. $$

In other words, the expectation with respect to a distribution $p(x)$ can be written in terms of the original distribution $p(z)$.

Is there proof of this? I think this should work even if $f(z)$ is not invertible and/or differentiable.

My Set-Up for a Solution

I will describe my measure theory set up.

Distribution of Z

Suppose we have two measurable spaces $(\Omega, \mathcal{F})$ and $(\mathsf{Z}, \mathcal{Z})$. The random variable $Z$ is a measurable mapping $$ Z: \Omega \to \mathsf{Z} $$

such that the pre-image $Z^{-1}(B)$ of any $\mathcal{Z}$-measurable set $B\in \mathcal{Z}$ is also $\mathcal{F}$-measurable:

$$ Z^{-1}(B) = \{\omega\in \Omega \, :\, Z(\omega) \in B \} \in \mathcal{F} \qquad \forall \, B \in \mathcal{Z} $$ Now the distribution of $Z$ is a push-forward measure. Suppose we have a probability space $(\Omega, \mathcal{F}, \mathbb{P})$. This means we can measure sets in $\mathcal{F}$. The push-forward or distribution for $Z$ is a way of measuring sets in $\mathcal{Z}$ via $\mathbb{P}$.

Basically, the distribution $Z_*\mathbb{P}$ assigns to sets $B\in\mathbb{Z}$ the same value as if we map $B$ back to $\mathcal{F}$ via $Z^{-1}$ first, and then we find its measure using $\mathbb{P}$.

$$ (\mathbb{P} \circ Z^{-1})(B) = Z_*\mathbb{P}(B) \qquad \forall \, B\in \mathcal{Z} $$

Distribution of X

Now, the new random variable $X$ is basically a function of the random variable $Z$ and therefore $X$ is also a random variable.

Consider the probability space $(\mathsf{Z}, \mathcal{Z}, Z_*\mathbb{P})$ for $Z$. Consider also a measurable function $$ X: \mathsf{Z} \to \mathsf{X} $$ where $(\mathsf{X}, \mathcal{X})$ is a measurable space. This essentially defines the random variable $X$. Since we can measure sets in $\mathcal{Z}$ using $Z_*\mathbb{P}$ we would like to measure sets in $\mathcal{X}$ too. To do this, we define the distribution of $X$ to be a push-forward measure. Essentially, to give a measure to a set $C\in \mathcal{X}$ it maps it to $\mathcal{Z}$ via $X^{-1}$ and then measures it with the distribution $Z_*\mathbb{P}$.

$$ (Z_*\mathbb{P} \circ X^{-1})(C) = X_*Z_*\mathbb{P}(C) \qquad \forall \, C\in\mathcal{X} $$

Expected Value with respect to $Z$

I am using this definition. $$ \mathbb{E}_{p(Z)}(Z) = \int_{\mathsf{Z}} Z(\omega_z) \,\,d Z_*\mathbb{P}(\omega_z) $$

Expected Value with respect to $X$

$$ \mathbb{E}_{p(X)}[X] = \int_{\mathsf{X}} X(\omega_x) \,\, d X_*Z_*\mathbb{P}(\omega_x) $$

$\endgroup$
3
  • 1
    $\begingroup$ Could the law of unconscious statistician be not applied here? As in, if define a function $h(z)\equiv g(f(z))$ and then use the law? $\endgroup$ – Dayne Oct 21 '20 at 11:35
  • $\begingroup$ @Dayne Mmm Interesting.. but here there's an additional change: the expectation on the right-hand side is taken with respect to $p(z)$, while on the left-hand side is taken with respect to $p(x)$. $\endgroup$ – Euler_Salter Oct 21 '20 at 11:38
  • 4
    $\begingroup$ @Euler_Salter: This is the so-called law of unconscious statistician, exactly. (The proof in Wikipedia is alas unnecessarily limited to the case of $g$ being invertible. The only constraint is actually that the transform $f$ is measurable in order for $X$ to be a random variable.) $\endgroup$ – Xi'an Oct 21 '20 at 12:00
3
$\begingroup$

Let $Z$ be a random variable with distribution $P^Z$, meaning that for any measurable set $A$,$$\mathbb P(Z\in A)=P^Z(A)$$ Then, for any measurable transform $f$, $X=f(Z)$ is a random variable with distribution $P^X$ such that, for any measurable set $A$,$$P^X(A)=\mathbb P(X\in A)=\mathbb P(f(Z)\in A)=\mathbb P(Z\in f^{-1}(A))=P^Z(f^{-1}(A))$$ where $$f^{-1}(A)=\{x;\ f(x)\in A\}$$ (which applies even when $f$ is not invertible).

This means that, when $g(\cdot)$ is an indicator function, $\mathbb I_A$, the equality \begin{align}\mathbb E^{P^X}[g(X)]&=\mathbb E^{P^X}[\mathbb I_A(X)]\\ &=\mathbb P^X(A)\\ &=\mathbb P^Z(f^{-1}(A)]\\ &=\mathbb E^{P^Z}[\mathbb I_{f^{-1}(A)}(Z)]\\ &=\mathbb E^{P^Z}[\mathbb I_A(f(Z))]=\mathbb E^{P^Z}[g(f(Z))] \end{align} stands. The conclusion follows (as usual) when writing any measurable function $g$ as a limit of weighted sums of indicator functions. The expectation under the push-forward measure $P^X$ is indeed the expectation of the $f$-transformed variate under the initial measure $P^Z$: $$\mathbb E^{P^X}[g(X)]=\mathbb E^{P^Z}[g(f(Z))]$$

$\endgroup$
4
  • $\begingroup$ Thank you! I can see that you've used the definition of a push-forward measure. But how does this prove the equality between the expected values? The definition of expected value that I am using is this $$ \mathbb{E}_{p(X)}[g(X)] = \int_{\mathsf{X}} g(X(\omega_x)) d P^X(\omega_x). $$ How does this relate to the other expected value? $$ \mathbb{E}_{p(Z)}[g(f(Z))] = \int_{\mathsf{Z}} g(f(Z(\omega_z))) dP^Z(\omega_z) $$ $\endgroup$ – Euler_Salter Oct 21 '20 at 12:49
  • $\begingroup$ @whuber I've added some more measure theory to ease the path towards a proof. $\endgroup$ – Euler_Salter Oct 21 '20 at 12:55
  • 2
    $\begingroup$ That's fine, but it strikes me that you are asking a non-measure-theoretic question framed in terms of distributions rather than random variables: "the expectation with respect to a distribution $p(x)$ can be written in terms of the original distribution $p(z).$" Although introducing random variables is natural, the way you have asked the question strongly suggests you were looking for more direct proofs. $\endgroup$ – whuber Oct 21 '20 at 13:00
  • 1
    $\begingroup$ (I cope with such situations by deleting my partial answer. Later I can edit it and undelete it.) $\endgroup$ – whuber Oct 21 '20 at 13:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.