5
$\begingroup$

Let $(\Omega, \mathcal{A}, \mathbb{P})$ be a probability space and $X:(\Omega,\mathcal{A})\rightarrow(\mathcal{X}, \mathcal{F})$ and $Z:(\Omega, \mathcal{A}) \rightarrow (\mathcal{Z}, \mathcal{G})$ two random variables. The unconditional version of the law of the unconcious statistician is $$\mathbb{E}[g(X)]=\int_{\Omega}g(X)(\omega)\mathbb{P}(d\omega)=\int_{\mathcal{X}}g(x)\mathbb{P}_X(dx)$$ 1.) How does it look like for the conditional case: $$\mathbb{E}[g(X)|Z]=\int_{\mathcal{X}}g(x)\mathbb{P}_{X|Z}(dx)$$

2.) How does it look for the case when we fix one event $z$: $$\mathbb{E}[g(X)|Z=z]=\int_{\mathcal{X}}g(x)\mathbb{P}_{X|Z=z}(dx)$$

I.e., what I want to find is an expression of these conditional expectation w.r.t an integration over the domain $\Omega$.

The underlying question here is: Of what conditional distribution is $\mathbb{P}_{X|Z}$ (and $\mathbb{P}_{X|Z=z}$) a pushforward measure of?

EDIT: $\mathbb{P}_{X|Z}$ is a regular conditional distribution from $(\Omega, \sigma(Z))$ into $(\mathcal{X}, \mathcal{F})$, while $\mathbb{P}_{X|Z=\cdot}$ is a regular conditional distribution from $(\mathcal{Z}, \mathcal{G})$ into $(\mathcal{X}, \mathcal{F})$.

Now, is it correct that $$\mathbb{E}[g(X)|Z](\omega)=\int_{\mathcal{X}}g(x)\mathbb{P}_{X|Z}(dx)(\omega)=\int_{\Omega} g(X(\omega'))\mathbb{P}(d\omega'|\sigma(Y))(\omega),$$ where $\mathbb{P}_{X|Z}(F)=\mathbb{P}(X^{-1}(F)|\sigma(Z))$, i.e., the conditional pushforward measure and $\mathbb{P}(A|\sigma(Y))$ is assumed to be a regular conditional probability?

$\endgroup$
5
  • $\begingroup$ Does this answer your question? Law of unconscious statistician for conditional expectation $\endgroup$ Commented May 18 at 13:39
  • $\begingroup$ Hi I had seen this question before posting my own one but unfortunately it doesnt answer my questions $\endgroup$
    – guest1
    Commented May 18 at 14:08
  • $\begingroup$ Please elaborate on why the apparent duplicate does not answer your questions. Also check out our closely related posts such as stats.stackexchange.com/questions/230545 or search our site. $\endgroup$
    – whuber
    Commented May 18 at 14:42
  • $\begingroup$ I dont understand how the change of variables works for conditional distributions $\endgroup$
    – guest1
    Commented May 18 at 15:25
  • 1
    $\begingroup$ @whuber can you not maybe just give a one liner of how the law of the unconscious statistician, i.e., the change of variables formula, looks like for the conditional expectation? I.e., how does the integral from (1) look like if integrated over $\Omega$ against some conditional probabiality/distribution? And how this conditional probability relates to $\mathbb{P}_{X|Z}$ $\endgroup$
    – guest1
    Commented May 19 at 7:53

3 Answers 3

6
$\begingroup$

Firstly,

Result $1.$ Let $(M,\mathscr G)$ be a Borel space. Let $X: (\Omega,\mathscr A) \to(M,\mathscr G)$ be a random object and let $\mathscr D\subset\mathscr A$ be a sub–$\sigma$–algebra. Then there exists a regular conditional distribution version of $X$ given $\mathscr D,~\mathbb P_{X\mid\mathscr D}(\cdot,\cdot)$ on $\mathscr G\times \Omega.$

Consider a real-valued measurable function $h$ defined on $M.$ Then one can write

$$\mathbf E(h\circ X\mid\mathscr D)(\omega) =\int_M h(x) ~\mathbb P_{X\mid \mathscr D}(\mathrm dx, \omega).\tag 1\label 1$$

By definition, for $B\in\mathscr G, ~B\mapsto \mathbb P_{X\mid \mathscr D}(B, \omega) $ is a measure almost surely on $\mathscr G$ and $\omega\mapsto \mathbb P_{X\mid \mathscr D}(B, \omega)\in\mathbb P\left(X^{-1}(B)\mid \mathscr D\right) (\omega).$ Using this, one can prove $\eqref 1$ for $h:=1\!\!1_B,~B\in\mathscr G. $

Take another random object $Z: (\Omega,\mathscr A) \to(S,\mathscr B);$ if $\mathscr D$ is taken to be $\sigma(Z) $ above, then one gets a regular conditional probability of $X$ given $Z,~\mathbb P_{X\mid Z}$ on $\mathscr G\times\Omega.$

However one can also define a regular image conditional probability of $X$ given $Z, ~\mathbb P_{X\mid Z=z}$ on $\mathscr G\times S$ as $B\mapsto \mathbb P_{X\mid Z=z} (B, z) $ being a probability measure on $\mathscr G$ and $z\mapsto \mathbb P_{X\mid Z=z}(B, z) \in\mathbb P(X^{-1}(B)\mid Z=z)$ almost surely $\mathbb P_Z.$

Similar to $\eqref 1,$ one can by using $\mathbb P_{X\mid Z=z}$ compute $\mathbf E(h\circ X\mid Z=z). $

One can also use conditional density of random variables $X$ given $Z$ defined as

$$g(x\mid z) :=\begin{cases}\frac{f(z,x)}{f_Z(z)} ~&\forall z:f_Z(z) \ne 0\\f_0(x)~&\forall z:f_Z(z) = 0\end{cases}\tag 2\label 2$$ where $f_0(\cdot) $ is an arbitrary density.

One can check using $\eqref 2, $ for $A\in\mathscr B_\mathbb R, $

\begin{align}\int_B\left[\int_A g(x\mid z) ~\mathrm dx\right]~\mathbb P_Z( \mathrm dz) &=\iint_{B\times A} f(z, x) ~\mathrm dz\times\mathrm dx\\&= \mathbb P((Z, X) \in B\times A) \\&=\mathbb P([X\in A]\cap Z^{-1}(B))\tag 3\end{align} that $\int_A g(x\mid z) ~\mathrm dx$ being a measurable function on $(\mathbb R, \mathscr B_\mathbb R) ,$ by Result $1.$ is a version, almost surely $\mathbb P_Z, $ of the regular conditional distribution $\mathbb P(X\in A\mid Z=z) . $

Consider a Borel map $h(\cdot) $ such that $\mathbf E\vert h(X) \vert<\infty.$ Then using $\eqref 2,$

\begin{align}\int_B\left[\int_\mathbb R h(x)g(x\mid z) ~\mathrm dx\right]~\mathrm d\mathbb P_Z(z) &=\iint_{B\times \mathbb R} h(x)f(z, x) ~\mathrm dz\times\mathrm dx\\ \iint_{Z^{-1}(B)} h(y) ~\mathrm d\mathbb P_{Z, X}(z, x) \\\int_B\mathbf E(h(X) \mid Z=z) ~\mathbb P_Z(\mathrm dz);\tag 4\label 5\end{align} from $\eqref 5,$ almost surely $\mathbb P_Z, $

$$\mathbf E[h(X) \mid Z=z]=\int_{-\infty}^\infty h(x) g(x\mid z) ~\mathrm dx. $$

--

References:

$\rm[I]$ Real Analysis and Probability, R. M. Dudley, CUP, $2004, $ sec. $10.2.$

$\rm[II]$ Probability for Statisticians, Galen R. Shorack, Springer-Verlag, $2017, $ pp. $144-146.$

$\rm[III]$ Martingales and Stochastic Analysis, J. Yeh, World Scientific, $1995, $ appendix $\rm C. $

$\endgroup$
2
  • $\begingroup$ Hi first of all thank you for all your efforts! I wanted to ask if you could write your answer with measures only and without densities. Moreover without Riemann integrals and only with Lebesgue integral? This would help me a lot! $\endgroup$
    – guest1
    Commented May 19 at 4:19
  • $\begingroup$ And a question: in your last formula, where does the &y& dependence of your cond distribution come from? Also this is still an integral over &M& but i would like to see how that looks like as an integral over &/Omega& such that it corresponds to the change of variables formula when compared to the equation in my question $\endgroup$
    – guest1
    Commented May 19 at 4:22
3
$\begingroup$

Before answering your formal questions, it is extremely important to clarify the (difficult!) notion of conditional distribution first. In order to do so, let me quote its rigorous definition from Theorem 33.3 of Probability and Measure (3rd ed.) by Patrick Billingsley below.

There exists a function $\mu(H, \omega)$, defined for $H$ in $\mathscr{R}^1$ and $\omega$ in $\Omega$, with these two properties:

  1. For each $\omega$ in $\Omega$, $\mu(\cdot, \omega)$ is a probability measure in $\mathscr{R}^1$.
  2. For each $H$ in $\mathscr{R}^1$, $\mu(H, \cdot)$ is a version of $P[X \in H | \mathscr{G}]$.

The probability measure $\mu(\cdot, \omega)$ is a conditional distribution of $X$ given $\mathscr{G}$. If $\mathscr{G} = \sigma(Z)$, it is a conditional distribution of $X$ given $Z$.

N.B. Before reading on, to link the above definition, as well as my subsequent write-ups to your question, please note that my primitive probability space (as Billingsley's book) is denoted by $(\Omega, \mathscr{F}, P)$, $\mathscr{G}$ is a sub-$\sigma$-field of $\mathscr{F}$. That said, the "$P$" in my answer corresponds to the "$\mathbb{P}$" in the OP. In addition, both $(\mathcal{X}, \mathcal{F})$ and $(\mathcal{Z}, \mathcal{G})$ in your post are interpreted as the Borel measurable space $(\mathbb{R}^1, \mathscr{R}^1)$ -- with an arbitrary space $(\mathcal{X}, \mathcal{F})$ or $(\mathcal{Z}, \mathcal{G})$, conditional distributions may fail (see the last two responses below) to exist at all then it is baseless to discuss your core questions.

This Theorem/Definition asserts that for each $\omega \in \Omega$, $\mu(\cdot, \omega)$ is a probability measure in $\mathscr{R}^1$. Intuitively, this enables one to reidentify the conditional expectation $E[g(X)|\mathscr{G}]$, when evaluated at a fixed $\omega$ in $\Omega$, as the integral of $x$ with respect to $\mu(\cdot, \omega)$ just as the unconditional expectation $E[g(X)] = \int_{\mathbb{R}^1}g(x)\mu(dx)$ is calculated (i.e., the law of the unconscious statistician), namely, it is reasonable to state \begin{align*} E[g(X)|\mathscr{G}](\omega) = \int_{\mathbb{R}^1}g(x)\mu(dx, \omega). \tag{1}\label{11} \end{align*}

This intuition is indeed true (in the "almost surely" sense, of course), as confirmed by Theorem 34.5 in the same reference (I slightly changed the $\varphi$ in the verbatim to $g$ to align with the notations in the post):

Let $\mu(\cdot, \omega)$ be a conditional distribution with respect to $\mathscr{G}$ of a random variable $X$, in the sense of Theorem 33.3. If $g: \mathbb{R}^1 \to \mathbb{R}^1$ is a Borel function for which $g(X)$ is integrable, then $\int_{\mathbb{R}^1}g(x)\mu(dx, \omega)$ is a version of $E[g(X)|\mathscr{G}]_\omega$.

In other words, this theorem says that $\eqref{11}$ holds for all $\omega \in \Omega$ outside a null set.

Having made the above theoretical preparations, now let me respond to your specific questions respectively.

1.) How does it look like for the conditional case: $$\mathbb{E}[g(X)|Z]=\int_{\mathcal{X}}g(x)\mathbb{P}_{X|Z}(dx)$$

Overall, this is a correct statement in view of $\eqref{11}$ (after identifying the "$\mathbb{P}_{X|Z}$" in your expression as the "$\mu$" in $\eqref{11}$). However, it can be made more precise if you can explicitly add "$\omega$" into the expression, to highlight that for different $\omega \in \Omega$, $\mathbb{P}_{X|Z}(\cdot, \omega)$ is a different probability measure in $\mathscr{R}^1$, which in turn results in different values of $E[g(X)|Z]$ evaluated at different $\omega$'s (as $E[g(X)|Z]$ itself is a random variable). Also note that $\eqref{11}$ may fail on an exceptional $\omega$-set of probability $0$. Therefore, a precise rephrase of the equation you presented should be

\begin{align*} \mathbb{E}[g(X)|Z]_\omega =\int_{\mathcal{X}}g(x)\mathbb{P}_{X|Z}(dx, \omega), \; \text{ for } \omega \in A \text{ with } P(A) = 1. \tag{2}\label{22} \end{align*}

2.) How does it look for the case when we fix one event $z$: $$\mathbb{E}[g(X)|Z=z]=\int_{\mathcal{X}}g(x)\mathbb{P}_{X|Z=z}(dx)$$

This is more like a question "given $E[g(X)|Z]$ (whose definition is clear), how does one interpret the notation $E[g(X)|Z = z]$?" I noticed that you have asked a similar question before, to which I have given a more detailed answer under that post. Long story short, since $E[g(X)|Z]$ is $\sigma(Z)$-measurable, it can be denoted as $f(Z)$ for some Borel function $f$. It then follows that $E[g(X)|Z = z]$ is just $f(z)$. Therefore, to this question, there is no need to bring an additional new notation "$\mathbb{P}_{X|Z=z}$" to this already crowded party. Instead, simply substituting $z$ into $f(Z) := E[g(X)|Z]$, whose expression can be given by $\eqref{22}$, to obtain the value $E[g(X)|Z = z]$.

$\mathbb{P}_{X|Z}$ is a regular conditional distribution from $(\Omega, \sigma(Y))$ into $(\mathcal{X}, \mathcal{F})$, while $\mathbb{P}_{X|Z=\cdot}$ is a regular conditional distribution from $(\mathcal{Z}, \mathcal{G})$ into $(\mathcal{X}, \mathcal{F})$.

This statement is largely imprecise, if not incorrect. First of all, as I said above, there is no need to further define "$\mathbb{P}_{X|Z=\cdot}$" given that you had defined "$\mathbb{P}_{X|Z}$". Secondly, by saying "$\mathbb{P}_{X|Z}$ is a regular conditional distribution from $(\Omega, \sigma(Y))$ into $(\mathcal{X}, \mathcal{F})$", I guess you probably referred to the second property of a conditional distribution (i.e., bullet point 2 in the first quotation block above). However, this property holds only when an event $H$ in $\mathscr{R}^1$ is fixed -- identifying a conditional distribution itself (which is a bivariate mapping from $\mathscr{R}^1 \times \Omega$ to $\mathbb{R}_+$) as a single measurable mapping is a misconception.

Now, is it correct that $$\mathbb{E}[g(X)|Z](\omega)=\int_{\mathcal{X}}g(x)\mathbb{P}_{X|Z}(dx)(\omega)=\int_{\Omega} g(X(\omega'))\mathbb{P}(d\omega'|\sigma(Y))(\omega),$$ where $\mathbb{P}_{X|Z}(F)=\mathbb{P}(X^{-1}(F)|\sigma(Z))$, i.e., the conditional pushforward measure ?

This is an interesting conjecture, but it (the second equality) is not correct. The profound reason is that in general, "$P(\cdot|\sigma(Y))$" is not a probability measure for each $\omega$ so it makes no sense to define an expectation with respect to it. Billingsley explained this paradox as follows:

It might be thought that for an arbitrary $\sigma$-field $\mathscr{G}$ in $\mathscr{F}$ versions of the various $P[A|\mathscr{G}]$ can be chosen that $P[A|\mathscr{G}]_\omega$ is for each fixed $\omega$ a probability measure as $A$ varies over $\mathscr{F}$. It is possible to construct a counterexample showing that this is not so. The example is possible because the exceptional $\omega$-set of probability $0$ where $P[\cup_n A_n | \mathscr{G}] = \sum_n P[A_n | \mathscr{G}]$ fails depends on the sequence $A_1, A_2, \ldots$; if there are uncountably many such sequences, it can happen that the union of these exceptional sets has positive probability whatever versions $P[A|\mathscr{G}]$ are chosen.

Moreover, by stating "$\mathbb{P}_{X|Z}(F) = \mathbb{P}(X^{-1}(F)|\sigma(Z))$" (this is wrong because of the reason outlined above), it shows that you did not fully understand how the conditional distribution "$P_{X|Z}$" is really constructed. At this moment, it is time to reveal that why the conditional distribution exists in $\mathscr{R}^1$ rather than any abstract measurable space: it is that the topological structure of the real line ensures that the union of "exceptional sets" (which has the same cardinality as the set of rational numbers) in $\mathscr{R}^1$ has zero probability. For details, check out the proof to Theorem 33.3 in the same reference.

$\endgroup$
11
  • $\begingroup$ Hi and thank you for your efforts in answering my question. I want to comment on a few things here: First, the quote "$\mathbb{P}_{X|Z}$ is a regular conditional distribution from $(\Omega, \sigma(Z)$ into $(\mathcal{X}, \mathcal{F})$" is an expression regularly used by Bauer in his book probability theory to say that it is a Markov kernel $\mathbb{P}_{X|Z}:\Omega\times\mathcal{F}\rightarrow [0,1]$. Second, by $\mathbb{P}(d\omega'|\sigma(Z))$ I meant a regular conditional probability, i.e., one that IS a probability measure for each $\omega$. $\endgroup$
    – guest1
    Commented May 19 at 19:25
  • $\begingroup$ Third, I would be surprised if the equation $\mathbb{P}_{X|Z}(F)=\mathbb{P}(X^{-1}(F)|\sigma(Z))$ was wrong since it is taken from the lecture notes of P.J.C Spreij on measure theoretic probability theory, p.88. staff.fnwi.uva.nl/p.j.c.spreij/onderwijs/master/mtp.pdf $\endgroup$
    – guest1
    Commented May 19 at 19:31
  • 1
    $\begingroup$ @guest1 I see your point. What I wrote above is for the most general case (i.e., without any further restrictions). If you manually requires that $P(\cdot|\sigma(Y))$ is a probability measure for each $\omega$ (as your source called it "regular"), then the answer to your last question should be affirmative (as a consequence of change-of-variable theorem). $\endgroup$
    – Zhanxiong
    Commented May 19 at 19:31
  • 1
    $\begingroup$ @guest1 It is correct, only if, $P(\cdot|\sigma(Z))$ itself is a probability measure a priori (which I have shown you that it is not the case in general). $\endgroup$
    – Zhanxiong
    Commented May 19 at 19:33
  • $\begingroup$ What I also realized is that you have not answered my actual question, which was if and how 1.) and 2.) can be written via the law of the unconsciuos statistician in terms of an integral over $\Omega$. Related to that, another question would be if the equation from my EDIT is correct now that I have clarified that by $\mathbb{P}(d\omega'|\sigma(Z))$ I mean a regular conditiona probaiblity $\endgroup$
    – guest1
    Commented May 19 at 19:34
-3
$\begingroup$

$$\mathbb{E}[g(X)|Z]=\int_{\Omega}g(X)dP_{\Omega/Z}=\int_{\mathcal{X}}g(x)dP_{X/Z}$$ $$\mathbb{E}[g(X)|Z=z]=\int_{\Omega}g(X)dP_{\Omega/Z}(z)=\int_{\mathcal{X}}g(x)dP_{X/Z}(z)$$

$\endgroup$
9
  • 6
    $\begingroup$ It is difficult to make sense of your contradictory notation: could you please explain it? $\endgroup$
    – whuber
    Commented May 18 at 14:43
  • 6
    $\begingroup$ Your notation is nonstandard and the inclusion of an argument "$z$" in the differentials in the second formula is strange. Because these formulas are unaccompanied by any explanation or derivation, it is crucial that your notation be conventional and clear, which is not the case. $\endgroup$
    – whuber
    Commented May 18 at 16:47
  • 2
    $\begingroup$ In both sets of equations, the conditional expectation and the conditional probability differential are functions. That's why I find it strange that in one case you do not write an explicit argument and in the other case you do. $\endgroup$
    – whuber
    Commented May 19 at 14:47
  • 2
    $\begingroup$ The former means the functions $f$ and $g$ are identical. The latter means they agree at the argument $x.$ $\endgroup$
    – whuber
    Commented May 19 at 15:42
  • 3
    $\begingroup$ I think I see where you're going with this. The voting here, though, indicates many are hoping you will explain your notation. Otherwise your post is likely to be underappreciated. $\endgroup$
    – whuber
    Commented May 19 at 18:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.