I am confused about something very trivial.

Suppose that we have 3 random variables, such that:


I am trying to understand the relation between the following expressions:

$E_Z[Z] =? E_{X,Y}[XY] =? E_{Z}[XY] =? E_{X,Y}[Z]$

I can't decide which integrals below yield which expectation and which are equal:

a) $\int p(z) z dz$

b) $\int\int q(x,y) xy dxdy$

c) $\int\int p(xy) xy dxdy$

d) $\int p(z) xy dz$

e) $\int\int q(x,y) z dxdy$

where $p$ is the distribution for $Z$,and $q$ is the joint distribution for $(X,Y)$.

(a) is clearly $E_Z[Z]$.

(b),(c) I am not sure which one is equal to (a).

(d) xy should come out of the integral according to basic calculus and integral should yield 1 but they are dependent on z, so should they?

I think I am confusing myself with random variables vs. symbols in expressions.

  • 3
    $\begingroup$ Please explain what the subscripts on "$E$" are intended to mean: in most cases they appear to be superfluous (as in $E_Z[Z]$) or meaningless (as in $E_Z[XY]$). This seems to be leading into absurdities such as the integrals in (d) and (e), which involve undefined variables. $\endgroup$ – whuber Jan 10 '17 at 19:30
  • $\begingroup$ @whuber They are very common in many probability texts. I suppose I never understood them properly, since I always used the lollipop books without measure theory. $\endgroup$ – Cagdas Ozgenc Jan 10 '17 at 19:34
  • $\begingroup$ Would you find it helpful to have answers that might refer to measure theory? One of its advantages is how it clarifies issues like this. $\endgroup$ – whuber Jan 10 '17 at 19:38
  • $\begingroup$ @whuber Sure I need to learn eventually. I have already too long avoided. $\endgroup$ – Cagdas Ozgenc Jan 10 '17 at 19:39
  • $\begingroup$ A related post you (or potential future readers) may find interesting is stats.stackexchange.com/questions/72613/… $\endgroup$ – Jesper for President Feb 14 '20 at 19:20

Recall the abstract definitions of random variables.

Sample spaces and probability

First, the setting. There is a "sample space" $\Omega$ with generic elements $\omega$. Certain subsets of $\Omega$ are distinguished: they are the ones to which we are willing to assign probabilities; they are called "events" (or "measurable sets"). Let $\mathbb{P}$ denote the probability function. Conceptually, it's a simple and natural thing: it assigns numbers between $0$ and $1$ to all the events, according to the axioms of probability.

Random variables

A random variable $X$ assigns a definite number, written $X(\omega)$, to each $\omega\in\Omega$. A pair of random variables $X$ and $Y$ can be thought of as assigning ordered pairs of numbers, written $(X(\omega), Y(\omega))$, to each $\omega$.

New random variables may be constructed out of existing ones through formulas (as well as in other ways). For instance, $Z=XY$ can be thought of as extending the ordered pairs into ordered triples

$$(X(\omega), Y(\omega), Z(\omega)) = (X(\omega), Y(\omega), X(\omega)Y(\omega)).$$

That is, $Z=XY$ means you take the definite numerical values of $X$ and $Y$ at each outcome $\omega$ and multiply them as numbers.

There is a technical condition applied to random variables $X$, called "measurability." It means that given any number $x$, the set of all outcomes $\omega$ for which the value of $X$ is $x$ or less must be an event (thereby allowing us to find its probability). For much practical and conceptual work we may ignore this technicality. It guarantees that the distribution functions $F_X$, to be defined below, always exist.


The expectation operator generalizes the idea of weighting all the values of a random variable by their probability and summing that up to get an "average" value. It is written

$$\mathbb{E}(X) = \int_\Omega X(\omega)\mathrm{d}\mathbb{P}(\omega).\tag{1}$$

The notation is intended to remind us of the key ideas: $\int$ (which looks like a stretched out "S", for "sum") refers to the generalization of summation; $\Omega$ references the sample space; $X(\omega)$ are the values of $X$; and $d\mathbb{P}(\omega)$ are the associated probabilities. There is a "$\mathrm{d}$" in front of it as a reminder that individual observations $\omega$ might not actually have probabilities, so some special steps have to be taken in general to make sense of this.

Law of the Unconscious Statistician

It is rare to use $(1)$ for computation. Instead, we derive an intermediate mathematical object called the distribution of $X$, often written $F_X$ (even though it depends on $\mathbb{P}$ as well). It simply is the chance of the event $X \le x$:

$$F_X(x) = \mathbb{P}\left(\{\omega\in\Omega\mid X(\omega) \le x\}\right).$$

The Law of the Unconscious Statistician (LOTUS) is a theorem that says the expectation $(1)$ can be computed by integrating $F$:

$$\mathbb{E}(X) = \int_\mathbb{R} x dF_X(x).\tag{2}$$

This is no longer an abstract integral: it's a more familiar object from elementary Calculus. In general it needs to be a Lebesgues-Stieltjes integral in order to handle discrete probability distributions, but it's perfectly fine to think of it as either one of two things:

  • When $F_X$ is discrete with probability $p_X(x)$ that $X=x$ for at most countably many distinct $x$, $(2)$ is the sum $$\mathbb{E}(X) = \sum_x x p_X(x).$$ It is defined only when that sum has the same value regardless of the order in which the $x$ are summed.

  • When $F_X$ is absolutely continuous with continuous probability density $f_X(x) = F^\prime_X(x)$, it is the Riemann integral $$\mathbb{E}(X) = \int_{-\infty}^\infty x f(x) dx.$$ It is defined only when this integral converges (it's a double limit as the right and left endpoints go to $\pm\infty$).

When there are multiple random variables, the distribution is defined similarly. For instance,

$$F_{(X,Y)}(x,y) = \mathbb{P}\left(\{\omega\in\Omega\mid X(\omega)\le x, Y(\omega) \le y\}\right)$$

for all numbers $x$ and $y$. The LOTUS now involves two-dimensional real integrals, frequently expressed as Riemann or Lebesgue integrals. The concept of expectation needs to be generalized slightly, though, because the ordered pair $(X,Y)$ is never just a number. Instead, let $h$ be a function of two real variables. Provided that $h$ is measurable and the chance of $h(X,Y)$ being undefined is zero (which permits, among other things, the possibility of analyzing functions like $h(x,y)=x/y$ which are not defined when $y=0$), $h(X,Y)$ is another random variable. This is meant in exactly the same sense $Z=XY$ is a random variable: it's a way of combining the numbers $X(\omega)$ and $Y(\omega)$ into a third number $h(X(\omega), Y(\omega))$ for each $\omega$ (throwing away any $\omega$ for which this combination is undefined, provided the set of such $\omega$ has zero probability). Now

$$\mathbb{E}(h(X,Y)) = \int_\mathbb{R}\int_\mathbb{R} h(x,y) \frac{\partial^2}{\partial x\partial y}F(x,y)\mathrm{d}x\mathrm{d}y$$

(with a double sum appearing instead of a double integral for discrete distributions). The function $\frac{\partial^2}{\partial x\partial y}F(x,y)$ (if it exists) is called the bivariate density of $(X,Y)$.


Let's use the LOTUS to resolve the questions. Assume $F_{X,Y}$ has a bivariate density $\frac{\partial^2}{\partial x\partial y}F(x,y) = q(x,y)$ and that $F_Z$ has a density $p(z)$.

  1. $\mathbb{E}(Z)$ (there's no need for a subscript) must refer to an expression like $(1)$; namely, $$\mathbb{E}(Z) = \int_\Omega Z(\omega)d\mathbb{P}(\omega).$$ LOTUS asserts this can be written as an "ordinary" integral, $$\mathbb{E}(Z) = \int_\Omega z\,p(z)dz.$$ The (bivariate) LOTUS, applied to $Z$ in in terms of $(X,Y)$, says this can be written $$\mathbb{E}(Z)= \mathbb{E}(XY)= \int_\Omega X(\omega)Y(\omega)\mathrm{d}\mathbb{P}(\omega)= \int_{\mathbb{R}^2} x y\, q(x,y)\mathrm{d}x\mathrm{d}y.$$

  2. $\mathbb{E}_{X,Y}(XY)$ asks for the expectation of $XY=Z$ while, via the subscripts, making it explicitly clear that both $X$ and $Y$ are considered random variables. (Alternatively, we might fix the value of one of them and take an expectation with respect to the other: those are the marginal expectations.) Thus $$\mathbb{E}_{X,Y}(XY) = \int_\Omega X(\omega)Y(\omega)d\mathbb{P}(\omega).$$ Clearly this is exactly the same thing as $\mathbb{E}(Z)$ (above).

  3. Expressions like "$\mathbb{E}_Z(XY)$" and "$\mathbb{E}_{XY}(Z)$" don't seem to make much sense. If they were to appear somewhere, I would first attempt to understand them both as $\mathbb{E}(Z)$.

  4. We can use LOTUS in reverse. Take, for instance, the expression $$\int_{\mathbb{R}} z p(z) \mathrm{d}z$$ from the question. That sure looks like an expectation. Here's what happens: $Z$ itself is a random variable. It therefore has a distribution $F_Z(z)$ defined in the usual way. Let this distribution have a density $p$. LOTUS asserts $$\mathbb{E}(Z) = \int_\mathbb{R} z p(z) \mathrm{d}z.$$ Comparing this to the foregoing shows that $$\int_\mathbb{R} z p(z) \mathrm{d}z = \mathbb{E}(Z) = \int_{\mathbb{R}^2} x y\, q(x,y)\mathrm{d}x\mathrm{d}y.$$

  5. The integral $(c)$ in the question is not an expectation and $(d)$ and $(e)$ are not well-defined because they involve undefined, uninstantiated variables. (For instance, $(d)$ involves $x$ and $y$ which aren't integrated out.)

I relied implicitly on many resources in writing this very brief account of the notation. One of the very best and most accessible introductions is

Steven Shreve, Stochastic Calculus for Finance II: Continuous-Time Models, Chapter 1. Springer, 2004.

I also had in mind some of the concepts (especially concerning bivariate distributions) introduced in

Roger Nelsen, An Introduction to Copulas, 2nd Ed., Chapter 2. Springer, 2006.

  • $\begingroup$ Thanks for the long post. Two quick follow ups. With the same logic for example E[f(Z)] will be equal to E[f(XY)] for some function f, right? Does integral (c) have some other meaningful interpretation? $\endgroup$ – Cagdas Ozgenc Jan 11 '17 at 5:57
  • $\begingroup$ Yes, $E[f(Z)]=E[f(XY)]$ becauase $Z=XY$. I am having trouble conceiving of any meaningful interpretation of integral $(c)$. $\endgroup$ – whuber Jan 11 '17 at 14:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.