113
$\begingroup$

What is the exact meaning of the subscript notation $\mathbb{E}_X[f(X)]$ in conditional expectations in the framework of measure theory ? These subscripts do not appear in the definition of conditional expectation, but we may see for example in this page of wikipedia. (Note that it wasn't always the case, the same page few months ago).

What should be for example the meaning of $\mathbb{E}_X[X+Y]$ with $X\sim\mathcal{N}(0,1)$ and $Y=X+1$ ?

$\endgroup$
3
  • 13
    $\begingroup$ No doubt someone will chime in with formal definitions, informally, all expectations are expectations over the distribution of (/expectation with respect to) some (possibly multivariate) random variable, whether it has been explicitly specified or left implied. In many cases it's obvious ($\text{E}(X)$ implies $\text{E}_X(X)$ rather than $\text{E}_W(X)$). Other times, it's necessary to distinguish; consider the law of total variance for example: $\text{Var}[Y] = \text{E}_X\left[\text{Var}[Y\mid X]\right] + \text{Var}_X\left[\text{E}[Y\mid X]\right]$. $\endgroup$
    – Glen_b
    Commented Oct 12, 2013 at 11:32
  • 4
    $\begingroup$ @Glen_b Is it really necessary to specify in the law of total variance? As $E[Y|X]=f(X)$, for some $f$, isn't it clear that $\text{Var}[E[Y|X]]$ is over $X$? $\endgroup$ Commented Oct 18, 2015 at 17:32
  • 5
    $\begingroup$ @ThomasAhle You're quite right -- "necessary" was too strong a word for that example. While strictly speaking it should be clear, it's often a point of confusion for readers unusued to working with it, so it's common, rather than necessary, to be explicit about it. There are some expressions involving expectations where you can't be sure without specifying, but that isn't really one of them $\endgroup$
    – Glen_b
    Commented Oct 18, 2015 at 21:39

3 Answers 3

151
$\begingroup$

In an expression where more than one random variables are involved, the symbol $E$ alone does not clarify with respect to which random variable is the expected value "taken". For example

$$E[h(X,Y)] =\text{?} \int_{-\infty}^{\infty} h(x,y) f_X(x)\,dx$$ or $$E[h(X,Y)] = \text{?} \int_{-\infty}^\infty h(x,y) f_Y(y)\,dy$$

Neither. When many random variables are involved, and there is no subscript in the $E$ symbol, the expected value is taken with respect to their joint distribution:

$$E[h(X,Y)] = \int_{-\infty}^\infty \int_{-\infty}^\infty h(x,y) f_{XY}(x,y) \, dx \, dy$$

When a subscript is present... in some cases it tells us on which variable we should condition. So

$$E_X[h(X,Y)] = E[h(X,Y)\mid X] = \int_{-\infty}^\infty h(x,y) f_{h(X,Y)\mid X}(h(x,y)\mid x)\,dy $$

Here, we "integrate out" the $Y$ variable, and we are left with a function of $X$.

...But in other cases, it tells us which marginal density to use for the "averaging"

$$E_X[h(X,Y)] = \int_{-\infty}^\infty h(x,y) f_{X}(x) \, dx $$

Here, we "average over" the $X$ variable, and we are left with a function of $Y$.

Rather confusing I would say, but who said that scientific notation is totally free of ambiguity or multiple use? You should look how each author defines the use of such symbols.

$\endgroup$
29
  • 7
    $\begingroup$ I have two questions. 1) Not sure if I understand this properly, can I interpret the expectation as one of the first two equations, if either X or Y has been fixed? 2) Can you give an example for EQ 4 and EQ 5? I have a hard time interpreting them and I think concrete examples would help. Thanks! $\endgroup$ Commented Oct 14, 2013 at 18:50
  • 2
    $\begingroup$ @ceiling cat 1) $E[h(X,\bar y)] = \int_{-\infty}^{\infty} h(x,\bar y) f_X(x)dx$ is correct because essentially you do not have two random variables any more. Likewise for fixing $X$ to $\bar x$. $\endgroup$ Commented Oct 14, 2013 at 19:10
  • 6
    $\begingroup$ @ceiling cat 2)-EQ5 : Consider $Z = X^2(Y-(Y+2)^3) = h(X,Y)$. $Z$ is a random variable alright (for an appropriate support). Then using the specific meaning for the short hand notation, $E_X(Z)=E_X[(h(X,Y)] = \int_{-\infty}^{\infty} x^2(y-(y+2)^2) f_{X}(x)dx$ where $f_{X}(x)$ is the density of $X$ (whatever that is). Obviously $Y$ is not integrated, and it will stay intact. But the result you will obtain won't be a number (as in my previous comment), but a random variable (a function of $Y$), since $Y$ here is not fixed, just not-integrated out. $\endgroup$ Commented Oct 14, 2013 at 19:18
  • 2
    $\begingroup$ @ceiling cat In both cases in my two previous comments, the "mechanics" of mathematical calculations will be the same. The end results though have different interpretations. $\endgroup$ Commented Oct 14, 2013 at 19:22
  • 2
    $\begingroup$ @ceiling cat 2)-EQ4: Consider the same random variable $Z$. Its expected value conditional on $X$ is (using the other meaning for the shorthand notation) $E_X[Z] = E(Z\mid X) = \int_{-\infty}^{\infty} z f_{Z|X}(z\mid x)dz$. Note that here the $x$'s and $y$'s do not appear directly in the integrand -they are "condensed" in the $z$ symbol. $\endgroup$ Commented Oct 14, 2013 at 19:29
6
$\begingroup$

I just want to add a follow-up to Alecos' great answer. Sometimes it doesn't matter the exact R.V. (or set of RV) the expectation is over. For instance,

$$ E_{X\sim P(X)} [X] = E_{X\sim P(X,Y)}[X] $$

In your particular question, I suspect that because you are given $h(X,Y)$ is linear in X and Y, then you will break it up into the "marginal" expectations $E_X[X]$ and $E_X[Y]$ (and then swap in $Y = X + 1$)

$\endgroup$
3
$\begingroup$

I will add to Aleco's answer by mentioning a few different notations which are common in the literature of statistics and machine learning. The most common interpretation seems to be the second one in Aleco's answer, i.e. we use the marginal pmf/pdf of the mentioned variable, and sum/integrate over all possible values of the mentioned variable.

The only text I found which explicitly mentions this notation is the PRML book [1], which mentions:

Sometimes we will be considering expectations of functions of several variables, in which case we can use a subscript to indicate which variable is being averaged over, so that for instance

$$ E_{X} [h(X,Y)] $$ denotes the average of the function $H(X, Y)$ with respect to the distribution of X. Note that $E_{X} [h(X,Y)] $ will be a function of Y.

As I understand this:

  1. If X & Y are discrete: $$ g(Y) = E_{X} [h(X,Y)] = \sum_{x \in Range(X)} h(x, Y) p_{X}(x) $$

  2. If X & Y are continuous $$ g(Y) = E_{X} [h(X,Y)] = \int_{-\infty}^\infty h(x, Y) f_{X}(x) \, dx $$

Notice that both of these are functions of $Y$, since we are "averaging" over all possible values of $X$ and multiplying by their respective probabilities.


Another common subscript is $X \sim D$, which means the same thing. By using this subscript, the author is trying to tell us that the r.v. X is distributed according to some probability distribution $D$ (pmf or pdf), and that the distribution is an important part of the equation:

  1. If X & Y are discrete: $$ g(Y) = E_{X\sim D} [h(X,Y)] = \sum_{x \in Range(X)} h(x, Y) D(x) $$ where $D(x) = p_{X}(x)$ is the marginal pmf of X.

  2. If X & Y are continuous: $$ g(Y) = E_{X\sim D} [h(X,Y)] = \int_{-\infty}^\infty h(x, Y) D(x) \, dx $$ where $D(x) = f_{X}(x)$ is the marginal pdf of X.

[1] Pattern Recognition and Machine Learning, Bishop, pg 20 (equation 1.36)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.