In the case of variables defined for a probability space with a finite sample space, a connection between the expected value $E[X|Y]$ and a linear projection in Euclidian space can be easily made.
A real valued variable is a function that maps the samples from a sample space to a real value $f:\Omega \to \mathbb{R}$. For example consider rolling two six sided dice. Then we have 36 possible outcomes $(d_1,d_2)$, and a variable maps those 36 outcomes to a value.
Examples of variables are:
- $f(d_1,d_2) = d_1$ the value of the first dice
- $f(d_1,d_2) = d_1+d_2$ the sum of the dice
- $f(d_1,d_2) = (d_1)^2$ the square of the first dice
- etc. any function giving a value to each of the 36 outcomes. This function/variable can be seen as a point in a 36 dimensional space, with each coordinate relating a value to a sample from the sample space.
Below is a table that may help to see these variables/functions as vectors in a 36 dimensional space. The table gives for each possible outcome/sample the corresponding mapped value of a variable. These columns are like row vectors of size 36, and each variable can be represented by such row vector.
| sample |
probability |
variable 1 |
variable 2 |
variable 3 |
etc. |
|
|
first dice |
sum of dice |
square of first dice |
any variable |
| (1,1) |
1/36 |
1 |
2 |
1 |
f(1,1) |
| (1,2) |
1/36 |
1 |
3 |
1 |
f(1,2) |
| (1,3) |
1/36 |
1 |
4 |
1 |
f(1,3) |
| (1,4) |
1/36 |
1 |
5 |
1 |
f(1,4) |
| (1,5) |
1/36 |
1 |
6 |
1 |
f(1,5) |
| (1,6) |
1/36 |
1 |
7 |
1 |
f(1,6) |
| (2,1) |
1/36 |
2 |
3 |
4 |
f(2,1) |
| (2,2) |
1/36 |
2 |
4 |
4 |
f(2,2) |
| (2,3) |
1/36 |
2 |
5 |
4 |
f(2,3) |
| (2,4) |
1/36 |
2 |
6 |
4 |
f(2,4) |
| (2,5) |
1/36 |
2 |
7 |
4 |
f(2,5) |
| (2,6) |
1/36 |
2 |
8 |
4 |
f(2,6) |
| (3,1) |
1/36 |
3 |
4 |
9 |
f(3,1) |
| (3,2) |
1/36 |
3 |
5 |
9 |
f(3,2) |
| (3,3) |
1/36 |
3 |
6 |
9 |
f(3,3) |
| (3,4) |
1/36 |
3 |
7 |
9 |
f(3,4) |
| (3,5) |
1/36 |
3 |
8 |
9 |
f(3,5) |
| (3,6) |
1/36 |
3 |
9 |
9 |
f(3,6) |
| (4,1) |
1/36 |
4 |
5 |
16 |
f(4,1) |
| (4,2) |
1/36 |
4 |
6 |
16 |
f(4,2) |
| (4,3) |
1/36 |
4 |
7 |
16 |
f(4,3) |
| (4,4) |
1/36 |
4 |
8 |
16 |
f(4,4) |
| (4,5) |
1/36 |
4 |
9 |
16 |
f(4,5) |
| (4,6) |
1/36 |
4 |
10 |
16 |
f(4,6) |
| (5,1) |
1/36 |
5 |
6 |
25 |
f(5,1) |
| (5,2) |
1/36 |
5 |
7 |
25 |
f(5,2) |
| (5,3) |
1/36 |
5 |
8 |
25 |
f(5,3) |
| (5,4) |
1/36 |
5 |
9 |
25 |
f(5,4) |
| (5,5) |
1/36 |
5 |
10 |
25 |
f(5,5) |
| (5,6) |
1/36 |
5 |
11 |
25 |
f(5,6) |
| (6,1) |
1/36 |
6 |
7 |
36 |
f(6,1) |
| (6,2) |
1/36 |
6 |
8 |
36 |
f(6,2) |
| (6,3) |
1/36 |
6 |
9 |
36 |
f(6,3) |
| (6,4) |
1/36 |
6 |
10 |
36 |
f(6,4) |
| (6,5) |
1/36 |
6 |
11 |
36 |
f(6,5) |
| (6,6) |
1/36 |
6 |
12 |
36 |
f(6,6) |
When we compute a variable $Z = E[X|Y] = g(Y)$ then the function/variable $X$ that is a vector in 36 dimensional space, becomes constraint to the possible values for the function $g(Y)$ which can be a smaller space.
The variable/function $Z = E[X|Y]$ is a constant for each sample that has the same value of $Y$. It is a vector in a space spanned by vectors that are constant for the values from $Y$. Say the variable $Y$ is the value of the first die roll, then the variable $Z$ is a sum of the following colums
[,1] [,2] [,3] [,4] [,5] [,6]
[1,] 1 0 0 0 0 0
[2,] 1 0 0 0 0 0
[3,] 1 0 0 0 0 0
[4,] 1 0 0 0 0 0
[5,] 1 0 0 0 0 0
[6,] 1 0 0 0 0 0
[7,] 0 1 0 0 0 0
[8,] 0 1 0 0 0 0
[9,] 0 1 0 0 0 0
[10,] 0 1 0 0 0 0
[11,] 0 1 0 0 0 0
[12,] 0 1 0 0 0 0
[13,] 0 0 1 0 0 0
[14,] 0 0 1 0 0 0
[15,] 0 0 1 0 0 0
[16,] 0 0 1 0 0 0
[17,] 0 0 1 0 0 0
[18,] 0 0 1 0 0 0
[19,] 0 0 0 1 0 0
[20,] 0 0 0 1 0 0
[21,] 0 0 0 1 0 0
[22,] 0 0 0 1 0 0
[23,] 0 0 0 1 0 0
[24,] 0 0 0 1 0 0
[25,] 0 0 0 0 1 0
[26,] 0 0 0 0 1 0
[27,] 0 0 0 0 1 0
[28,] 0 0 0 0 1 0
[29,] 0 0 0 0 1 0
[30,] 0 0 0 0 1 0
[31,] 0 0 0 0 0 1
[32,] 0 0 0 0 0 1
[33,] 0 0 0 0 0 1
[34,] 0 0 0 0 0 1
[35,] 0 0 0 0 0 1
[36,] 0 0 0 0 0 1
Below we show this graphically. The horizontal axis displays the 36 possible samples from the sample space, the vertical axis displays the value of the variable.
For the expectation the function is now constrained to a function that is constant for the same values of $Y$.
You can see this like a projection of the vector in 36 dimensional space onto the 6 dimensional space defined by the columns. Below is an R code that created the graphs and used explicitly a projection
$$E[X|Y] = A (A^TA)^{-1} A^T X$$
Here $A$ is a matrix with rows for each sample and columns for each possible value of $Y$ and the entries are equal to $A_{ij} = \mathbb{1}_{f(\omega_i) = y_j}$ it is equal to one if the $i$-th sample is equal to the $j$ possible value of the variable and it is zero otherwise. An example is the matrix above for the case that Y is the value of the first dice roll.
In the case of the dice roll example, each sample has the same probability mass. If this is not the case, then instead a weighted regression should be use, with the weights the probability of the sample.
$$E[X|Y] = A (A^TWA)^{-1} A^T W X$$
### one variable
Y = as.vector(outer(1:6,1:6, FUN = function(x,y) {y}))
### a other variable
X = as.vector(outer(1:6,1:6, FUN = "+"))
plot(Y, xlab = "samples", xaxt = "n",
pch = 21, bg = 1, main = "variable Y 'first dice roll'" )
for (i in 1:6) {
for (j in 1:6) {
axis(1,at = i+j*6-6, paste0("(",j,",",i,")"), las = 2, cex = 0.6)
}
}
plot(X, xlab = "samples", xaxt = "n",
pch = 21, bg = 1, main = "variable X 'sum of dice rolls' \n and variable Z = E[X|Y]")
for (i in 1:6) {
for (j in 1:6) {
axis(1,at = i+j*6-6, paste0("(",j,",",i,")"), las = 2, cex = 0.6)
}
}
### related linear projection matrix
A = sapply(1:6, function(x) {x == Y})
### projection giving conditional expectations
Z = A %*% solve(t(A) %*% A) %*% t(A) %*% X
points(Z, pch = 21, col = 1, bg = 0)
legend(1,12, c("variable X", "variable Z = E[X|Y]"), pch = 21, pt.bg = c(1,0))
