Given a multivariate random variable $\mathbf{X}=(X_1, ..., X_n)^\intercal : \Omega \rightarrow \mathbb{R}^n$ I want to determine the expectation value of this RV. Now wikipedia says the expectation is simply given by: $$\mathbb{E}[\mathbf{X}]=(\mathbb{E}[X_1], ..., \mathbb{E}[X_n])^\intercal.$$ However, the distribution of a random vector is the joint distribution of all its components, i.e., the distribution of $\mathbf{X}$ is $\mathbb{P}_{X_1, ..., X_n}$. Hence, for instance for $n=2$ if $X_1$ and $X_2$ are discrete the expected value would be computed like: \begin{align} \mathbb{E}[\mathbf{X}]&= \sum_{i=1}^\infty \mathbf{X}_i \mathbb{P}_{X_1,X_2}(x_1^{(i)}, x_2^{(i)})\\ &=\sum_{i=1}^\infty \binom{x_1^{(i)}}{x_2^{(i)}} \mathbb{P}_{X_1,X_2}(x_1^{(i)}, x_2^{(i)})\\ &=\binom{\sum_{i=1}^\infty x_1^{(i)} \mathbb{P}_{X_1,X_2}(x_1^{(i)}, x_2^{(i)})}{\sum_{i=1}^\infty x_2^{(i)} \mathbb{P}_{X_1,X_2}(x_1^{(i)}, x_2^{(i)})} \end{align} And this doesnt occur to me to be the same as $$\mathbb{E}[\mathbf{X}]=(\mathbb{E}[X_1], \mathbb{E}[X_2])^\intercal.$$ How do these two definitions match?
-
3$\begingroup$ Keep going with your derivation: write those sums in terms of the marginal distributions of $\mathbf X.$ $\endgroup$– whuber ♦Mar 4, 2022 at 14:16
-
$\begingroup$ So I wanted to do that but for marginalization I would require another sum that does not depend on the respective variable right? I.e., like this I cannot marginalize because I also always have to sum over the sample point $x_k^{(i)}$, k=1,2. $\endgroup$– guest1Mar 4, 2022 at 14:19
-
2$\begingroup$ I see no obstacle to marginalizing. I would invite you to tackle a tiny concrete case. How about making a table of the values and probabilities of such a distribution when both marginal variables are binary? Working through this (which should take only a minute or two) will reveal what happens generally. $\endgroup$– whuber ♦Mar 4, 2022 at 14:34
-
$\begingroup$ hmm but if they are not independent then I also have to define the conditional probabilites to get the joint distribution, no? $\endgroup$– guest1Mar 4, 2022 at 15:20
-
$\begingroup$ But by properly conditioning you can eliminate the nuisance component of the joint in the sum... $\endgroup$– Xi'anMar 4, 2022 at 15:43
1 Answer
It looks like you might be getting tripped up with the indexing and summations. Here's how to handle the 2d discrete example using the same approach you were trying to take. The same logic applies to higher dimensions and/or continuous variables (in that case, replace sums with integrals).
Notation
Let $x = \begin{bmatrix} x_1 \\ x_2 \end{bmatrix}$ be a realization of the random vector $X = \begin{bmatrix} X_1 \\ X_2 \end{bmatrix}$. I'll use the shorthand $p(x_1, x_2)$ for the joint probability $Pr(X_1=x_1, X_2=x_2)$, and $p(x_1)$ for the marginal probabillity $Pr(X_1=x_1)$. Sums will be taken over all possible values of each variable. For example, if $X_1$ has range $\mathcal{X}_1$ then $\sum_{x_1}$ is shorthand for $\sum_{x_1 \in \mathcal{X}_1}$.
Expectation
$$E[X] = \sum_x x p(x)$$
$$= \sum_{x_1} \sum_{x_2} \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} p(x_1, x_2)$$
$$= \sum_{x_1} \sum_{x_2} \begin{bmatrix} x_1 p(x_1, x_2) \\ x_2 p(x_1, x_2) \\ \end{bmatrix}$$
$$= \begin{bmatrix} \sum_{x_1} \sum_{x_2} x_1 p(x_1, x_2) \\ \sum_{x_1} \sum_{x_2} x_2 p(x_1, x_2) \\ \end{bmatrix}$$
$$= \begin{bmatrix} \sum_{x_1} \sum_{x_2} x_1 p(x_1, x_2) \\ \sum_{x_2} \sum_{x_1} x_2 p(x_1, x_2) \\ \end{bmatrix}$$
$$= \begin{bmatrix} \sum_{x_1} x_1 \sum_{x_2} p(x_1, x_2) \\ \sum_{x_2} x_2 \sum_{x_1} p(x_1, x_2) \\ \end{bmatrix}$$
$$= \begin{bmatrix} \sum_{x_1} x_1 p(x_1) \\ \sum_{x_2} x_2 p(x_2) \\ \end{bmatrix}$$
$$= \begin{bmatrix} E[X_1] \\ E[X_2] \\ \end{bmatrix}$$
-
$\begingroup$ Thanks for the answer, but why can you rewrite the sum over $x$ as two sums over $x_1$ and $x_2$. I mean there should be the same amount of $x_1$ as there are $x_2$ and this should be the same as the amount of vectors $x$ since the $x_1$ and $x_2$ are entries of the vector $x$, no? $\endgroup$– guest1Mar 7, 2022 at 16:47
-
2$\begingroup$ @guest1 Note that the sums over $x_1$ and $x_2$ are nested, not separate. For example, in the binary case, we need to take a sum $\sum_{x \in \mathcal{X}} \cdots$ over all possible vectors $\mathcal{X} = \big\{ [0,0], [0,1], [1,0], [1,1] \big\}$. This can be written as a nested sum over the individual components: $\sum_{x_1 \in \{0,1\}} \sum_{x_2 \in \{0,1\}} \cdots$. We have two possible values for $x_1$ and two possible values for $x_2$, so the nested sum has 4 terms--one for each possible vector. $\endgroup$ Mar 7, 2022 at 18:13
-
1$\begingroup$ @guest1 To be more specific, a nested sum over binary values would be written out as: $\sum_{x_1 \in \{0,1\}} \sum_{x_2 \in \{0,1\}} f(x_1, x_2) = f(0,0) + f(0,1) + f(1,0) + f(1,1)$ $\endgroup$ Mar 7, 2022 at 18:47