Provide an intuitive example of the linearity of expectation

Question

Can anyone explain the linearity of expectation in an intuitive way? I have been trying to understand this for far too long now. Please don't use any equations and such, try to use real world examples or simple tools such as couples, red cards black cards etc.

For example, if there are 4 red cards and 5 black cards, the expectation of the amount of alternating cards would be equal to $$E[\text{alternating first pair} + \text{alternating second pair} + \dotsm + \text{alternating 8th pair}]$$, even though the first pair and second pair should be dependent. Why? Pleaes don't link to external articles and such, I really would like an intuitive explanation, and it seems like one hasn't been provided before.

I have a feeling linearity holds for this kind of example, because the next outcome is symmetric for each case, I.E if first card is black, second is red, the third is black, its symmetric in a way to see the first card is red, second is black and third is red, but I'm not sure how this relates to linearity.

Answer 1

Linearity of expectation has everything to do with algebra. The concept is quite intuitive though because we often think in linear categories and we solve many linear equations in school. I am not sure if it is possible to answer your question without equations, but I'll try to make it intuitive though.

First of all, let's start with linearity. By "linear" we mean here the $f(x) = a + b x$ functions. To understand the following equations you only need to remember that the multiplication is distributive, so $cx + cy = c(x+y)$ and definitions of things like expected value, and joint distribution, etc.

Multiplication by a constant

For a random variable $X$ and a constant $c$, the following holds

$$ E[cX] = cE[X] $$

Example: According to some random source on the internet, the average height of women worldwide is 163 cm, this is the same as saying that the average height is 1.63 m, as 1 meter = 100 centimeters. Intuitively this is what we would expect, but to answer why does it hold, a little bit of algebra is needed.

First, recall that for a discrete^† random variable, the expected value is

$$ E[X] = \sum_x x\, p(x) $$

then we have

$$ \begin{align} E[cX] &= \sum_x c \, x \,p(x) \\ &= c \sum_x x \,p(x) \\ &= cE[X] \end{align} $$

Adding a constant

$$ E[X + c] = E[X] + c $$

Example: according to another random internet source, the average annual salary in the US is \$53,490. Now imagine that the US decides to introduce universal basic income and will give every citizen \$2,000 every month. How would the average income change? It would be \$53,490 + \$24,000 (12 months). It's also quite intuitive: every person would get \$24,000 extra money, so the total amount of money earned by US citizens would be their salaries + \$24,000 times the number of citizens. To get an average from it, divide the total by the number of citizens. Everyone got the same extra amount, so on average also everyone got that more money, hence by this amount the average income has changed.

$$ \begin{align} E[X + c] &= \sum_x (x + c) \, p(x) \\ &= \sum_x x \,p(x) + \sum_x c \, p(x) \\ &= \sum_x x \,p(x) + c \sum_x p(x) & \text{move } c \text{ outside of the summation}\\ &= \sum_x x \,p(x) + c & \text{because by definition } \sum_x p(x) = 1\\ &= E[X] + c \end{align} $$

In the universal basic income example, $\sum_x c$ would be equal to \$24,000 times the number of citizens, and $p(x) = 1/N$ where $N$ is the number of citizens. But as the math above shows, it would be the same if $p(x)$ would differ for every $x$^‡.

Sum of two random variables

If $X$ and $Y$ are two random variables, then

$$ E[X + Y] = E[X] + E[Y] $$

Example: Let's say that you are interested what is your average consumption of salt and sugar (daily total in grams). You could calculate this by looking at your diet every day, for every meal checking how much sugar it contained, adding it to the amount of salt it contained, summing up to daily totals, and then looking at the average of those daily totals. Alternatively, you could create an Excel sheet where in one column you collect the amount of salt per meal, in another the amount of sugar, then calculate separate daily totals for consumed sugar and salt, calculate averages of those, and sum the two averages. They would be the same. The order of how you sum them should not matter.

$$ \begin{align} E[X + Y] &= \sum_x \sum_y (x + y) \, p(x, y) \\ &= \sum_x \sum_y x \, p(x, y) + \sum_x \sum_y y \, p(x, y) \\ &= \sum_x x \sum_y p(x, y) + \sum_y y \sum_x p(x, y) \\ &= \sum_x x \, p(x) + \sum_y y \, p(y) & \text{by the law of total probability} \\ &= E[X] + E[Y] \end{align} $$

where $p(x, y)$ is the joint distribution of $X$ and $Y$. This is possible by the law of total probability, which tells us that summing over all possible values gives us the marginal distribution $\sum_x p(x,y) = p(y)$.

Finally, keep in mind that the expected value is linear, but this is not the case for every statistic. For example, the median isn't.

---

^† The same results would hold for continuous variables if you replace things like $\sum_x x \, p(x)$ with $\int_x x \, p(x) \, dx$, because integrals follow similar rules in this case.

^‡ If the "every person" and "everyday" examples don't appeal to you because the $p(x)$ probabilities they use are uniform, consider that you could group the people (or days) in the examples into some groups and calculate things per group. In such a case, $p(x)$ would not be $1/N$ anymore, but rather $n_i/N$, where $n_i$ is the number of people in each group. Calculating things with such groups would be the same as with raw data if all the people within the group are the same.

Answer 2

In general, the properties of the expected value for a discrete random variable arise directly from the properties of the (weighted) arithmetic mean, because the expected value of a random variable is defined precisely as the weighted arithmetic mean of the possible outcomes of that random variable, weighted by the probabilities of those outcomes.

The definition of the expected value for a continuous random variable is a natural extension of this principle, if you understand integrals well enough. The extension is particularly interesting in the "frequentist" conception of probability as long-run proportions of events under the hypothetical ability to re-run some data generating process an infinite number of times.

So the question here amounts to: why is linearity of the arithmetic mean intuitive?

Consider some real-world quantity with plenty of random variation, such as the body lengths of snow crabs in the Bering Sea. Brushing aside all mathematical rigor, it is reasonable to "expect" that the body length of any random crab should be around the average body length of that type of crab in that region. So if you were to magically double all the body lengths of all the crabs, then naturally the average body length should double (linearity!), and therefore that the "expected" body length doubles accordingly.

Similarly, you might be interested in the weights of luggage items being loaded onto an airplane, because you are interested in the total takeoff weight of the airplane. As with the crabs, if you double the weight of every item loaded on the plane, it is completely natural that you should double the average weight of those items, and therefore that you should double the "expected" weight of any one item.

Furthermore, the total weight of all items is just the sum of the individual item weights. Here we can turn to the frequentist interpretation of probability for intuition about expected values. Imagine that there is some complicated and unobservable (and therefore random from our perspective) data-generating process by which the weight of each luggage item is chosen. Consider the average outcome of this data generating process. The total weight in this average outcome is naturally the sum of the individual weights. Because this is just a sum, we can use the basic properties of the arithmetic mean and state that this average total weight should also be the sums of the individual luggage weight averages. Finally, by equivalence between arithmetic means and expected values, we should conclude that the expected value of the sum is the sum of the expected values.

To extrapolate this reasoning to categorical data, it might help to remember that we can think about categorical random variables as a collection of mutually-independent binary random variables. More abstractly, we can think of categorical data as vectors in barycentric space. In both cases, we have extended our scenario from univariate to multivariate, so the interpretation of the "expectation" must extend accordingly to mean the expected value of each element, and the intuition from above holds in that case.

Answer 3

Let's compute it with a smaller case such that we can write it out completely and look under the hood what happens.

Consider 2 red cards and 3 black cards. What are all possible cases?

$$\begin{array}{cccc} \text{combinations} & \text{alternating pairs} & \text{total alternations}\\ & 1 \quad 2 \quad 3 \quad 4 & \\\hline\\ \text{rrbbb} & 0 \quad 1 \quad 0 \quad 0 & 1\\ \text{rbrbb} & 1 \quad 1 \quad 1 \quad 0 & 3\\ \text{rbbrb} & 1 \quad 0 \quad 1 \quad 1 & 3\\ \text{rbbbr} & 1 \quad 0 \quad 0 \quad 1 & 2\\ \text{brrbb} & 1 \quad 0 \quad 1 \quad 0 & 2\\ \text{brbrb} & 1 \quad 1 \quad 1 \quad 1 & 4\\ \text{brbbr} & 1 \quad 1 \quad 0 \quad 1 & 3 \\ \text{bbrrb} & 0 \quad 1 \quad 0 \quad 1 & 2 \\ \text{bbrbr} & 0 \quad 1 \quad 1 \quad 1 & 3 \\ \text{bbbrr} & 0 \quad 0 \quad 1 \quad 0 & 1 \\ \hline \text{sums} & 6\quad 6 \quad 6 \quad 6 & 24 \end{array}$$

And the computation of the expectation goes like a sum where we multiply each case with it's frequency/probability, which is $\frac{1}{10}$ here

$$E[\text{alternations}] = \begin{array}{cr} (0+1+0+0) \times \frac{1}{10} &\\ (1+1+1+0) \times \frac{1}{10} &\\ (1+0+1+1) \times \frac{1}{10} &\\ (1+0+0+1) \times \frac{1}{10} &\\ (1+0+1+0) \times \frac{1}{10} &\\ (1+1+1+1) \times \frac{1}{10} &\\ (1+1+0+1) \times \frac{1}{10} &\\ (0+1+0+1) \times \frac{1}{10} &\\ (0+1+1+1) \times \frac{1}{10} &\\ (0+0+1+0) \times \frac{1}{10} & +\\ \hline 2.4 \end{array} = \begin{array}{cr} 0.1&\vphantom{ \frac{1}{10} }\\ 0.3&\vphantom{ \frac{1}{10} }\\ 0.3&\vphantom{ \frac{1}{10} }\\ 0.2&\vphantom{ \frac{1}{10} }\\ 0.2&\vphantom{ \frac{1}{10} }\\ 0.4&\vphantom{ \frac{1}{10} }\\ 0.3&\vphantom{ \frac{1}{10} }\\ 0.2&\vphantom{ \frac{1}{10} }\\ 0.3&\vphantom{ \frac{1}{10} }\\ 0.1&+\vphantom{ \frac{1}{10} }\\ \hline 2.4 \end{array}$$

But instead of computing all the terms in the row first we can also compute the columns first

$$E[\text{alternations}] = \begin{array}{cccccccccccr} &(&0&+&0.1&+&0&+&0&)&\\ &(&0.1&+&0.1&+&0.1&+&0&)&\\ &(&0.1&+&0&+&0.1&+&0.1&)&\\ &(&0.1&+&0&+&0&+&0.1&)&\\ &(&0.1&+&0&+&0.1&+&0&)&\\ &(&0.1&+&0.1&+&0.1&+&0.1&)&\\ &(&0.1&+&0.1&+&0&+&0.1&)&\\ &(&0&+&0.1&+&0&+&0.1&)&\\ &(&0&+&0.1&+&0.1&+&0.1&)&\\ &(&0&+&0&+&0.1&+&0&)&+\\ \hline &&&&&2.4 \end{array} = \begin{array}{cr} 0&\\ 0.1&\\ 0.1&\\ 0.1&\\ 0.1&\\ 0.1&\\ 0.1&\\ 0&\\ 0&\\ 0&\\ \hline 0.6 \end{array}+ \begin{array}{cr} 0.1&\\ 0.1&\\ 0&\\ 0&\\ 0&\\ 0.1&\\ 0.1&\\ 0.1&\\ 0.1&\\ 0&\\ \hline 0.6 \end{array}+ \begin{array}{cr} 0&\\ 0.1&\\ 0.1&\\ 0&\\ 0.1&\\ 0.1&\\ 0&\\ 0&\\ 0.1&\\ 0.1&\\ \hline 0.6 \end{array}+\begin{array}{cr} 0&\\ 0&\\ 0.1&\\ 0.1&\\ 0&\\ 0.1&\\ 0.1&\\ 0.1&\\ 0.1&\\ 0&\\ \hline 0.6 \end{array} \quad \begin{array}{cr} \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ = 2.4 \end{array} $$

---

The reason here is that we can change a product over a sum into a sum of products, like $(1+2+3)\times 4 = (1\times 4) + (2 \times 4) + (3 \times 4)$, and that makes that we get a matrix of multiple terms that we can compute either in rows first or columns first.

This is the distributive property of multiplication over summation.

Answer 4

Sticking to your example, here's an intuitive way to understand the linearity of expectation (at least to me):

Consider the $i$th-consecutive pair in the sequence of 9 cards of two different colors, 4 red and 5 black (see image below) and denote by $X_i$ the event that this pair consists of two cards of the same color, the expectation of $X_i$ by $\mathbf{E}[X_i]$, and also:

$$ X_i= \begin{cases} 1\text{ if i(th)-card and (i+1)th-card have the same color},\\ 0\text{ otherwise}. \end{cases} $$

The linearity of expectation says:

$$ \mathbf{E}[X_1+...+X_n]=\mathbf{E}[X_1] +...+ \mathbf{E}[X_n], \hspace{2em}n= 8\ \text{ in this case}.\tag{1}\label{eq:eq1} $$

Intuitively, the expectation of $X_i$ can be interpreted as the proportion of times the event $i$th-pair consisting of two cards with the same color happens when we perform a random assignment of 9 cards into 9 places (in the sequence) $N$ times ($N\gg 0$). Notice that this proportion is at most 1 and equal to the probability of $X_i$ (in the frequentist interpretation) and this can be shown to be true for all indicator random variable.

Summing up over all the expectations of $X_i, i=1,...,n$, we end up at the quantity on the right-hand side of the equation \eqref{eq:eq1} and it's the proportion of times that all the (consecutive) pairs in the sequence having two cards of the same color after performing $N$ times of random assignments. Since $N$ is large, this proportion can be interpreted as the expected (or average) number of all pairs in the sequence consisting of both cards with the same color. This is denoted by the quantity $\mathbf{E}[X_1+...+X_n]$ that appears on left-hand side of the above equation. So, this explains why \eqref{eq:eq1} holds.

For detailed calculation, Sextus Empiricus has provided a thorough answer.