5
$\begingroup$

Can anyone explain the linearity of expectation in an intuitive way? I have been trying to understand this for far too long now. Please don't use any equations and such, try to use real world examples or simple tools such as couples, red cards black cards etc.

For example, if there are 4 red cards and 5 black cards, the expectation of the amount of alternating cards would be equal to $$E[\text{alternating first pair} + \text{alternating second pair} + \dotsm + \text{alternating 8th pair}]$$, even though the first pair and second pair should be dependent. Why? Pleaes don't link to external articles and such, I really would like an intuitive explanation, and it seems like one hasn't been provided before.

I have a feeling linearity holds for this kind of example, because the next outcome is symmetric for each case, I.E if first card is black, second is red, the third is black, its symmetric in a way to see the first card is red, second is black and third is red, but I'm not sure how this relates to linearity.

$\endgroup$
6
  • 6
    $\begingroup$ Linearity is a direct consequence of the basic properties of addition and multiplication of numbers. Any example of those properties would suffice, but is unlikely to be of any statistical interest! $\endgroup$
    – whuber
    Mar 31, 2023 at 13:49
  • $\begingroup$ I have a dice, the game is that I throw it... in expectation you will get $E[X] = 3.5$... now say that you win what you get and an additional 5 dollars, then $E[X+5] = E[X]+5=8.5$, because you can think "ok I will get 5 dollars... and then there is a stochastic game where there is some probability involved"... same holds for if I say that I'm going to give you double the amount you get... however, if you write the expectation as $E[f(x)] = \sum p(x)f(x)$, using the basic properties of summation, you can clearly see this $\endgroup$
    – Alberto
    Mar 31, 2023 at 13:54
  • 2
    $\begingroup$ What are alternating cards? $\endgroup$
    – Eli
    Mar 31, 2023 at 14:39
  • 2
    $\begingroup$ For anyone trying to answer this question, I'd like to highlight that there are scenarios in which linearity of expectation is (IMO) genuinely not intuitive: math.stackexchange.com/a/2282639 I'd find it interesting if some answers shed light on scenarios like this (other than just by saying "it follows directly from the definition"). $\endgroup$
    – helloworld
    Mar 31, 2023 at 23:25
  • $\begingroup$ Building on the comment of @helloworld , "expectation" is somewhat of a misnomer to the extent that it doesn't necessarily correspond to the (natural language meaning) "value that we would expect from the distribution". (That's why intro classes often also discuss the median and mode, because those are often better candidates for "value we'd expect".) I.e. "expectation" should be understood as a term of art, and intuition for it should be separate from intuition for "values that we'd expect". $\endgroup$ Apr 1, 2023 at 15:33

4 Answers 4

8
$\begingroup$

Linearity of expectation has everything to do with algebra. The concept is quite intuitive though because we often think in linear categories and we solve many linear equations in school. I am not sure if it is possible to answer your question without equations, but I'll try to make it intuitive though.

First of all, let's start with linearity. By "linear" we mean here the $f(x) = a + b x$ functions. To understand the following equations you only need to remember that the multiplication is distributive, so $cx + cy = c(x+y)$ and definitions of things like expected value, and joint distribution, etc.

Multiplication by a constant

For a random variable $X$ and a constant $c$, the following holds

$$ E[cX] = cE[X] $$

Example: According to some random source on the internet, the average height of women worldwide is 163 cm, this is the same as saying that the average height is 1.63 m, as 1 meter = 100 centimeters. Intuitively this is what we would expect, but to answer why does it hold, a little bit of algebra is needed.

First, recall that for a discrete random variable, the expected value is

$$ E[X] = \sum_x x\, p(x) $$

then we have

$$ \begin{align} E[cX] &= \sum_x c \, x \,p(x) \\ &= c \sum_x x \,p(x) \\ &= cE[X] \end{align} $$

Adding a constant

$$ E[X + c] = E[X] + c $$

Example: according to another random internet source, the average annual salary in the US is \$53,490. Now imagine that the US decides to introduce universal basic income and will give every citizen \$2,000 every month. How would the average income change? It would be \$53,490 + \$24,000 (12 months). It's also quite intuitive: every person would get \$24,000 extra money, so the total amount of money earned by US citizens would be their salaries + \$24,000 times the number of citizens. To get an average from it, divide the total by the number of citizens. Everyone got the same extra amount, so on average also everyone got that more money, hence by this amount the average income has changed.

$$ \begin{align} E[X + c] &= \sum_x (x + c) \, p(x) \\ &= \sum_x x \,p(x) + \sum_x c \, p(x) \\ &= \sum_x x \,p(x) + c \sum_x p(x) & \text{move } c \text{ outside of the summation}\\ &= \sum_x x \,p(x) + c & \text{because by definition } \sum_x p(x) = 1\\ &= E[X] + c \end{align} $$

In the universal basic income example, $\sum_x c$ would be equal to \$24,000 times the number of citizens, and $p(x) = 1/N$ where $N$ is the number of citizens. But as the math above shows, it would be the same if $p(x)$ would differ for every $x$.

Sum of two random variables

If $X$ and $Y$ are two random variables, then

$$ E[X + Y] = E[X] + E[Y] $$

Example: Let's say that you are interested what is your average consumption of salt and sugar (daily total in grams). You could calculate this by looking at your diet every day, for every meal checking how much sugar it contained, adding it to the amount of salt it contained, summing up to daily totals, and then looking at the average of those daily totals. Alternatively, you could create an Excel sheet where in one column you collect the amount of salt per meal, in another the amount of sugar, then calculate separate daily totals for consumed sugar and salt, calculate averages of those, and sum the two averages. They would be the same. The order of how you sum them should not matter.

$$ \begin{align} E[X + Y] &= \sum_x \sum_y (x + y) \, p(x, y) \\ &= \sum_x \sum_y x \, p(x, y) + \sum_x \sum_y y \, p(x, y) \\ &= \sum_x x \sum_y p(x, y) + \sum_y y \sum_x p(x, y) \\ &= \sum_x x \, p(x) + \sum_y y \, p(y) & \text{by the law of total probability} \\ &= E[X] + E[Y] \end{align} $$

where $p(x, y)$ is the joint distribution of $X$ and $Y$. This is possible by the law of total probability, which tells us that summing over all possible values gives us the marginal distribution $\sum_x p(x,y) = p(y)$.

Finally, keep in mind that the expected value is linear, but this is not the case for every statistic. For example, the median isn't.


The same results would hold for continuous variables if you replace things like $\sum_x x \, p(x)$ with $\int_x x \, p(x) \, dx$, because integrals follow similar rules in this case.

If the "every person" and "everyday" examples don't appeal to you because the $p(x)$ probabilities they use are uniform, consider that you could group the people (or days) in the examples into some groups and calculate things per group. In such a case, $p(x)$ would not be $1/N$ anymore, but rather $n_i/N$, where $n_i$ is the number of people in each group. Calculating things with such groups would be the same as with raw data if all the people within the group are the same.

$\endgroup$
3
  • $\begingroup$ I love how you used affine functions instead of linear functions and it works because constants are random variables who are their own expected value. $\endgroup$
    – Stef
    Apr 1, 2023 at 9:46
  • $\begingroup$ @Stef I think that considering a constant as a regular constant here makes things simpler. $\endgroup$
    – Tim
    Apr 1, 2023 at 9:51
  • $\begingroup$ "First of all, let's start with linearity. By "linear" we mean here the f(x)=a+bx functions." In this context, "linear" means zero-intercept (or homogeneously linear). What you gave is more precisely an affine function. Also, we are dealing with vectors, and using lowercase, normal type for f(x)=a+bx implies scalars. $E$ is a homogeneously linear function of the vector of observations. If $E(X) = a+bX$, then $E(X_1+X_2) = a+b(X_1+X_2)$, but since $E(X_1+X_2) =E(X_1)+E(X_2)$, we have $a+b(X_1+X_2)=E(X_1)+E(X_2)=a+bX_1+a+bX_2 \rightarrow a = 2a \rightarrow a= 0$. $\endgroup$ Apr 1, 2023 at 18:54
6
$\begingroup$

In general, the properties of the expected value for a discrete random variable arise directly from the properties of the (weighted) arithmetic mean, because the expected value of a random variable is defined precisely as the weighted arithmetic mean of the possible outcomes of that random variable, weighted by the probabilities of those outcomes.

The definition of the expected value for a continuous random variable is a natural extension of this principle, if you understand integrals well enough. The extension is particularly interesting in the "frequentist" conception of probability as long-run proportions of events under the hypothetical ability to re-run some data generating process an infinite number of times.

So the question here amounts to: why is linearity of the arithmetic mean intuitive?

Consider some real-world quantity with plenty of random variation, such as the body lengths of snow crabs in the Bering Sea. Brushing aside all mathematical rigor, it is reasonable to "expect" that the body length of any random crab should be around the average body length of that type of crab in that region. So if you were to magically double all the body lengths of all the crabs, then naturally the average body length should double (linearity!), and therefore that the "expected" body length doubles accordingly.

Similarly, you might be interested in the weights of luggage items being loaded onto an airplane, because you are interested in the total takeoff weight of the airplane. As with the crabs, if you double the weight of every item loaded on the plane, it is completely natural that you should double the average weight of those items, and therefore that you should double the "expected" weight of any one item.

Furthermore, the total weight of all items is just the sum of the individual item weights. Here we can turn to the frequentist interpretation of probability for intuition about expected values. Imagine that there is some complicated and unobservable (and therefore random from our perspective) data-generating process by which the weight of each luggage item is chosen. Consider the average outcome of this data generating process. The total weight in this average outcome is naturally the sum of the individual weights. Because this is just a sum, we can use the basic properties of the arithmetic mean and state that this average total weight should also be the sums of the individual luggage weight averages. Finally, by equivalence between arithmetic means and expected values, we should conclude that the expected value of the sum is the sum of the expected values.

To extrapolate this reasoning to categorical data, it might help to remember that we can think about categorical random variables as a collection of mutually-independent binary random variables. More abstractly, we can think of categorical data as vectors in barycentric space. In both cases, we have extended our scenario from univariate to multivariate, so the interpretation of the "expectation" must extend accordingly to mean the expected value of each element, and the intuition from above holds in that case.

$\endgroup$
4
$\begingroup$

Let's compute it with a smaller case such that we can write it out completely and look under the hood what happens.

Consider 2 red cards and 3 black cards. What are all possible cases?

$$\begin{array}{cccc} \text{combinations} & \text{alternating pairs} & \text{total alternations}\\ & 1 \quad 2 \quad 3 \quad 4 & \\\hline\\ \text{rrbbb} & 0 \quad 1 \quad 0 \quad 0 & 1\\ \text{rbrbb} & 1 \quad 1 \quad 1 \quad 0 & 3\\ \text{rbbrb} & 1 \quad 0 \quad 1 \quad 1 & 3\\ \text{rbbbr} & 1 \quad 0 \quad 0 \quad 1 & 2\\ \text{brrbb} & 1 \quad 0 \quad 1 \quad 0 & 2\\ \text{brbrb} & 1 \quad 1 \quad 1 \quad 1 & 4\\ \text{brbbr} & 1 \quad 1 \quad 0 \quad 1 & 3 \\ \text{bbrrb} & 0 \quad 1 \quad 0 \quad 1 & 2 \\ \text{bbrbr} & 0 \quad 1 \quad 1 \quad 1 & 3 \\ \text{bbbrr} & 0 \quad 0 \quad 1 \quad 0 & 1 \\ \hline \text{sums} & 6\quad 6 \quad 6 \quad 6 & 24 \end{array}$$

And the computation of the expectation goes like a sum where we multiply each case with it's frequency/probability, which is $\frac{1}{10}$ here

$$E[\text{alternations}] = \begin{array}{cr} (0+1+0+0) \times \frac{1}{10} &\\ (1+1+1+0) \times \frac{1}{10} &\\ (1+0+1+1) \times \frac{1}{10} &\\ (1+0+0+1) \times \frac{1}{10} &\\ (1+0+1+0) \times \frac{1}{10} &\\ (1+1+1+1) \times \frac{1}{10} &\\ (1+1+0+1) \times \frac{1}{10} &\\ (0+1+0+1) \times \frac{1}{10} &\\ (0+1+1+1) \times \frac{1}{10} &\\ (0+0+1+0) \times \frac{1}{10} & +\\ \hline 2.4 \end{array} = \begin{array}{cr} 0.1&\vphantom{ \frac{1}{10} }\\ 0.3&\vphantom{ \frac{1}{10} }\\ 0.3&\vphantom{ \frac{1}{10} }\\ 0.2&\vphantom{ \frac{1}{10} }\\ 0.2&\vphantom{ \frac{1}{10} }\\ 0.4&\vphantom{ \frac{1}{10} }\\ 0.3&\vphantom{ \frac{1}{10} }\\ 0.2&\vphantom{ \frac{1}{10} }\\ 0.3&\vphantom{ \frac{1}{10} }\\ 0.1&+\vphantom{ \frac{1}{10} }\\ \hline 2.4 \end{array}$$

But instead of computing all the terms in the row first we can also compute the columns first

$$E[\text{alternations}] = \begin{array}{cccccccccccr} &(&0&+&0.1&+&0&+&0&)&\\ &(&0.1&+&0.1&+&0.1&+&0&)&\\ &(&0.1&+&0&+&0.1&+&0.1&)&\\ &(&0.1&+&0&+&0&+&0.1&)&\\ &(&0.1&+&0&+&0.1&+&0&)&\\ &(&0.1&+&0.1&+&0.1&+&0.1&)&\\ &(&0.1&+&0.1&+&0&+&0.1&)&\\ &(&0&+&0.1&+&0&+&0.1&)&\\ &(&0&+&0.1&+&0.1&+&0.1&)&\\ &(&0&+&0&+&0.1&+&0&)&+\\ \hline &&&&&2.4 \end{array} = \begin{array}{cr} 0&\\ 0.1&\\ 0.1&\\ 0.1&\\ 0.1&\\ 0.1&\\ 0.1&\\ 0&\\ 0&\\ 0&\\ \hline 0.6 \end{array}+ \begin{array}{cr} 0.1&\\ 0.1&\\ 0&\\ 0&\\ 0&\\ 0.1&\\ 0.1&\\ 0.1&\\ 0.1&\\ 0&\\ \hline 0.6 \end{array}+ \begin{array}{cr} 0&\\ 0.1&\\ 0.1&\\ 0&\\ 0.1&\\ 0.1&\\ 0&\\ 0&\\ 0.1&\\ 0.1&\\ \hline 0.6 \end{array}+\begin{array}{cr} 0&\\ 0&\\ 0.1&\\ 0.1&\\ 0&\\ 0.1&\\ 0.1&\\ 0.1&\\ 0.1&\\ 0&\\ \hline 0.6 \end{array} \quad \begin{array}{cr} \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ = 2.4 \end{array} $$


The reason here is that we can change a product over a sum into a sum of products, like $(1+2+3)\times 4 = (1\times 4) + (2 \times 4) + (3 \times 4)$, and that makes that we get a matrix of multiple terms that we can compute either in rows first or columns first.

This is the distributive property of multiplication over summation.

$\endgroup$
2
  • $\begingroup$ +1: I would have just used your first table (adding that each row was equally likely) and said that the $24$ (which you are going to divide by $10$ to get the expectation) is of course both the sum of the final row and the sum of the final column $\endgroup$
    – Henry
    Apr 1, 2023 at 23:33
  • $\begingroup$ @Henry my idea/motivation about going a step further is because sometimes you get multiplications/division where each row is multiplied/divided by a different number depending on the probability of the case. (And yes, in the end I ended up not doing it with different divisions/multiplication because that would make it too complicated) $\endgroup$ Apr 2, 2023 at 9:29
4
$\begingroup$

Sticking to your example, here's an intuitive way to understand the linearity of expectation (at least to me):

Consider the $i$th-consecutive pair in the sequence of 9 cards of two different colors, 4 red and 5 black (see image below) and denote by $X_i$ the event that this pair consists of two cards of the same color, the expectation of $X_i$ by $\mathbf{E}[X_i]$, and also:

$$ X_i= \begin{cases} 1\text{ if i(th)-card and (i+1)th-card have the same color},\\ 0\text{ otherwise}. \end{cases} $$

alternating pairs of cards

The linearity of expectation says:

$$ \mathbf{E}[X_1+...+X_n]=\mathbf{E}[X_1] +...+ \mathbf{E}[X_n], \hspace{2em}n= 8\ \text{ in this case}.\tag{1}\label{eq:eq1} $$

Intuitively, the expectation of $X_i$ can be interpreted as the proportion of times the event $i$th-pair consisting of two cards with the same color happens when we perform a random assignment of 9 cards into 9 places (in the sequence) $N$ times ($N\gg 0$). Notice that this proportion is at most 1 and equal to the probability of $X_i$ (in the frequentist interpretation) and this can be shown to be true for all indicator random variable.

Summing up over all the expectations of $X_i, i=1,...,n$, we end up at the quantity on the right-hand side of the equation \eqref{eq:eq1} and it's the proportion of times that all the (consecutive) pairs in the sequence having two cards of the same color after performing $N$ times of random assignments. Since $N$ is large, this proportion can be interpreted as the expected (or average) number of all pairs in the sequence consisting of both cards with the same color. This is denoted by the quantity $\mathbf{E}[X_1+...+X_n]$ that appears on left-hand side of the above equation. So, this explains why \eqref{eq:eq1} holds.

For detailed calculation, Sextus Empiricus has provided a thorough answer.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.