# Random sampling and independence in a real world problem

In the book "Introduction to Econometrics" by Stock and Watson It is used this example to illustrate the relation between random sampling and independence of the random variables :

My problem is that in this case I don't get why the random sampling should imply independence.

In a simpler experiment I could have an urn with $$N$$ balls inside so that each of these balls have a probability of $$\frac{1}{N}$$ to be drawn: if I draw a ball and then I insert the ball again inside the urn and repeat, the two draws are independent because the urn has still the same composition (or even if I don't insert the ball I can say they are approximately independent if $$N$$ is big enough) and the two extracted balls together are a random sample.

In the example above, instead, there is something different (at least it seems so to me). It is true that every element has the same probability to be extracted at each draw, but I don't see the link between this fact and the independence of the random variables. Why is this the case? In the simpler experiment I mentioned it was because I inserted the ball again in the urn and the composition of the urn was the same as before; but here, after I select at random the first day and I observe the commuting time I know something new because that day has a specific commuting time and not anymore just a cumulative distribution function which measures the probability of the commuting time so, when I insert that day again inside the "urn", the commuting time of that specific day is known so it is not the same as before. Can someone clarify please? Is the difference not important? Why?

• imagine somewhat of an extreme case, AR series $cov(x_t,x_{t-k})=f(k)$. Now pick the random times $\tau=[t_1,t_2,...]$. The new series are independent and uncorrelated $cov(x_\tau,x_{\tau-k})=\delta(k)$. if you can convince yourself that this is the case, then you got the idea why the statement is correct in the textbook Mar 29, 2021 at 19:43
• thank you @Aksakal, but I'm a beginner in econometrics, is there something more elementary to guide me? Mar 29, 2021 at 19:55
• random sampling destroys the dependency. the easiest way to think of it is as if you were shuffling the deck of cards. suppose they come in a perfect order, by color and rank, then you shuffle and there's no order. Mar 29, 2021 at 21:59

This extract from the text suffers from ambiguity and incorrectness.

Let's deal with the latter first. Independence of two random variables $$X$$ and $$Y$$ is not about one variable "providing no information about the first" (a remarkably ambiguous phrase in its own right!). Independence is strictly about probabilities and it means nothing more nor less than the chance of any joint event (namely, that the value of $$X$$ lies in some set $$\mathcal A$$ and the value of $$Y$$ simultaneously lies in some other set $$\mathcal B$$) is determined from the separate chances alone (namely, by multiplying them).

In this context it is natural to set up an urn model to understand the sampling. An extreme instance of this situation occurs with a truncated school year (as many have recently experienced!) in which the student commutes on just two days. The urn would contain two slips of paper representing the two commutes. On each slip is written the time of that commute. A random sample of size one is obtained by withdrawing a single slip blindly. Let $$X$$ be the value on that slip: it is a random variable. Let $$Y$$ be the collection of values on all remaining slips in the urn (namely, the commuting day that was not selected). It is straightforward to show that the random variable $$(X,Y)$$ is not independent: indeed, the correlation between $$X$$ and $$Y$$ is $$-1$$ and any variables with nonzero correlation are not independent.

If you find samples of size $$1$$ conceptually objectionable, extend this example to a school year with three commuting days and consider a random sample (without replacement) of size $$2.$$ This sample consists of withdrawing two tickets -- in order -- without replacement. Let $$X_1$$ be the value written on the first ticket and $$X_2$$ the value on the second. The correlation of the random variable $$(X_1,X_2)$$ is $$-1/2,$$ again nonzero: these two commuting times are not independent. (Question on Covariance for sampling without replacement explains how to calculate this covariance.)

It is possible the authors had in mind a model in which the urn is filled with gazillions of tickets reflecting some distribution of "hypothetical" commuting times. If so, the sample values will behave practically as if they were independent. But what would be the conceptual basis for constructing such a model?

The authors might also have (implicitly) been appealing to the idea that when there is a "large" number of tickets in the urn and "relatively few" are withdrawn for the sample, the values on the sampled tickets are approximately independent. But that sounds just too qualitative and slippery to serve as a decent explanation for any audience.

The more we think about this situation, the more reality intrudes. For instance, even when a school year comprises a full 180 (or so) days, why should we suppose the commuting times sampled during winter months "provide no information" about other nearby commuting times? In regions with serious winter weather nobody would believe this. "I see it took you two hours to get to school yesterday. Must be a lot of snow out there. I bet your ride during the next week is going to be extra long."

We have already glossed over several ambiguities concerning what is meant by "no information" and what model is in use. There are other ambiguities. For the purposes of evaluating independence of values in the sample, should we -- or should we not -- suppose we might inspect the full contents of the urn? If one commuting time "provides no information" about any other commuting time in the sample, then how much less information must it provide about commuting times that weren't sampled! How, then, could it be possible to make any inferences at all about the year's commuting times based on the sampled values?

Although it might seem painful or excessively technical to do so, the only way to demonstrate independence of random variables must appeal to its probabilistic definition. That requires clearly indicating a probability model and showing that the probabilities in that model obey the product law that is characteristic of independence. Anything else is just hand-waving and threatens to confuse the thoughtful student.

• thansk @whuber, I really appreciate. Can I ask you if there's a book on econometrics you could suggest (or something related to the main topics treated in this area)? Mar 29, 2021 at 23:08
• @Tortar: The Stock and Watson book should be fine for the most part, but you need to supplement this with some other statistical work, particularly for parts like this that are wrong. If you want to know more on this specific topic, you can look at books and papers on sampling theory.
– Ben
Mar 30, 2021 at 0:51
• thanks @Ben ! anyway I appreciate also your answer. Could you suggest me an entry level book on sampling theory ? Mar 30, 2021 at 1:28
• "Let's deal with the latter first. Independence of two random variables X and Y is not about one variable "providing no information about the first" (a remarkably ambiguous phrase in its own right!)." I don't see how it's ambiguous. It means that knowing the value of X doesn't make any values of Y any more or less likely. Which is what independence means. Mar 30, 2021 at 5:57
• Let me see if I understand when you say "it is possible the authors had in mind a model in which the urn is filled with gazillions of tickets..." and that means that the value in the sample are independent, but that this model is not good to represent reality , does this mean that when I randomly select in such a model the days in my sample, they are unrelated by definition in some sense because than I pick from the urn a commuting time for that day and because the number of them in the urn is large nothing changes inside it? Mar 30, 2021 at 14:55

### This part of the book is wrong

Your misgivings on this matter are appropriate, because this part of the book is wrong. Even when making inferences about infinite populations, random sampling does not give independent random variables --- it gives conditionally independent random variables, conditional on the empirical distribution of the underlying superpopulation from which they were sampled. Once we reduce to sampling from a finite population, the sampled values are not conditionally independent given the underlying distribution of the population. In any case, it is not true that knowledge of the commuting time on one day gives no information about the commuting times on other days. Indeed, this is obviously not true, because if it were true then there would be no basis for making inferences from random samples at all.

If you would like to understand more about this topic, you can find a full elaboration (looking at both classical and Bayesian statistics) in O'Neill (2009). I'll give a short outline of the issue here, looking first at inferences in an infinite superpopulation. Consider a sequence of random variables $$X_1,X_2,X_3,...$$ with limiting empirical distribution $$F_\infty$$. De Finetti's representation theorem (as extended by Hewitt and Savage) shows that if the sequence is exchangeable (so that the first $$n$$ values are a simple random sample) then we have:

$$X_1,X_2,X_3,... | F_\infty \sim \text{IID } F_\infty.$$

This result means that random sampling from an infinite superpopulation gives values that are conditionally independent, conditional on the limiting empirical distribution. Once we condition on a finite population, things become even trickier. We can still say that the values are conditionally independent, conditional on $$F_\infty$$ (i.e., conditional on the distribution of an infinite superpopulation where the finite population is embedded), but this is not much use. Suppse we let $$F_N$$ denote the empirical distribution of the first $$N$$ values (a finite population), and we look at behaviour conditional on this distribution. Each sample value has this marginal distribution, but we cannot validly assert even conditional independence:

$$\quad \quad \quad \ \ X_i |F_N \sim F_N \quad \quad \quad \quad \quad \ \text{(valid result)}$$

$$X_1,X_2,...,X_N | F_N \sim \text{IID } F_N. \quad \quad \quad \text{(erroneous result)}$$

In fact, for any population values $$i \neq k$$ we have the conditional distribution:

$$X_i|X_k, F_N \sim F_{N,-k}$$

where $$F_{N,-k}$$ is the empirical distribution of the population, excluding the $$k$$th value (see below for details). Except in the case where all population values are the same, we have $$F_N \neq F_{N,-k}$$, which means that the values $$X_i$$ and $$X_k$$ are not independent. Your remarks about urn models illustrate exactly this.

Empirical distributions: The population empirical distribution $$F_N: \mathbb{R} \rightarrow [0,1]$$ is defined by:

$$F_N(x) = \frac{1}{N} \sum_{i=1}^N \mathbb{I}(x_i \leqslant x).$$

Excluding the $$k$$th value gives the corresponding empirical distribution $$F_{N,-k}: \mathbb{R} \rightarrow [0,1]$$, defined by:

$$F_{N,-k}(x) = \frac{1}{N-1} \sum_{i=1}^N \mathbb{I}(i \neq k) \cdot \mathbb{I}(x_i \leqslant x).$$

Since $$(N-1) F_{N,-k}(x) + \mathbb{I}(x_k \leqslant x) = N F_N(x)$$, we can write the latter in terms of the former as:

$$F_{N,-k}(x) = \frac{N}{N-1} \cdot (F_N(x) - \mathbb{I}(x_k \leqslant x)).$$

• I was used to thinking that two rolls of a die are unconditionally independent in the frequentist sense that $P(X_1=i,X_2=j)=P(X_1=i)P(X_2=j)$ for all pairs $(i,j)$ with $i,j\in\{1,2,3,4,5,6\}$, but now I am confused. Would you say they are only conditionally independent when conditioned on the empirical distribution of the underlying infinite superpopulation generated by repeated "independent" (in the nontechnical sense) throws of a die (or something along these lines)? Could you offer an example of unconditional independence? Thank you! Mar 30, 2021 at 11:23
• Yes, that is correct. The die-rolls sould be conditionally independent conditional on the underlying outcome probabilities (which is the form of the IID model). If you are willing to say that the outcome probabilities are known constants (e.g., known fair die) then you also get marginal independence. However, if the outcome probabilities are unknown then the die rolls are no longer marginally independent (and they will generally be positively correlated).
– Ben
Mar 30, 2021 at 22:12
• This issue is discussed in O'Neill (2009), which looks at conditional and marginal dependence in exchangeable sequences of random variables. Associated results for statistical dependence in coin-flipping are in O'Neill and Puza (2005); O'Neill (2012); O'Neill (2015).
– Ben
Mar 30, 2021 at 22:14
• Note that in the frequentist context it is a bit tricky, since there is no distinction made between marginal and conditional probability conditional on the parameters --- since they are taken as "unknown constants". I would say that this means we are always implicitly conditioning on them though.
– Ben
Mar 30, 2021 at 22:15
• Thank you! I think your last comment gets to the core of the issue. I do find your formulation unsual, though. It is my impression that a typical frequentist textbook or paper would not say dice are conditionally independent. Your formulation (I guess inspired by the Bayesian notion of independence) may be more intuitive, but I would rather respect the traditional frequentist formulation as it makes sense in the frequentist paradigm. But I am out of my depth here. Thanks again. Mar 31, 2021 at 6:04

I think you're overthinking it. When we conceptualize a study, and its sampling frame, we consider looking retrospectively at the outcomes, measured and unmeasured, from the study's completion. As noted in the example, the student plans to randomly sample days from the school year. So we can consider each of the year's 52*5 school days, and its associated commute time, as being one of the proverbial balls in the urn.

• So it is not like I look at every outcome one by one, right? I look at all outcome at the same time. However, shouldn't the results in one case or the other being the same? Mar 29, 2021 at 19:39
• @Tortar I think you are asking: in what sense is there any variability or is there any population to make inference on when the set that we draw from is both fixed and ordered? As if the urn in question were actually a trough with all balls lying in order? Hence if I randomly sample the 1st, 14th, and 300th ball, I get the same results every time? Mar 29, 2021 at 20:01
• I would say something like this : if I want to imagine the example as a urn I would say it is a urn with 52*5 balls as you said , but I don't know what are the numbers of the balls because they follow the cumulative distribution , after I draw, say, the 14th ball I know what number is on it so the "randomness" in the urn when I reinsert the ball is decreasing in respect to the first draw . This confuses me a lot, is this no sense? Mar 29, 2021 at 20:16
• @Tortar you are inventing dependence. Has no relationship to the example. Mar 29, 2021 at 20:29