Why is randomness so useful?

Where by "why" I do not mean "list of use cases for randomness." If one has a quantitative question Q about topic X, it does not seem intuitive that values that by definition have absolutely nothing whatsoever to do with topic X would be so crucial in answering Q. And yet, a source of randomness is absolutely essential to most statistical operations. Probably the most common such example is the need to obtain random samples for inferential methods, however the need for randomness is so ubiquitous it seems almost fantastical. So, by "why" I am perhaps looking for something information-theoretical, or even possibly philosophical?

To create a context around Q and X, suppose a statistics student has an assignment to estimate the average height of students at the local college. They come to you and ask about where they should start. Naturally, one place to start involves collecting a random sample of students, so you say: "well, first, you'll need to obtain a Geiger counter." (with the intent that the student use it for generating random numbers). The student adopts a confused expression.

@Tim has suggested that requiring a Geiger counter for this task "at face value is rather ridiculous", which is precisely my point. Other options for obtaining randomness to help with the experiment include lottery balls, atmospheric noise, or repeatedly squaring your zip code: none of which have anything whatsoever to do with measuring heights. In fact, absent a priori knowledge about the population distribution of heights, involving students' heights in the method for obtaining your sample is probably a bad plan. An important part of estimating the average height of the students is finding an activity that is totally unrelated to the heights of students in any way, and obtaining measurements of that thing.

Is there some intuitive explanation to illustrate why a Geiger counter would be so immensely useful for someone interested in investigating the average height of students at the local college?

• I have answered some questions of this nature at stats.stackexchange.com/a/54894/919. – whuber May 18 at 16:19
• At least for me it isn’t. You seem to be misunderstanding what is meant by random variables in probability theory & statistics, but if that’s the case, it seems to be answered in the thread linked by @whuber. – Tim May 18 at 17:20
• Ok, but you do not connect the ridiculous claim anyhow with statistics. Nobody uses Geiger counter to measure human height in statistical research. I will not be continuing this discussion because it is already overly long. – Tim May 18 at 19:42
• I'm probably repeating everything already said, but we need randomness in order to not "leak" any external information to the collected sample. We can imagine the collection process as orthogonal (independent) to the data generating process, therefore it actually feels natural that various unrelated stuff could be used, as you mention, for the draws (as long as we agree that the outcomes are unpredictable & uniform). Cosmic noise works, ticket draws works, etc., although in practice we use a combination of pseudo-random numbers with cpu-collected-randomness. – runr May 18 at 21:21
• @Him If you want to measure the average height of students at the university, there is a quite simple way to do this without a Geiger counter (or without any random generator). Simply measure the height of every student at the school and take the average. The issue, of course, is that this is hard to do in most applications. You can try to take a sample without randomness, but many forms of bias can be introduced to your estimate. Simple random sampling simply allows us to use mathematical theorems (e.g., law of large numbers) which guarantee our results say what we want them to say. – knrumsey May 19 at 1:55

If you want to know the average height of students at a university, there is a simple way to do this without a Geiger counter or any other random generator. The solution is quite simple: record the height of every student at the university and compute the average height.

Of course, this is hard to do in practice because the population size is too large. The natural thing to do is to take a sample of your population and use the average of this sub-sample as a proxy. You can try to take a sample without using randomness, but your resulting answer is susceptible to several forms of bias. Here are two simple examples where you can obtain a biased answer.

1. Suppose you stand at the front door of the gymnasium and measure the height of students as they walk in. This is known as a convenience sample. Are you confident that the height of students that frequent the gymnasium are representative of the student body? What if the Basketball team is arriving for practice today?

2. Suppose you send an email to the entire student body and request that they fill out an optional survey recording their height. Not every student is going to respond: are you confident that those who do are representative of the population? It may be the case that taller men are more likely to respond. This is an example of voluntary response bias (albeit, not a very good one). In this scenario, you also have to deal with the fact that respondents may not be truthful, with a few subjects exaggerating their height.

By taking a simple random sample, we are able to avoid these (and many other) forms of bias. The Law of Large Numbers essentially guarantees that you can get as close to the true answer as you would like, by taking a large enough sample. But this result (at least the basic version) only holds under simple random sampling.

Okay, so I think the question you are asking relates a little bit to the causal inference literature and to RCTs (randomized control trials). I am going to change your example a little bit to a classic question studied by econometricians. What is the effect of college on wages. Now we might be tempted to run the regression,

$$wage_i = \beta\times college_i + \epsilon_i$$

And take $$\beta$$ to be our effect but this would be wrong. One reason is that there may be selection bias. In particular, it is not hard to imagine that choose to enter college do so because they believe it will increase their wage. That might be a problem because those students may have some intrinsic quality that makes them achieve higher wages. This is related to an endogeneity problem in which we may believe students who enter college may have higher ability and so they will also earn higher wages. Essentially, these issues confound our ability to infer what the effect of college on wages are. So we turn to randomness.

The idea behind randomness is that if we are able to randomly assign and force individuals to go to college (the treatment group) or not go to college (the control group) and then compare their wages we should then be able to extract an effect of college on wages. Why? Well essentially randomizing lets us average out the effect of ability because we imagine that both the treatment and control groups have the same distribution of people with ability since they were randomly chosen and no selection into treatment (in this case a college education occurred).

Formally we want to know the counterfactual effect of what would have happened if person $$i'$$ who went to college had not gone to college:

$$Y_{1i'}-Y_{0i'}$$

Where the 1 indicates going to college and the 0 indicates not going to college. This notation is called potential outcomes notation and I highly encourage you to check out some of the references at the bottom that go into more detail about it.

It turns out that under random assignment we can actually identify the distribution of $$Y_d$$ where I use $$d$$ to refer to the potential outcome (college or no college here). To see this consider,

$$\Pr[Y_d\leq y] = \Pr[Y_d\leq y|D=d]=\Pr[Y\leq y|D=d]$$

Where the first equality holds due to random assignment. Since the assignment is random conditioning on $$D=d$$ does not change $$Y_d$$. Thus we have identified the distribution of the potential outcomes as the conditional distribution of $$Y|D$$ for which we have data.

This is essentially the idea behind RCTs and this idea has been expanded on in many different cases. I encourage you to look up terms like: "Average Treatment Effect" and "Selection on Observables"

As you might have guessed we usually cannot force someone to go to college or not go to college. In these cases, another popular technique that can be used to "induce randomness" are instrumental variables. In particular, the "Local Average Treatment Effect" is a very popular way to apply the same idea as above in observational studies where we cannot randomly assign individuals.

I will leave you with some references:

Notes on Treatment effects: https://economics.mit.edu/files/32

Paper on LATE: https://www.jstor.org/stable/2951620