Flaw in all scientific studies/experiments?

Question

If you draw conclusions in your study based on reasoning like this paragraph below, it's automatically flawed, is it not?

"At the end, we compare the average energy levels of the two groups based on the observational study even if we find the difference between the average energy levels of these two groups of people, we can't attribute this difference solely to working out.

Because there may be other variables that we didn't control for in this study, that contribute to the observed difference. For example, people who are in better shape might be more likely to regularly work out and also have higher energy levels.

However, in the experiment, such variables that might also contribute to the outcome are likely equally represented in the two groups due to the random assignment. Therefore, if we find a difference between the two averages, we can indeed make a causal statement attributing this difference to working out. "

The paragraph says you can make a causal conclusion because of random assignment:

"such variables that might also contribute to the outcome are likely equally represented in the two groups due to the random assignment"

but that isn't guaranteed. Random means you could have, by chance, ended up selecting only participants, or a large majority of patients that have higher energy levels to start with..

So, is this just a really bad hypothetical example, or are all science experiments just as flawed as this one because they believe random assignment fixes everything?

I realize that the paragraph might need the rest of the context for anyone to really judge what I'm talking about, it's from

https://www.coursera.org/learn/probability-intro/lecture/Qw8iF/observational-studies-experiments

and, the entire study is described here:

In an observational study, researchers collect data in a way that does not directly interfere with how the data arise. In other words, they merely observe. And based on observational studies, we can only establish an association. In other words, correlation between the explanatory and the response variables. If an observational study uses data from the past, it's called a retrospective study. Whereas if data are collected throughout the study, it's called prospective. In an experiments on the other hand, researchers randomly assign subjects to treatments and can, therefore, establish causal connections between the explanatory and response variables.

Let's pause for a moment to clarify what we mean by random assignment with an example, suppose we want to evaluate the relationship between regularly working out and energy level. We can design this study as an observational study or an experiment. In an observational study, we sampled two types of people from the population. Those who choose to work out and those who don't, then find the average energy level for the two groups of people and compare. On the other hand, in an experiment, we sample a group of people from the population, then we randomly assign these people into two groups. Those who will regularly work out through the course of the stud and those who will not. The difference is that the decision of whether to work out or not is not left up to the subjects as in the observational study, but is instead imposed by the researcher.

At the end, we compare the average energy levels of the two groups based on the observational study even if we find the difference between the average energy levels of these two groups of people, we can't attribute this difference solely to working out. Because there may be other variables that we didn't control for in this study, that contribute to the observed difference. For example, people who are in better shape might be more likely to regularly work out and also have higher energy levels.

However, in the experiment, such variables that might also contribute to the outcome are likely equally represented in the two groups due to the random assignment. Therefore, if we find a difference between the two averages, we can indeed make a colossal statement attributing this difference to working out.

My entire point being:

Answer 1

The reasoning in the first text is not flawed. It is correct.

Theory of random assignment

If treatment is randomly assigned, any unobservables in the treatment group will be balanced with unobservables in the control group. The same effects from unobservables in the treatment group will also be present in the control group! The difference in effect size between the two groups will be an estimate of the treatment effect.

The key concept here is that you don't need to control for everything to produce consistent estimates of an effect. You need treatment to be orthogonal of confounding, unobservable variables.

Can spurious things happen with small populations?

With small groups, you may, by chance have an unbalanced assignment of unobservables. Randomly pick a group of 10 people and you might have 7 guys and 3 girls.

But if you randomly pick a group of 10,000, the split will be close to 50-50 (or whatever the split is in the population.) With larger populations, it becomes less and less likely that your control group differs from your treatment group by chance.

This may be a problem for small studies, but as they get larger, this is less of a concern.

The bigger problem?

In the social setting, you have many problems where random assignment may diverge from a mathematical ideal. For example, there may be selection bias.

Example: 100 kids are randomly accepted to a preschool program. Another 100 kids are randomly denied. It looks like random assignment of treatment (and a control group), but what if parents of the 100 kids that are denied INSTEAD find alternatives to the preschool program?

Denial of treatment causes kids to get unobserved, supplemental education! And unobservables aren't equally balanced between treatment and control. In some sense everybody got treatment! Your experiment then isn't comparing apples to no apples, it's comparing one kind of apple to another...

In the example in your question, I would think assigning people to work out may be quite hard! Does the treatment group actually work out? How do you prevent the control group from not working out?

Summary

Do additional experiments help? Yes. They may use better techniques, have different error etc... More knowledge is better.

Can we have an unrepresentative control and treatment group by chance? Yes. But with true random assignment, it becomes increasingly unlikely as $n$ increases. Flipping coins, 47 or fewer heads in a sample of 100 is quite likely. 4700 or fewer heads in a sample of 10000 is close to impossible.

Should I trust my intuition on probability? Probably not. People (me included) have awful intuition about probability. It's hard.

So all science is wonderful? No! There are so many ways things can be wrong. Assumptions of the statistics can be horribly violated in so many different. Courses, series of courses are devoted to experimental methods for good reason! In practice, random assignment doesn't work as nicely with people as it does in the laboratory because people are clever, smart and can respond in ways you didn't even imagine!