The reasoning in the first text is not flawed. It is correct. ### Theory of random assignment If treatment is randomly assigned, any unobservables in the treatment group will be balanced with unobservables in the control group. The *same* effects from unobservables in the treatment group will also be present in the control group! The difference in effect size between the two groups will be an estimate of the treatment effect. The key concept here is that you *don't* need to control for *everything* to produce consistent estimates of an effect. You need treatment to be orthogonal of confounding, unobservable variables. ### Can spurious things happen with small populations? With *small* groups, you may, by chance have an unbalanced assignment of unobservables. Randomly pick a group of 10 people and you might have 7 guys and 3 girls. But if you *randomly* pick a group of 10,000, the split will be close to 50-50 (or whatever the split is in the population.) With larger populations, it becomes less and less likely that your control group differs from your treatment group by chance. This may be a problem for small studies, but as they get larger, this is less of a concern. ### The bigger problem? In the social setting, it is [selection bias][1]. Randomly assignment of treatment works quite well in a science laboratory, but it is much more difficult to accomplish in the social setting. Example: 100 kids are randomly accepted to a preschool program. 100 kids are randomly denied. It looks like random assignment, but what if parents of the 100 kids that are denied *INSTEAD* find alternatives to the preschool program? Denial of treatment causes kids to get unobserved, supplemental education! And unobservables aren't equally balanced between treatment and control. ### Summary Do additional experiments help? Yes. They may use better techniques, have different error etc... More knowledge is better. Can we have an unrepresentative control and treatment group by chance? Yes. But with true random assignment, it becomes increasingly unlikely as $n$ increases. Flipping coins, 47 or fewer heads in a sample of 100 is quite likely. 4700 or fewer heads in a sample of 10000 is close to impossible. Should I trust my intuition on probability? Probably not. People (me included) have awful intuition about probability. It's hard. So all science is wonderful? No! There are so many ways things can be wrong. Assumptions of the statistics can be horribly violated in so many different. Whole courses are devoted to experimental methods for good reason! In practice, random assignment doesn't work as nicely with people as it does in the laboratory *because people are clever, smart and can respond in ways you didn't even imagine!* [1]: https://en.wikipedia.org/wiki/Selection_bias