12
$\begingroup$

Tests like Z, t, and several others assume that the data is based on a random sampling. Why?

Suppose that I'm doing experimental research, where I care much more for the internal validity than the external one. So, if my sample might be a little bit biased, okay, as I've accepted to not infer the hypothesis for the whole populations. And the grouping will still be random, i.e., I'll choose for convenience the sample participants, but I will randomly assign them to different groups.

Why can't I just ignore this assumption?

Improve this question
$\endgroup$
8
  • $\begingroup$ If the sampling technique introduces a bias, then it's not 'random'. If it does not introduce any bias then it is 'random' (for some definition of random;-). I've had sampling schemes that simply took every 7th sample to create a matched sample size to the counter sample. However I knew that there was no special aspect to that selection, so what may be thought of as a non-random sampling process was still effectively random. It's the same as selecting balls 1,2,3,4,5,6 on the lottery. It's just as random as any other sequence. $\endgroup$
    – Philip Oakley
    Commented Apr 11, 2018 at 12:42
  • 1
    $\begingroup$ @PhilipOakley: selecting balls 1,2,3,4,5,6 on the lottery gives you the same chance of winning as any other selection, but reduces your expected winnings as you are more likely to have to share the prize with others who had the same idea $\endgroup$
    – Henry
    Commented Apr 11, 2018 at 14:25
  • 1
    $\begingroup$ Systematic sampling, such as described by @Philip, often is analyzed as if it produced simple random samples, but it has pitfalls. For instance, if you were to measure a manufacturing process every day and sample every seventh measurement, you would be subject to confounding your results with a day-of-the-week effect, since (obviously) you would be sampling on the same day each week. You need to work harder to think of and address such subtleties when dealing with non-random samples. $\endgroup$
    – whuber ♦
    Commented Apr 11, 2018 at 17:48
  • 1
    $\begingroup$ @whuber, Absolutely. One must think hard (and widely) about these things!! In my case I had hours of video, with hundreds of event, with long gaps between, so needed to reduce the data size of the non-event set for a simple logistic regression (each frame considered independently, little change between frames), so dropping lots of non-event frames was reasonable. The time sequence aspect was considered separately. $\endgroup$
    – Philip Oakley
    Commented Apr 12, 2018 at 9:41
  • 1
    $\begingroup$ @Philip Interestingly, at almost the same time you were writing that comment about randomness not existing, the NIST issued a press release claiming it does. An account appears in today's (4 April 2018) issue of Nature. $\endgroup$
    – whuber ♦
    Commented Apr 12, 2018 at 13:30

3 Answers 3

Reset to default
18
$\begingroup$

If you are not making any inference for a wider group than your actual sample, then there is no application of statistical tests in the first place, and the question of "bias" does not arise. In this case you would just calculate descriptive statistics of your sample, which are known. Similarly, there is no question of model "validity" in this case - you are just observing variables and recording their values, and descriptions of aspects of those values.

Once you decide to go beyond your sample, to make inferences about some larger group, then you will need statistics and you will need to consider issues like sampling bias, etc. In this application, random sampling becomes a useful property to assist in getting reliable inferences of the wider group of interest. If you don't have random sampling (and you don't know the probabilities of your samples based on the population) then it becomes hard/impossible to make reliable inferences about the population.

Improve this answer
$\endgroup$
5
$\begingroup$

In real scientific research, it is quite rare to have data that came from true random sampling. The data are almost always convenience samples. This primarily affects what population you can generalize to. That said, even if they were a convenience sample, they did come from somewhere, you just need to be clear about where and the limitations that implies. If you really believe your data aren't representative of anything, then your study is not going to be worthwhile on any level, but that probably isn't true1. Thus, it is often reasonable to consider your samples as drawn from somewhere and to use these standard tests, at least in a hedged or qualified sense.

There is a different philosophy of testing, however, that argues we should move away from those assumptions and the tests that rely on them. Tukey was an advocate of this. Instead, most experimental research is considered (internally) valid because the study units (e.g., patients) were randomly assigned to the arms. Given this, you can use permutation tests, that mostly only assume the randomization was done correctly. The counterargument to worrying too much about this is that permutation tests will typically show the same thing as the corresponding classical tests, and are more work to perform. So again, standard tests may be acceptable.

1. For more along these lines, it may help to read my answer here: Identifying the population and samples in a study.

Improve this answer
$\endgroup$
4
$\begingroup$

Tests like Z, t, and several others are based on known sampling distributions of the relevant statistics. Those sampling distributions, as generally used, are defined for the statistic calculated from a random sample.

It may sometimes be possible to devise a relevant sampling distribution for non-random sampling, but in general it is probably not possible.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.