Many clinical studies are based on non-random samples. However, most standard tests (e.g. t-tests, ANOVA, linear regression, logistic regression) are based on the assumption that samples contain "random numbers". Are results valid if these non-random samples were analyzed by standard tests? Thank you.
$\begingroup$
$\endgroup$
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
2
There are two general models to testing. The first one, based on the assumption of random sampling from a population, is usually called the "population model".
For example, for the two-independent samples t-test, we assume that the two groups we want to compare are random samples from the respective populations. Assuming that the distributions of the scores within the two groups are normally distributed in the population, we can then derive analytically the sampling distribution of the test statistic (i.e., for the t-statistic). The idea is that if we were to repeat this process (randomly drawing two samples from the respective populations) an infinite number of times (of course, we do not actually do that), we would obtain this sampling distribution for the test statistic.
An alternative model for testing is the "randomization model". Here, we do not have to appeal to random sampling. Instead, we obtain a randomization distribution through permutations of our samples.
For example, for the t-test, you have your two samples (not necessarily obtained via random sampling). Now if indeed there is no difference between these two groups, then whether a particular person actually "belongs" to group 1 or group 2 is arbitrary. So, what we can do is to permute the group assignment over and over, each time noting how far the means of the two groups are apart. This way, we obtain a sampling distribution empirically. We can then compare how far the two means are apart in the original samples (before we started to reshuffle the group memberships) and if that difference is "extreme" (i.e., falls into the tails of empirically derived sampling distribution), then we conclude that group membership is not arbitrary and there is indeed a difference between the two groups.
In many situations, the two approaches actually lead to the same conclusion. In a way, the approach based on the population model can be seen as an approximation to the randomization test. Interestingly, Fisher was the one who proposed the randomization model and suggested that it should be the basis for our inferences (since most samples are not obtained via random sampling).
A nice article describing the difference between the two approaches is:
Ernst, M. D. (2004). Permutation methods: A basis for exact inference. Statistical Science, 19(4), 676-685 (link).
Another article that provides a nice summary and suggest that the randomization approach should be the basis for our inferences:
Ludbrook, J., & Dudley, H. (1998). Why permutation tests are superior to t and F tests in biomedical research. American Statistician, 52(2), 127-132 (link).
EDIT: I should also add that it is common to calculate the same test statistic when using the randomization approach as under the population model. So, for example, for testing the difference in means between two groups, one would calculate the usual t-statistic for all possible permutations of the group memberships (yielding the empirically derived sampling distribution under the null hypothesis) and then one would check how extreme the t-statistic for the original group membership is under that distribution.
-
$\begingroup$ Besides the inferential errors that abound in the interpretation of p-values, the probabilistic pre-conditions (i.e. random sampling or equivalent) for using them at all are not often met by observational studies in the social sciences. Can p-values be meaningfully interpreted without random sampling? $\endgroup$– KuJCommented Nov 16, 2022 at 3:25
-
$\begingroup$ A related discussion: Random sampling versus random allocation/randomization- implications for p-value interpretation $\endgroup$– KuJCommented Nov 16, 2022 at 3:34
$\begingroup$
$\endgroup$
Your question is very good, but it doesn't have a straightforward answer.
Most tests like those you mention are based on the assumption that a sample is a random sample, because a random sample is likely to be representative of the sampled population. If the assumption is invalid then any interpretation of the results has to take that into account. When the sample is very non-representative of the population then the results are likely to be misleading. When the sample is representative despite being non-random then the results will be perfectly OK.
The next level of the question is then to ask how one can decide whether the non-randomness matters in any particular case. I can't answer that one ;-)
answered Jul 29, 2011 at 4:52
$\begingroup$
$\endgroup$
You ask a very general question, so the answer can't be suitable for all cases. However, I can clarify. Statistical tests generally have to do with the distribution observed versus a hypothetical distribution (so-called null distribution or null hypothesis; or, in some cases, an alternative distribution). Samples may be non-random, but the test being administered is applied to some value obtained from the samples. If that variable can have some stochastic properties, then its distribution is compared to some alternative distribution. What matters then is whether or not the sample's test statistic would hold for some other population of interest and whether the assumptions regarding the alternative or null distribution are relevant for the other population of interest.
