Statistical testing on non-random sample?

Question

I am working with a non-random sample in an observational study but would like to do statistical testing to show certain trends and processes. Statistical tests assume random sampling, which is not met in my experiment so my concern is that my results (or interpretation?) are invalid. What options do I have? E.g. should I just be extra careful with the interpretation? Should I remove all text concerning statistical testing?

And the context:

I am conducting a scientometric/bibliometric analysis in discipline A. Discipline A is relatively new, but the consensus is that it evolved into a discipline on its own merit over the last 3 or so decades. I identified a certain number of journals based on other scientometric studies and the larger literature, and consider these journals to make up the bulk of discipline A. This list is arguably not exact science and is subjective. The problem of identifying journals is described and explained and justified in my opinion.

I then take this list of journals and analyze trends, such as publication volume over time, contributing countries, etc. so that I can provide insights into journal characteristics, or into the discipline (i.e. all samples journal combined). Sure, the sample is not random, but it still provides insights into the discipline.

Let's say I want to examine a county's economic activity (measured by GPD per capita) and and I fit a linear regression model, which shows that the relationship is statistically significant. How do I go about this? Do I not include statistical testing at all, or do I just make it explicitly clear that what we are seeing refers to the selected set of journals (i.e. that is the whole population).

My worry is that what I am doing may seem dishonest based on the discrepancy between the journals I sampled and the discipline (which is larger), but I am not sure how to resolve the conflict of wanting to provide insights backed by statistical testing vs. the flaw in sampling in the process.

Thanks.

Answer 1

The result of your analysis (regression, or whatever you did) are valid for your sample; the effect sizes are accurate estimates for your sample. But the validity of any generalization relies on your sample being random, or, at least, non-biased. But you can't prove that the sample isn't biased. So, you can either not generalize at all (no inferential statistics) or you can be explicit about what you are doing.

The latter course is done a lot in many disciplines. It is often impossible or extremely difficult to get a random sample. You can argue that your sample is sort-of random, or close to random. You can look at descriptive statistics about your sample and (if possible) compare them to the larger population of journals. You can add covariates to your model that you think will control some of the bias.

But you can't be certain.

The real danger, in my experience, is that, no matter how careful you are to give limitations and warnings, the findings may be quoted and the warnings and limitations ignored. Then that citation of your findings gets copied and copied. And, if it's wrong by a lot, that's a problem.

OTOH, lots of good studies that had important and useful results were based on non-random samples.

Sorry that that is not a definitive DO THIS answer.

Answer 2

I come to this from a very different angle than @JohnX. For me the first thing to acknowledge here is that statistical models are thought constructs. There is an essential difference between formal models and reality, and the job of models is not to be "true" (starting from the rather trivial observation that no real data are continuous, everything in reality in some way depends on everything else, and most or even all random numbers aren't really random).

If we use model based inference on data that are not a result of random sampling, what we do is we compare the data to expected outcomes of models that are idealised thought constructs. As long as these models model ideas that we think may be a good description or a good way of thinking about the real situation, this can be relevant even if we know that reality is not like this (although then obviously a result that just says "reality is not like this" like "we reject the null hypothesis" shouldn't be celebrated as a big discovery).

The importance of acknowledging the fact that the model implies random sampling and this is not the case in reality is that we need then to ask whether the result can then be explained by the deviation from random sampling, in which case the substantial interpretation of the result that one may have in mind may not hold. It is therefore important to try hard to find reasons that explain the result, i.e., specific issues with the sampling that may have caused it (like for example journal editors and reviewers being dependent in ways that promote certain trends in the "story" in the question; dependence may result in lower estimated error variances compared to independence, and in turn may explain significance of certain tests). This would imply that such results can be attributed to issues with sampling than to what really was the focus of interest.

Such explanations can be found often, so in fact deviations from random sampling are indeed often problematic, but in some cases one may be able to argue that such problems are mild or aren't enough to explain certain results, in which case these results still may be meaningful. They will, however, come with larger uncertainty than what is quantified in model-based inference, because some degree of model and assumptions uncertainty always remains on top of what can be obtained from the model. We could say such results are still somewhat tentative, because not having found problems with random sampling or other model assumptions doesn't mean that there are in fact none, but often this is the best we can achieve.

I think that it is grossly misleading in statistics to say that model assumptions "have to be fulfilled" in order to apply model-based inference, because if this were true, model-based inference could never be used. Same holds for random sampling, which formally just translates into a model assumption.

On the other hand model assumptions are important, because in order to appreciate model-based results, we need to think hard about how their violation can affect results in the situation in question, and of course efforts to improve matters (such as finding as good a random sample as possible) will help.

And also, people shouldn't think that they have to use model-based inference such as tests for making any statement from data. It has its role, but sometimes a test with inappropriate model assumptions just doesn't add information to using, for example, good data visualisation and descriptive statistics (and if such a test is carried out and interpreted as if it added information and meaning, it is worse than not using it).

Answer 3

As a practicing statistician, I have observed a troubling disconnect between the principles of mathematical statistics theory and the application of these principles in real-world data analysis. In essence, many statisticians are not practicing what they preach.

Random sampling from a well-defined target population and randomisation in experimental studies are two of the most fundamental principles of statistics theory for any statistical inferential analysis. Random sampling is crucial for ensuring the external validity of statistical results, while randomisation in experimental design safeguards internal validity by minimising potential confounding effects, whether known or unknown.

The process begins with a necessary random sample and continues with many more random samples until a reasonable sampling distribution is constructed, which is sufficient for achieving external validity. It is through the sampling distribution of sample statistics that we approach the core objective of scientific inference: distinguishing between scientific truth and falsity.

However, technical limitations are common, such as research populations rarely being finite and unchanging, ethical constraints preventing randomisation, non-random samples, missing values, violations of model assumptions (including but not limited to independence, equal variance, and normality). Any of these factors compromises the reliability and validity of statistical inference. These issues are rarely acknowledged or adequately addressed in practice, often deliberately ignored.

In the ASA President’s Task Force Statement on Statistical Significance and Replicability[1], the authors assert that “P-values and significance tests, when properly applied and interpreted, increase the rigor of the conclusions drawn from data.” However, they do not clarify what exactly constitutes the proper application and interpretation of these tools.

I propose that, in addition to meeting any assumption conditions for a specific statistical model, statistical inference should only be considered properly applied when the following three conditions are met: (1) random sampling is used to obtain the sample for inference, (2) experimental units are randomised with respect to the treatment conditions of interest, and (3) the study is repeated multiple times to adequately establish the sampling distribution.

In reality, it is rare to find cases where these criteria are fully satisfied, and the level of deviations is often unknown. This leads to the conclusion that most statistical inferences in real-world research are invalid either internally, externally, or frequently both.

On the other hand, a 2019 editorial in The American Statistician, “Moving to a World Beyond ‘p < 0.05’” [2], makes a specific and necessary call to end the use of the pseudo-scientific concept of ‘statistical significance’ in statistical analysis practice. This call represents the bare minimum required to ensure good practice in statistical analysis. Therefore, I sincerely urge professional institutions such as the American Statistical Association and the Royal Statistical Society to provide clear, operational, and theoretically defensible guidelines for statistical practice.

The true role of statistical analysis lies in statistically describing or characterising quantitative evidence regarding scientific research findings and offering what-if scenarios through logically consistent statistical modelling. The validation or justification of scientific research findings, however, is a task for the scientific method itself, not achievable through statistical inference. For example, no matter how many survey studies are conducted or how statistically rigorous (or flawed) these studies are, statistical inference alone cannot prove that certain chemicals in tobacco cause lung cell damage, thereby establishing causation between smoking and lung cancer. I therefore strongly resonate with the perspectives of R. Hubbard et al. and C. Tong, who argue that statistical inference has a limited role in the broader process of scientific inference [3]. As Tong aptly put it, "Statistical Inference Enables Bad Science; Statistical Thinking Enables Good Science."[4] This distinction highlights the need for a deeper understanding of the role and limitations of statistical tools and the importance of integrating them into a rigorous scientific framework.

As statisticians, we must collectively acknowledge that most current statistical practices do not align with the principles we teach in mathematical statistics. A hard truth that we must accept is: ‘Statistical inference is built upon layers of assumptions.’ Yet, the reality is that ‘ Assumptions are not met. Period.’ [5] It is time to realign our practice with our theory, ensuring that our analyses are both scientifically sound and practically applicable.

References: [1] Benjamini, Y., De Veaux, R., Efron, B., Evans, S., Glickman, M., Graubard, B. I., He, X., Meng, X.-L., Reid, N., Stigler, S. M., Vardeman, S. B., Wikle, C. K., Wright, T., Young, L. J., & Kafadar, K. (2021). ASA President’s Task Force Statement on Statistical Significance and Replicability. Harvard Data Science Review, 3(3). https://doi.org/10.1162/99608f92.f0ad0287

[2] Ronald L. Wasserstein, Allen L. Schirm & Nicole A. Lazar (2019), Moving to a World Beyond “p<0.05”, The American Statistician, Vol. 73, No. S1, 1-19: Editorial.

[3] Raymond Hubbard, Brian D. Haig, and Rahul A. Parsa (2019), The Limited Role of Formal Statistical Inference in Scientific Inference, The American Statistician, Vol. 73, No. S1, 91-98.

[4] Christopher Tong (2019), Statistical Inference Enables Bad Science; Statistical Thinking Enables Good Science, The American Statistician, Vol. 73, No. S1, 246-261.

[5] MD Higgs (2019), https://critical-inference.com/assumptions-are-not-met-period/