# Looking for resources (papers, books) that explain the impact that non-random sampling has in test statistics

The majority (if not all) of test statistics assume random sampling. Consequently, probability values obtained in t-tests, ANOVAs, regression, HLM, etc., are intrinsically linked to the assumption of random sampling. However, in social sciences, it is often the case that random sampling is not possible, as you resort to convenience sampling (e.g., depressed individuals, autistic individuals, etc). Do you know of accessible - meaning not heavily mathematical - resources to better understand how results can be interpreted in light of convenience sampling and the overall trends/impact non-random sampling has in the interpretation of results?

• Some tests rely not on random sampling but random assignment to treatment (the problem then becomes one of generalizing the conclusions to unrepresented or heavily under--represented groups, but -- if the groups of main interest are well-represented this may not be an issue, as long as the limitations of the conclusions are made clear). – Glen_b -Reinstate Monica Apr 25 '17 at 22:41

Missing data is an important source of non-random sampling, in particular if you are running multivariate analysis, where some particular regressor might be missing. This paper is a gentle introduction to the issue. This is a great book.

In the context of panel data, I would start by looking at the literature on unbalanced panels. This is the case of some units not having the full set of observations. This is directly related to non random samples, as the patter of selection might be non-random. Here is a nice presentation on the topic. Here is a fairly recent review paper.

A common solution to non-random samples is the use of weights. I find this introductory note, by the UK Data Service, very illuminating. It describes the issue and possible solutions.

The issue of clustering, related to the above, is also of major importance. Some datasets are constructed based on clustered sampling. This article, albeit a bit mathematical, deals with robust inference when non-random sampling is due to clusters.

• What a great reply. Thank you so much luchonacho, I will make the best of it by reading the documents you suggested. Have a great day! – Stats Iliterate Apr 25 '17 at 11:02
• @StatsIliterate You are welcome! Once you feel this answer provided you the answer you were expecting for, do not forget to makr it as "solved". – luchonacho Apr 25 '17 at 11:50
• The sense in which samples with missing data are non-random is very different from the sense in which convenience samples (the subject of this question) are non-random. The references you supply all implicitly assume the samples indeed were random at the outset but, for various reasons, not all data could be observed. In a convenience sample no randomization is used at all. Thus, the methods of interpreting and analyzing convenience samples tend to differ substantially from methods of dealing with missing data in random samples. – whuber Apr 25 '17 at 14:41
• Hi whuber. Thank you for your comment. Based on your assessment, are there any references that you'd feel comfortable recommending? – Stats Iliterate Apr 25 '17 at 16:30

However trivial it may seem, Wikipedia has a nice article on non-probability sampling. The bottom line of the article is that non-probability sampling techniques are not intended to be used to infer from the sample to the general population in statistical terms.

As far as statistical inference in case of complex probability sampling designs beyond simple random sampling is concerned Sharon L. Lohr's SAMPLING: DESIGN AND ANALYSIS seems to be a good resource.