I have a convenience sample, and want to show readers how biased it is relative to the population it's taken from. It's absolutely certain that the sample is biased, and I want to give readers as many details as possible regarding the bias.

Some parameters of the population are already known (e.g. distribution of gender, of places of living, etc.), so that can be used to make a comparison between the sample and the population.

For example, there are 53% of women in the sample, while they are 45% in the population, so that's a difference of 8%. (NB: the sample has other variables of interest not available at the population level, that's why I've been working with a sample in the first place, instead of simply using data from the population).

I know I could simply create a table showing the differences between the sample and the population, for example something like:

Place of living Sample Population % difference
Place A 130 (16.2%) 3,250 (13%) +3.2%
Place B 237 (29.6%) 9,000 (36%) -6.4%
Place C 270 (33.8%) 6,750 (27%) +6.8%
Place D 163 (20.4%) 4,250 (17%) +3.4%
Place E 0 (0%) 1,750 (7%) -7%
Total 800 (100%) 25,000 (100%) n/a

But what I'd like to show is how unexpected or surprising the difference of percentages is between the two.

I've been thinking of calculating confidence intervals of percentages of the sample, and then check if the population parameter lies in the interval - but as it's absolutely not a random sample, I find it questionable to do that.

Are there methods or statistical tests that I could reliably use to show how biased a non-random sample statistic is compared to a population parameter? Or should I simply stick to something purely descriptive like the table above?

If my question does not even make sense, I'd also be happy to know why!



1 Answer 1


I would go with probabilities. For ex. for binary variables such as sex you can use the binomial distribution. The question you are trying to answer is: how likely is it that I would obtain a sample with $800 \cdot 0.53=424$ (or more, for a one-sided test) females in my sample, given that in my population there are $45 \%$ females. In R code

> 1-pbinom(800*0.53,800,0.45)
[1] 2.441409e-06

which is practically zero, so no chance.

  • $\begingroup$ Thanks! I'll check multinomial tests for variables with more than 2 categories. $\endgroup$
    – J-J-J
    Mar 24, 2022 at 14:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.