1
$\begingroup$

An opinion poll should be done on a representative sample of the population, so that its outcomes give a reliable estimate of the real public opinion.

Suppose I have a large database of users, who are not a representative sample of the population, but I have their demographic data. I want to do an opinion poll on these users, but I do not want to sample - I want to let every user answer the poll.

Is there a way to post-process the poll results so that the outcome would be as accurate as a poll on a representative sample?

As a simple example: suppose my user-base has 60% men and 40% women, which is not representative. I can give each men a weight of 0.4 and each woman a weight of 0.6, and compute the total weight for each answer (rather than the total count), and thus women and men will have the same total weight. But, gender is only one of the relevant demographic aspects; there are many others.

Are there procedures for "correcting" poll outcomes done on non-representative samples?

$\endgroup$
3
  • 1
    $\begingroup$ A simple but fundamental limitation is what was published with the opinion poll results. It's fairly common to include marginal breakdowns by age, sex, education level, etc. but you won't usually get the finest resolution of what the 40-44 year females with two degrees said, etc. There are good reasons for that, such as the very small size of many subsamples. In short, although this is not an answer and this is not authoritative. the procedures that might spring to mind are rarely applicable, and so rarely applied, in practice. $\endgroup$
    – Nick Cox
    Commented Dec 10, 2023 at 12:55
  • $\begingroup$ @NickCox In my use-case, I can add to the questionnaire demographic questions of arbitrarily fine resolution. $\endgroup$ Commented Dec 10, 2023 at 15:51
  • $\begingroup$ Sure, but my point is that you don't have the equivalent for the poll. $\endgroup$
    – Nick Cox
    Commented Dec 10, 2023 at 16:12

1 Answer 1

5
$\begingroup$

In general, the term "representative sample" is ill-defined. There are many different, conflicting meanings people intend when they use that term, and so it's not terribly helpful. For example, some people mean that the survey was conducted with random sampling with a 100% response rate, while others mean that the demographics of respondents look similar to the population. Those two definitions of representative have very different implications for the quality of estimates based on the sample. The following StackExchange post discusses this point a bit further:

https://stats.stackexchange.com/a/229213/94994

The survey methods literature has a couple of terms that are helpful to know about for digging into the literature related to your problem.

  1. "Coverage error": This refers to systematic error in survey estimates caused by drawing samples from a sampling frame which differs from the population of interest. This seems to be one of the problems you're facing.
  2. "Non-probability sampling": This broad term refers to sampling methods where each member of the population has an unknown probability of being included in the sample.

Volunteer web panels are a very common source of survey data, and they are typically subject to coverage error: for example, they systematically exclude people without internet access or digital literacy skills. But data from such panels are also often viewed as non-probability samples, both because the samples drawn from such panels are often done using non-probability sampling (e.g., quota sampling of panel members to invite to take a particular survey) and because the panels themselves can be viewed as a non-probability samples from the population.

So in general, it's helpful to use methods from the non-probability sampling literature to work with such data.

Pew Research published a nice accessible explanation of non-probability surveys here:

https://www.pewresearch.org/short-reads/2018/08/06/what-are-nonprobability-surveys/

This fairly recent paper in the Journal of Survey Statistics and Methodology provides a nice overview of approaches for estimation from non-probability samples:

Cornesse et al. "A Review of Conceptual Approaches and Empirical Evidence on Probability and Nonprobability Sample Survey Research." Journal of Survey Statistics and Methodology, Volume 8, Issue 1, February 2020, Pages 4–36, https://doi.org/10.1093/jssam/smz041

$\endgroup$
2
  • $\begingroup$ +1. I didn't know that some people are purposefully calling "representative samples" random samples, even if they know that a random sample does not necessarily reflects the demographics of the population of interest. So I learned something today. And thanks for the reference to the Cornesse et al. article $\endgroup$
    – J-J-J
    Commented Dec 11, 2023 at 8:45
  • 1
    $\begingroup$ Aside from the immediate factual mistake in the second sentence of the Cornesse paper (Netanyahu was a prime minister, not a president), the paper is quite good. $\endgroup$
    – bschneidr
    Commented Dec 11, 2023 at 17:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.