3
$\begingroup$

For a long time I have been meaning to set up a bunch of online surveys asking a whole range of social questions and publish the results.

I am well aware that there are various difficulties to overcome in this goal (e.g. privacy, excluding bots etc.). The hardest problem to overcome for me is the fact that any online survey will always have an inherently biased audience. People who visit my websites will certainly be biased (young, technical, probably left-wing), and the same is true for every single website on the internet.

However, my feeling is that some data is always better than no data, especially if I'm going into it with my eyes open to the potential for biases.

My question

Are there established methods for controlling for and understanding exactly how your audience is biased? Are there certain control questions I could include in my surveys which can then be compared against a well-known normal distribution to understand a particular area of bias? Is this a common practice?

Edit: Phrased a different way: What methods / principles / areas of study exist for helping to understanding the ways in which the set of respondents to a given survey may be skewed as compared to a known population? And how reliably can this understanding then be used to draw conclusions from said survey about the known population?

Off the top of my head I can think of asking age or race (although some respondents might baulk at being asked these questions) and using these to better understand my sample, but are there guidelines around how to ask these sort of questions so as not to put people off? And are there particular subtle questions I can ask to uncover and understand more subtle audience biases, e.g. in political leaning or social class?

Most importantly, is this a well-known area of study which has a specific term in academia? If I know what it's called then I can more easily explore further reading on the subject.

$\endgroup$
  • $\begingroup$ So 2 years later, did you learn anything? I have exactly the same concerns going into a large survey of an online population. $\endgroup$ – thecity2 Mar 2 '18 at 21:28
2
$\begingroup$

The sources of bias are many in survey research from the type of scale used, the wording of the questions, the order in which the questions are asked, the sampling frame (e.g., random, stratified, convenience), the method used (e.g., online, mall intercept, telephone, f2f), nonresponse, data management, and so on. In fact, each of the topics mentioned here constitute classic subdisciplines within the broader domain of survey research or else overlap with independent and massively researched topic areas such as psychometrics, cognitive psychology, sampling, methodology, and so on. Degreed programs are offered in it, e.g., the University of Michigan's ICPSR (Inter-university Consortium for Political and Social Research). Entities exist which profit from leveraging these techniques, supposedly in an unbiased manner, to produce and publish the results from national and international opinion polls such as Gallup, Harris, Pew, Research International, NORC, the US Census Bureau, to name just a few.

The field of behavioral economics, in large part, exists as a function of its decomposition of the various "heuristics and biases" which we fallible humans are subject to. Judgement and decision-making broadly overlaps with this field as well.

So, yes, this is a "well-known area of study" in academia.

That said, a commonly used heuristic (at least in marketing) is to acknowledge that online surveys are biased convenience samples and reweight them to nationally available demographic marginals. This doesn't remove the biases but it makes the information more palatable to a naive and gullible audience which never looks to closely under the rocks.

$\endgroup$
  • $\begingroup$ Thanks @DJohnson. I wasn't asking if "bias" at large is well-known area of study. Obviously it is. I have a masters degree which involved social science research. My question is specifically about the part you touch on in your last paragraph - what methods are there for measuring and controlling for the obvious audience bias in online surveys. You mention that such methods exist, but not what they are or where to find them. $\endgroup$ – Robin Winslow Mar 31 '16 at 15:27
  • $\begingroup$ Why would the methods for controlling for bias be that different for online research? I think you should elaborate more on that. Of course, one obviously different issue is the difficulty of generating a representative, random probability sample, e.g., consider so-called "river" samples which make no attempt to do so. On the other hand, I know of at least one vendor that claims to have solved this...Knowledge Networks... knowledgenetworks.com/ganp/docs/… $\endgroup$ – Mike Hunter Mar 31 '16 at 15:55
  • $\begingroup$ For instance, the heuristic mentioned in my last paragraph is not unique to online surveys. It's been around since statistical sampling went out the door with the advent of mall intercepts. It doesn't change the reality that mall intercepts (and most online surveys) are convenience samples for which, strictly speaking, statistical tests don't apply since there is no "population" to project to. And that's the heart of the problem in marketing: the economics and realities of cost reduction have driven the quality of the results to abysmal levels. Online surveys are integral in this trend. $\endgroup$ – Mike Hunter Mar 31 '16 at 16:14
  • $\begingroup$ Agreed it's not at all unique to online surveys. Audience bias is absolutely a problem with surveys in general, but nonetheless particularly true with online surveys. What I'm asking (let's see if I can rephrase this any better this time) is: what methods / principles / areas of study exist for helping to understanding the ways in which the set of respondents to a given survey may be skewed as compared to a known population? And how reliably can this understanding then be used to draw conclusions from said survey about the known population? $\endgroup$ – Robin Winslow Mar 31 '16 at 21:43
  • $\begingroup$ I'm glad we could clarify your question. $\endgroup$ – Mike Hunter Apr 1 '16 at 0:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.