13
$\begingroup$

I was looking at a WashingtonTimes/ABC poll, and the results breakdown, especially by age, seemed surprising to me. So I went looking for bias.

It said: "poll was conducted by telephone Dec. 11-14, 2014, among a random sample of 1,000 adults. Interviews were conducted in English and Spanish on landlines and cell phones."

The poll did not give me the numbers of respondents in each age group, and my first thought was what kind of 18-29 year old is home to answer the phone?! OK, they include cellular, that makes it a bit fairer. But still, what kind of 18-29 year old answers their mobile phone, gets asked if they want to take a survey, and says "yeah, sure".

(I don't mean that as a rhetorical question; my hunch is that certain groups are much more likely, e.g. someone with low self-esteem might enjoy the process of being asked for their opinion by a stranger over the phone.)

So, is there any research on this topic? Would an organization interested in manipulating public opinion choose a telephone poll, over online polls, or interviewing in the street, if they were after a specific bias?

$\endgroup$
8
  • 2
    $\begingroup$ There is no "general bias" for general kind of studies. They vary from study to study. $\endgroup$
    – Tim
    Commented Dec 22, 2014 at 11:39
  • 3
    $\begingroup$ When you see results from a (reputable) poll, the numbers are almost always adjusted for selection bias. It's also worth noting there are much easier ways to bias a poll than forcing a specific poll methodology. $\endgroup$
    – Andy Jones
    Commented Dec 22, 2014 at 14:59
  • 3
    $\begingroup$ Fwiw I am that 18-29 year old $\endgroup$ Commented Dec 22, 2014 at 20:12
  • $\begingroup$ @ssdecontrol The low-self-esteem one? I just want you to know we all think you are fantastic, and I would definitely talk with you at parties. :-) $\endgroup$ Commented Dec 22, 2014 at 21:20
  • $\begingroup$ @DarrenCook hahaha no, the broader category that actually responds to phone interviews! But thanks for the confidence boost $\endgroup$ Commented Dec 22, 2014 at 21:21

3 Answers 3

8
$\begingroup$

There's a nice study by Pew that looks at how much response rates have declined, and also at what the effects are.

http://www.people-press.org/2012/05/15/assessing-the-representativeness-of-public-opinion-surveys/

"A new study by the Pew Research Center for the People & the Press finds that, despite declining response rates, telephone surveys that include landlines and cell phones and are weighted to match the demographic composition of the population continue to provide accurate data on most political, social and economic measures. This comports with the consistent record of accuracy achieved by major polls when it comes to estimating election outcomes, among other things.

"This is not to say that declining response rates are without consequence. One significant area of potential non-response bias identified in the study is that survey participants tend to be significantly more engaged in civic activity than those who do not participate, confirming what previous research has shown. People who volunteer are more likely to agree to take part in surveys than those who do not do these things. This has serious implications for a survey’s ability to accurately gauge behaviors related to volunteerism and civic activity. For example, telephone surveys may overestimate such behaviors as church attendance, contacting elected officials, or attending campaign events."

$\endgroup$
1
  • 1
    $\begingroup$ The other answer was also excellent; I gave this the tick as this report has exactly the kind of numbers I was looking for. $\endgroup$ Commented Dec 24, 2014 at 8:40
16
$\begingroup$

Is there a known bias with telephone polls in the U.S.?

Yes, there is a known bias with telephone polls in the U.S.: cellphones.

For a long time, the gold-standard of randomized telephone polling in the US was random digit dialing: machines generated random phone numbers and pollsters asked questions of whoever picked up. Assuming everyone had one and only one phone, the poll was a simple random sample of phone owners.

However, the arrival of cellphones changed that: many people, especially the young, have no landline. Even within specific demographics (age bracket, gender, race), these cellphone-only populations have distinct political attitudes from populations with landlines or both types of phones. Since it's illegal for automated dialers to call cellphones, this wreaks havoc with polls.

For pollsters, this means that they have to use human beings to call all cellphones, which is obviously much more expensive. So pollsters must, a priori, decide how many cellphones they need to call to control costs. The number of cellphone calls must be balanced against the number of automatic landline calls in a scientifically-sound way -- dialing enough cellphone-only houses, landline-only houses and houses with both in proportion to their incidence in the population, all while not exceeding their budget!

The effect of cellphones probably isn't the source of potential bias.

I suspect that pollsters that the Washington Times/ABC poll used is well aware of all of this, since how to construct a random sample in light of the "cellphone effect" it's pretty much the topic that all contemporary political polling is dealing with today. So it's likely that the pollsters took steps to mitigate the "cellphone effect" in this poll. However, perhaps the pollsters had a bad day and their random dialing of cellphone numbers did not yield enough actual responses.

If there is bias, it could be the sample weights.

Another potential source of bias in these polls is the sample weighting. The goal of weighting samples is to reduce the number of people you need to reach in various (combinations of) demographic groups by attributing different weights to different population segments. This, of course, leads to the question "how do you know you've weighted the sample correctly?" This is why some polling firms will have estimates that are consistently above/below those of other polling firms: they have different methods for weighting their sample, and that consistently pulls the result in a particular direction.

This is an example of the bias-variance trade-off. The pollsters could use a small, simple random sample to achieve a high-variance estimate. Or they could use weights to reduce the variance of the estimate from the small sample, but at the cost of (statistical) bias.

Pollsters (and statisticians generally) all have to grapple with the bias-variance trade-off in one way or another. It doesn't necessarily mean that they're malevolent or attempting to manipulate the public.

Alternative explanations abound.

There are lots of ways to mess up a poll, accidentally or deliberately. Rephrasing a question can yield different answers among different groups. Push-polling, in which questions are framed to elicit a particular response, is the most obvious and nefarious example of this.

Would an organization interested in manipulating public opinion choose a telephone poll, over online polls, or interviewing in the street, if they were after a specific bias?

This question isn't entirely statistical, as it calls for speculation about the motives and methods of some hypothetical, malevolent entity. I can only answer it partially.

Interviewing people on the street isn't really random, and is probably biased (in the sense that it will tend to include some groups at greater proportion than others).

The goal of random-digit dialing is to achieve a state where everyone has equal probability of selection. Going to some street corner and interviewing people doesn't quite achieve this, because you'll tend to include people who live and work nearby. If you go to a street in Washington, DC, you'll have nearly 0 chance of selecting a person who is not from DC, Maryland or Virginia.

Moreover, consider which street you go to. Interviewing people on the street in, say, the Cleveland Park neighborhood of Washington, DC will be very different than interviewing people on the street in Anacostia: these places have very distinct demographic composition.

Polling people online isn't really random, but it's unclear whether it's worse than phone polling.

The population of people opting-in to an online poll is a self-selecting population, not necessarily randomly sampled from all potential respondents. Traditional pollsters who tend to favor telephone polling are very skeptical of online polls for this reason. However, people like Andrew Gelman are less certain that the end result of a randomly-dialed phone poll and a self-selected online poll is meaningfully different. This is very much an open area of research.

$\endgroup$
4
  • $\begingroup$ Do you really need so much bold formatting? $\endgroup$ Commented Dec 22, 2014 at 17:43
  • $\begingroup$ @gung it actually reads very well on the iPhone app $\endgroup$ Commented Dec 22, 2014 at 20:13
  • 3
    $\begingroup$ @gung I wanted to break it up so it could be easily navigated and not appear as a wall of text. Is this better? $\endgroup$
    – Sycorax
    Commented Dec 23, 2014 at 2:00
  • 1
    $\begingroup$ I can't speak for others (eg, @ssdecontrol) or how it looks on mobile devices, but it looks a lot nicer to me. Thanks. $\endgroup$ Commented Dec 23, 2014 at 2:31
1
$\begingroup$

Would an organization interested in manipulating public opinion choose a telephone poll, over online polls, or interviewing in the street, if they were after a specific bias?

All polling methods can be biased, including some others not mentioned in the question.

To over-represent elder viewpoints, choose a telephone poll and 'forget' to normalise the age quota groups, knowing that younger respondents were poorly represented.

Street polling without any quota setting and normalisation would probably be the easiest way to get a bias. As per Sycorax, just choose your street. Polling in working hours or out of them would also transform the results. I was once a street interviewer. We worked to strict set quotas combining age range, gender and social class, but within this I was conscious of my own tendency to approach people within quota that I liked the look of (people who were a bit like me, if you like) rather than people within quota that had less immediate visual appeal. And I don't think they had quotas for interviewer types!

British tabloid newspapers regularly conduct self-selecting yes/no 'surveys' of their own readers, typically on an emotive leading question, reporting the results as if they were a genuine survey representative of the population, and often with greater prominence than proper, ethically conducted surveys.

As Andy Jones has commented, there are much easier ways to bias a poll. Choice of dates and times, wording adjustments (especially in Before and After surveys), leading questions, directive mood-setting prior or filtering questions, reductionist assumptions, restricted response choices and so on. In London, various council traffic management, parking management and public consultation sections seem to make such underhand techniques their speciality.

Typically, if they are bringing in traffic restrictions, they will consult only residents within a narrow nearby area, and drivers not at all. The residents will get a tick-box card through their door, or maybe two cards in a big old property that is divided into five or seven separate flats.

A truly unbeatable form of poll bias, though not part of the polling process itself, is to conduct it in whatever way (eg perfectly) and then simply suppress the results if they don't suit. A residents' association for an area I once lived in, controlled with an iron hand by a very manipulative trade union official, did precisely that.

Finally, in 2004, coming back from a holiday at Stansted airport I found myself confronted by a "doorway" consisting of a double perspex wall and a motion detector. Passengers were invited to walk through this optional doorway if they wanted to make the Olympics happen in London. A sizeable minority of public opinion at that time was unenthusiastic about getting a massive bill for a vanity event, recalling the Greek fiscal burden and the corruption in Utah, and the London Back the Bid team were trying to drum up support.

You could walk through the doorway as many times as you wanted in order to register a 'vote' (it was called that) each time for the Olympics, to the delight of families with young kids. Alas, they forgot to provide a second doorway for people who didn't want a big bill for 30 years, or for people like me who thought that Paris was worthy of the event.

Oddly enough, the Soviet-style outcome was that 100% of Britons wanted the Olympics. I'm going to be still paying for them long into my retirement. Handy things, polls.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.