3
$\begingroup$

When I conduct a survey for a client, they are often very concerned about the margin of error for the survey, and that’s totally reasonable. Oftentimes however, when they say, “We want a margin of error of +/- 5%,” what they really mean is, “We expect that every estimate will be wrong by no more than 5%.”

So they see an ordinary poll in a newspaper (like this one, with 1,051 Florida voters with a margin of error of +/-3%), and they expect something similar. The problem is, they want this “magic” +/-5% margin of error for estimates in tiny subsamples, like for an estimate of: the percent of women in Texas with a college degree who have traveled to Europe at least once in the last year who are over the age of 65. Because inevitably, they want everything-cross-tabulated-by-everything.

So, I try to explain that surveys don’t really have a margin of error for the whole survey but that individual estimates have standard errors and confidence intervals. And for the record, I do understand that there’s this idea of a “maximum margin of error” or “average margin of error,” which despite whatever it’s called, usually looks suspiciously like $1/\sqrt(N)$ maybe multiplied by a design effect (but pretty much always ignoring cluster effects, even when cluster sizes are absurdly large). It almost never aligns with reasonable client expectations. And that would be fine, more-or-less, if we only ever measured one thing in a survey, with a phenomenal design, well-specified client hypotheses, no non-response, etc., and there was no interest in estimates within subsamples$\dots$ which is pretty much never.

So, here’s the conundrum: your client is naïve-but-not-stupid, so they have unreasonable expectations about survey sampling and error, largely because they’ve been misinformed by reading about polls in newspapers (let’s just assume they haven’t been misled by dishonest statisticians or polling firms offering fraudulently low margins of error$\dots$), and therefore you must choose from one of the following options:

  • In the design/proposal stage, you propose an enormous sample with a complicated design that will actually deliver what they expect but will cost far more than they are willing to pay for.
  • You try to explain that you can’t get a margin of error that small for every estimate, but you can come up with a nifty model that might be able to answer their question but which they probably won’t be able to understand or make reasonable use of.
  • You tell them, “Sure, the margin of error is just $1/\sqrt(N)$, so with just 400 respondents you’ll get exactly what you want!” (please do not choose this answer$\dots$)
  • Or finally, you educate them about the statistics of real life sample surveys.

Which option would you choose? If you choose “educate them about the statistics of real life sample surveys,” please tell me how you would explain this in a five-minute conversation without loosing your job/contract. Or better, yet propose a “fifth way” for me!

Also, for the record: I do not think that a document like the AAPOR Code of Ethics adequately answers this question. It does not adequately define “margin of error,” does not state how error should be communicated to clients, does not discuss subsamples, and really goes no further than saying, “it is best to avoid using the term ‘margin of error’ or ‘margin of sampling error’ in conjunction with non-probability samples.” My question isn’t about snowball sampling, or other non-probability methods. I don’t think specifics are necessary, but if this information is helpful, what I have in mind is a rather conventional stratified cluster design, where ideally we'd have a reasonable size of something like $N\le2400$. And by "client," I don't means an ultra-sophisticated client like the NLRB; I'm thinking of something more along the lines of a big NGO or INGO, which employs reasonably intelligent people but not survey methodologists.

$\endgroup$
2
$\begingroup$

The level of statistical rigor your company chooses to abide by is a business question that should be answered by the people paid to answer these questions. If this person is you, think hard about where you are on that spectrum. If this person is not you, find out where your company is on this spectrum and decide whether you're going to follow the company line, fight to change it, or find a new job.

What I have done in my career is go with “educate them about the statistics of real life sample surveys” as I had the philosophy that being correct will win business. I would explain this by saying something like, "As you know, a margin of error of +/- 5 has to do with the precision* in the survey estimate; which is highly related to the sample size in the survey. Assuming X, we would need a sample size of N. The options are either to pay for this sample size or accept a larger margin of error".

*Do not say "uncertainty", to a client this can mean you're not certain you did your job correctly. Do not say "error", to a client this can mean you made a mistake.

$\endgroup$
  • $\begingroup$ Thanks for the answer, but I'm going to leave this open just a little longer. I'm less interested in this as a business question, and more interested in this as a communications question. I want to be able to better communicate honestly and clearly the pros and cons of different approaches. Your footnote is so very very sadly and truly spot-on... $\endgroup$ – 5ayat May 13 '16 at 17:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.