When I conduct a survey for a client, they are often very concerned about the margin of error for the survey, and that’s totally reasonable. Oftentimes however, when they say, “We want a margin of error of +/- 5%,” what they really mean is, “We expect that every estimate will be wrong by no more than 5%.”
So they see an ordinary poll in a newspaper (like this one, with 1,051 Florida voters with a margin of error of +/-3%), and they expect something similar. The problem is, they want this “magic” +/-5% margin of error for estimates in tiny subsamples, like for an estimate of: the percent of women in Texas with a college degree who have traveled to Europe at least once in the last year who are over the age of 65. Because inevitably, they want everything-cross-tabulated-by-everything.
So, I try to explain that surveys don’t really have a margin of error for the whole survey but that individual estimates have standard errors and confidence intervals. And for the record, I do understand that there’s this idea of a “maximum margin of error” or “average margin of error,” which despite whatever it’s called, usually looks suspiciously like $1/\sqrt(N)$ maybe multiplied by a design effect (but pretty much always ignoring cluster effects, even when cluster sizes are absurdly large). It almost never aligns with reasonable client expectations. And that would be fine, more-or-less, if we only ever measured one thing in a survey, with a phenomenal design, well-specified client hypotheses, no non-response, etc., and there was no interest in estimates within subsamples$\dots$ which is pretty much never.
So, here’s the conundrum: your client is naïve-but-not-stupid, so they have unreasonable expectations about survey sampling and error, largely because they’ve been misinformed by reading about polls in newspapers (let’s just assume they haven’t been misled by dishonest statisticians or polling firms offering fraudulently low margins of error$\dots$), and therefore you must choose from one of the following options:
- In the design/proposal stage, you propose an enormous sample with a complicated design that will actually deliver what they expect but will cost far more than they are willing to pay for.
- You try to explain that you can’t get a margin of error that small for every estimate, but you can come up with a nifty model that might be able to answer their question but which they probably won’t be able to understand or make reasonable use of.
- You tell them, “Sure, the margin of error is just $1/\sqrt(N)$, so with just 400 respondents you’ll get exactly what you want!” (please do not choose this answer$\dots$)
- Or finally, you educate them about the statistics of real life sample surveys.
Which option would you choose? If you choose “educate them about the statistics of real life sample surveys,” please tell me how you would explain this in a five-minute conversation without loosing your job/contract. Or better, yet propose a “fifth way” for me!
Also, for the record: I do not think that a document like the AAPOR Code of Ethics adequately answers this question. It does not adequately define “margin of error,” does not state how error should be communicated to clients, does not discuss subsamples, and really goes no further than saying, “it is best to avoid using the term ‘margin of error’ or ‘margin of sampling error’ in conjunction with non-probability samples.” My question isn’t about snowball sampling, or other non-probability methods. I don’t think specifics are necessary, but if this information is helpful, what I have in mind is a rather conventional stratified cluster design, where ideally we'd have a reasonable size of something like $N\le2400$. And by "client," I don't means an ultra-sophisticated client like the NLRB; I'm thinking of something more along the lines of a big NGO or INGO, which employs reasonably intelligent people but not survey methodologists.
