1
$\begingroup$

Survey margin of error contracts as the proportions become more extreme. Its validity and applicability in such cases has always concerned me, but I suppose much depends on the context.

Where we have mere traces of data on one side, an extreme proportion such a 1% vs 99% opinion in a survey, can we validly assert that the MoE is only 0.6 points? At a certain level of wispiness, I assume we have to consider errant clicks (online survey), reading errors, other forms of measurement error, etc. as overwhelming our ability to speak of the 1% responses as substantively real or validly refer to the MoE. (Sample size 1145. 10 respondents on one side and 1135 on the other.) What's best practice in such cases?

The known population rate on the question is 6% vs 94% at minimum, often much higher than 6% depending on the country (known in the sense of repeated random samples by reputable polling organizations.) How would the known population figures enter into an assessment of MoE for the 1% sample? I don't see much in the literature, and the formulae don't seem to take known population proportions into consideration. Ch-square will tell us the 1% sample is significantly different from any other survey we'd compare it to, but otherwise I don't know what we could say about the MoE for the new sample. Ideas?

$\endgroup$
2
  • $\begingroup$ "Where we have mere traces of data on one side, an extreme proportion such a 1% vs 99% opinion in a survey, can we validly assert that the MoE is only 0.6 points?" --- where does 0.6 come from? How did you compute it and what was your $n$? Also, it's not quite clear what you're discussing in your last paragraph. What's known vs what's estimated? Can you explain the situation? $\endgroup$
    – Glen_b
    Commented Sep 21, 2014 at 2:39
  • $\begingroup$ 1. Measurement error will affect the MoE regardless of its value. 2. The reason for calculating a MoE would be to infer something about the population proportion. You know the population proportion. $\endgroup$
    – rolando2
    Commented Sep 21, 2014 at 2:40

1 Answer 1

0
$\begingroup$

Validity of some margin of error calculation depends on how you're calculating margin of error, the validity of the assumptions made, and what interpretations are put on it.

Typically, margin of error calculations one sees assume large numbers and middling proportions (often, by default, assuming the worst case of a proportion of 0.5 to get an upper bound on the required sample size); this yields a normal approximation.

When expected numbers are quite small, the normal approximation isn't much use. If you have some idea of the proportion, you can use a suitable binomial proportion confidence interval to obtain a confidence interval for the proportion. There's also a useful Poisson approximation, though with decent software you shouldn't need it.

t a certain level of wispiness, I assume we have to consider errant clicks (online survey), reading errors, other forms of measurement error, etc. as overwhelming our ability to speak of the 1% responses as substantively real or validly refer to the MoE.

Yes, the assumptions matter; one can make different, more general assumptions (for example, considering binomial mixtures like the beta-binomial, or by considering some contaminating process, or by allowing for potential dependence across trials, for example), or one can assess the sensitivity of the usual binomial assumptions to such effects.

The known population rate on the question is 6% vs 94% at minimum,

Here you just seem to have launched into something else without context. What is "the question"? What is known? What is being estimated? What is it you're trying to find out from data there?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.