4
$\begingroup$

From what I understand, to generate a margin of error to have confidence intervals for a given estimate one needs the standard error of the estimate (SE). For the SE one needs information about the sample size. Is it the fact that the sample size is non-existent when doing say, purposive or quota sampling, what impedes us from generating sampling errors of the estimate to later calculate confidence intervals?

I then thought, well, regardless of the sampling method use (probability based or not), you'll end up with a certain sample size. Can this not be used to then calculate SE of the estimate, and ultimately confidence intervals. Or perhaps, it all comes down to my final suspicion: one can certainly calculate SE for an estimate and then proceed to generate the CIs, but these will ultimately be wrong given the poor consideration of the sampling design.

$\endgroup$
1
$\begingroup$

I have a different take on this. A confidence interval confronts the sample with a certain probability model. It basically gives a set of parameters, under the model, that are consistent with the sample. Obviously the model is wrong in various respects (as all models are, according to George Box at least), independence and others; the question is whether it is still sensible enough so that it makes sense to think about the underlying reality in terms of the model. An appropriate random sampling method is a device that makes some (but not all) of the model assumptions plausible. In the absence of such a sampling method, one needs to think about whether and how the way the sample was actually collected can affect the conclusions that are made from the confidence interval. The confidence interval itself is a mathematically well defined set of numbers - there's nothing wrong with it. What is the problem is really how this is interpreted and used in practice. There are many procedures for collecting samples that will invalidate the model in critical ways, thus invalidating conclusions (take for example something that was recently discussed on Andrew Gelman's blog, researchers doing animal experiments grabbing whatever mouse comes to their hand from the cage, which systematically will choose mice that move more, and are potentially more healthy and energetic).

However, in other cases even critical appraisal does not throw up clear reasons why a sample might deviate systematically from a proper random sample, and confidence intervals are also interpreted cautiously as potentially imprecise indicators of uncertainty, maybe just for comparing different aspects in a study where the same data collection was used across the board, and then the standard confidence interval may be OK. It's not without risks (there can be problems that the researchers miss), but then model-based reasoning always comes with this kind of risk, even with "proper" random sampling.

Of course what whuber alludes to in their comment to the other answer is also true - there may be a model for the specific non-i.i.d. sampling mechanism that has actually been applied, and this can be used to compute more appropriate confidence intervals in that case.

$\endgroup$
0
$\begingroup$

Your final suspicion is correct. Without a sampling methodology that guarantees independent, identically distributed variables you can't generalize your sample to the population, which is what a confidence interval is for, i.e. saying something about how your estimate (sample mean) relates to the population mean.

$\endgroup$
1
  • 4
    $\begingroup$ This answer is too extreme: there are many valid ways to compute confidence intervals from samples that are not iid. $\endgroup$
    – whuber
    Nov 20 '19 at 20:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.