4
$\begingroup$

I have sent a survey to 92 persons but so far I have received only 40 responses. By using sample size requirement calculation I need 75 responses for the desired criteria

Margin of error = 5%
Confidence level = 95%
Response distribution = 50%

Is it appropriate to use this sample size calculation for a small population (N=92)? I used this Sample size calculator

Thanks in advance for your help !

$\endgroup$
4
$\begingroup$

Take a step back and consider what you really need to know. Remember, statistics is concerned with Type I (false positive) and Type II (false negative) errors. Sample size calculators like the one you linked to focus on Type I errors expressed as p-values or confidence intervals, i.e. the probability that any effect you detect is due to random error vs. "true" population differences. That is important because you want to make sure you have statistical significance.

But you also need to be able to detect a difference if it is there - that is where you consider Type II errors - you need enough sample to detect an effect when it is really present. This is the Power of your test. Power analysis tells you the minimally detectable effect size your sample can attain.

Power analysis can get incredibly confident, but for your purpose, it is relatively straightforward. With teaching to fish in mind, go ahead and download G*Power, a free power analysis program that will work great for your purposes.

For example, assuming you are doing a test of proportions, this chart can help you out: enter image description here

The three lines show you a curve of how much sample size you need to attain a given power (0.80 is kind of a standard, but it is more a convention than a rule, much like why we choose 0.05 rather than 0.051 or 0.06 with pvalues), for a given difference in proportions. The chart assumes that the reference sample (p1) is 0.5, so the lines represent the proportion for the comparison group (p2).

The main point here is: depending on the size of the differences between the two groups you are looking at, you may need different sample sizes. Power analysis for means, regression, etc., involves other factors. Cohen's work (e.g. this is the key reference here, though there are tons of other primers out there.

A couple of other points:

You have a small population (92) - that could call for a finite population correction, which would reduce your standard errors further and make your "small" sample size not so small, and help you out. Of course, you have to make the argument that the pop of 92 doesn't reflect a broader super-population. That's somewhat of a philosophical argument. See Andrew Gelman's short piece or an older article that seems to kick it off.

I hope you are thinking about response rates - you've got a response rate of 44%. Some fields that's good, others that's bad. Response rate is a concern in that it may reflect non-response bias. There is a ton of literature on that in survey research it wouldn't hurt to be familiar with. I've got a blog post summarizing it.

$\endgroup$
4
  • $\begingroup$ Thanks for your answer. Yes, I'm thinking about response rate, considering responses as sample. The survey is regarding a teaching satisfaction. Can we accept a low response rate in this study? $\endgroup$ – Learner Dec 4 '14 at 7:05
  • 1
    $\begingroup$ I don't know if that's something I can really answer for you. How will you use the results? Is it for programmatic improvements? Is it based on some intervention and you want to see the effects? Different objectives call for different coverage I would say. Do you have demographics on the whole population (e.g. tenure, age, highest degree, gender, race/ethnicity, subject, grades taught) - if so you should do a non-response analysis. The most basic is to run a logit model with response as a the outcome and the demographics as predictors. $\endgroup$ – robin.datadrivers Dec 5 '14 at 4:26
  • 1
    $\begingroup$ I thought, if 70% said they want two additional coaching sessions, for example, the 70% is within a plus or minus 5% accuracy range. So it could be anywhere from 65% to 75%. Is that right ? $\endgroup$ – Learner Dec 6 '14 at 7:09
  • $\begingroup$ 70% of the 40 that responded. If you believe the 40 are a random sample then it may be fine. But if there is selection, then you should consider assessing that using any demographics you have. $\endgroup$ – robin.datadrivers Dec 7 '14 at 4:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.