How to calculate number of participants required to compare mean scores on a questionnaire between two groups?

Question

I have to create a questionnaire that compares the sportiness of Facebook users vs. non-Facebook users. I need to calculate the number of participants that are required so that the result is statistically relevant. Unfortunately I have no idea how I can calculate that. Could you help me?

Answer 1

Defining the problem

• I am guessing that you are measuring sportiness on some form of multi-item scale. Thus, you will have a numeric measure of sportiness. I am assuming that you will be running an independent groups t-test to test the difference between group means.

• Your problem is then a form of a priori power analysis. To determine the sample size you need to specify:

  1. Your desired statistical power (often 80% is considered adequate)
  2. Your expected effect size (typically specified as standardised group mean difference; of course this is not known ahead of time, so you have to think about what is the minimal effect that you would consider interesting)
  3. Your alpha level for your significance test (typically .05)

Software options

• A nice free GUI for calculating statistical power is G Power 3 available on Mac and Windows. I have an explanation of basic use with some sample power analysis graphs relevant to your example (standardised group mean differences).

• R also has options as summarised on the quick-r page on power analysis. Check out pwr.t.test in the pwr package for one option.

• The basic rule is that more participants is always better, and that any power analysis rests on assumptions about population effect which are unknown (if they were known, you wouldn't need to do a study).

R Example:

> install.packages("pwr")
> library(pwr)
> pwr.t.test(power= .8, d=.5, sig.level=.05, type="two.sample")

     Two-sample t test power calculation    
              n = 63.76561
              d = 0.5
      sig.level = 0.05
          power = 0.8
    alternative = two.sided    
 NOTE: n is number in *each* group 

Thus, assuming what is often labeled as a medium difference in group means and conventional values for alpha and power, you would need 64 participants per group.

Graph from G Power

The following graph was generated from G Power and is taken from my post on power analysis.

It shows for different levels of d, what power you will obtain for a given total sample size.

Final complication if non-experimental design

The above calculations are all done on the assumption that you have equal numbers of facebook and non-facebook users in your sample. This feature is common to experiments and to studies where participants are sampled in some systematic way. However, if you are just taking a general sample from the community, you will end up with uneven numbers of facebook and non-facebook users.

All else being equal statistical power decreases as group sizes become less equal. G Power 3 allows you to specify the ratio of group sizes when calculating required sample size.

Answer 2

Here is a link to a website that do that for you, but it is in german... :-\ But mybe you can use it...

http://www.bauinfoconsult.de/Stichproben_Rechner.html

This site calculates the number of needed respondents according to the standard error and the expected power.

Answer 3

Here's an English version

Questionnaire or survey sample size How do you work out what sample size to use for your survey? It is actually a complex calculation. And consequently, in my experience, people use rules of thumb - like 10%. Such rules of thumb cannot hope to give an adequate estimate of the needed sample size and consequently people either under-sample or over-sample. Often these samples are vastly too small - causing inappropriate decisions - or much too large - a wasted effort and expense.

What sample size should you take? The answer is a balance between your intolerance for 'false positives' and 'false negatives'.

You can also try the Sample Size Calculator at gpra.net - here's the Intro and Install PDFs

Answer 4

Lehr's rule, as quoted by Van Belle is $$n = \frac{16}{\Delta^2},$$ where $\Delta$ is the posited effect size, which in your case would be $(\mu_{\mbox{fb}} - \mu_{\mbox{non fb}}) / \sigma$, where $\mu$ is the mean 'sportiness', and $\sigma$ is the (pooled) standard deviation of sportiness. You want to collect $n/2$ participants from Facebook and the remaining half not from facebook.

This rule gives you approximately 80% power for a 2-sample 2-sided t-test at the 0.05 type I rate.