I'm trying to compute the minimum sample size for a psychometric test based on 7 point Likert scales. I'd like to run ANOVA on each scale to look for differences between groups.

Most online survey sample size calculators seem to be designed for polls, e.g. Yes/No, Agree/Disagree. They take as input population size, a confidence interval and a proportion (50% Yes/50% no) and then return the required sample size.

Most statistical books suggest using power tests (such as R's power.t.test), which take as input a minimum effect size, alpha, beta and a statistical test and then return the required sample size.

For my purposes power tests seems to make the most sense, but what has me concerned is that none of them take into account the population size, which seems like it ought to have at least some effect on the outcome.

So my question is, what is the correct calculation to use in my specific survey situation and more generally what is the connection between power tests and these online survey sample size calculators, does population size matter in some way, perhaps helping to capture the notion of representative sample?


The commonly used statistical methods assume that you take a sample of an infinite or very large population. ANOVA, too, has this assumption. When the subjects of your survey can be viewed as a representative sample of an existing or hypothetical much larger population, you do not need the finite population methods.

The second question is if ANOVA is appropriate to analyse the data collected. 7 point Likert scales are strictly speaking ordinal scales, so methods for ordinal dependent variables may be best. However, in psychometry it's usual to assume that the values from a Likert scale will follow a distribution that may be approximated with a normal distribution. In this case ANOVA is an acceptable method; the t-test too, although the latter compares two groups only. (The methods designed for binary (yes/no) outcomes may be used after setting a threshold in your Likert scale and dichotomising your data, however unless this threshold also exists in the psychological mechanism it will lead to loss of detail in your data and loss of power in your test. So not generally recommended.)

You need to check or think over if the homoscedasticity assumption of ANOVA is likely to be met. If yes, use a power formula for ANOVA and you need not worry about not having to specify the population size.

  • $\begingroup$ In my case the "population" size is typically around 100-2000, so at least at the low end, is it wrong to use tests that assume the sample is drawn from an infinite population? $\endgroup$ – Curried Lambda Apr 11 '11 at 3:01
  • 2
    $\begingroup$ Google search found a JSTOR paper for linear models in finite populations. Substituting your figures should decide if the difference between the finite and infinite population setting is negligible or not. $\endgroup$ – GaBorgulya Apr 11 '11 at 11:05
  • $\begingroup$ Interestingly this question is the top hit when googling "anova assumes infinite population". $\endgroup$ – Curried Lambda Apr 12 '11 at 5:59

1. Power analysis for one-way ANOVA:

Download G-Power 3. It allows you to do for a range of statistical tests including ANOVA

  • a priori power analysis (sample required given effect size, desired power, and alpha), and
  • post hoc power analysis (observed power given sample size, effect size, and alpha)

Here's a G Power 3 tutorial for ANOVA.

2. Power analysis and Likert items

Assuming that the 7-point Likert item is a continuous variable might be a useful simplifying assumption. This is especially true given that assumptions made about effect size for power analysis are only guesses (if they weren't, then you wouldn't need to do the study).

3. Finite populations

Also, even when there is a finite population, you are often wanting to determine whether any differences reflect a theoretically meaningful difference or just random sampling. For example, assume that you measured the whole population of both groups, and you found that group A had a mean of 6.12, and group B had a mean of 6.11. You could conclude that there is a difference between the groups. Alternatively, you could assume that the participants in each group are drawn from some theoretical data generating process, and that you are interested in drawing inferences about this. Then, you would apply standard t-tests or ANOVAs to test whether the observed difference is statistically significant. If this was your purpose, you would not need to apply corrections for finite populations.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.