Figuring out how many participants I need

Question

I have a test of a skill which will return a value between 0 and 30 for a participant as a measure of that skill. I have five ordinal groups of participants (they are corresponding to university level attainment - first years, honours students, PhD students, etc) and I intend to have a number of participants from each group take the test. I then intend on performing an ANOVA to investigate the difference in skill level across each group.

Although I have an idea of what I want to do, I want to be clear on a few of things:

1. How can I determine that my results are statistically significant?
2. How many participants should I take from each group (which are of varying sizes) to ensure that my results are reliable?
3. How would I go about calculating the statistical power of the experiment?

Answer 1

1. With an appropriate significance test. Any introductory statistics textbook can help you choose null hypotheses and tests for them. If you were hoping for specific advice on that, you need to provide much more detail.

2. I guess that by "reliable" you mean to refer to the uncertainity in your conclusions. The uncertainty of an estimate can be quantified with tools such as confidence intervals and Bayesian probability, but these tools need data.* You can't tell in advance how precise your estimates are going to be, because that depends on the data. If you collect at least some data, you can estimate the variability in your variables of interest and hence get a sense of what sort of sample sizes will get you what sort of precision. In any case, though, you generally want as large a sample as you can feasibly obtain, at least in human-subjects research, where within-group variance tends to be large.

3. You can't, because power depends on the true population effect size, which is unobservable. The closest you can get is a power analysis, which tells you what the power would be if the population effect size were equal to some value you select. Power analysis isn't very useful in practice because estimating an effect size is typically the whole point of a study.

* Technically, you can do a fully Bayesian analysis with no data, but all this will do is reproduce your preexisting beliefs, of course.

Answer 2

It sounds like you are intending to test the hypothesis, "Does skill level on the test increase with increasing educational attainment?" This corresponds to the null hypothesis: avg skill of ed. level 1 = avg skill of ed. level 2 = avg skill of ed. level 3 = avg skill of ed. level 4 = avg skill of ed. level 5.

To answer your first question, a 1-way ANOVA will test whether there is a difference amongst these groups, but will not capture the ordinal nature of the educational attainment. An alternative approach would be to treat educational attainment as continuous and use a linear regression.

For your second question, let me clarify: you have not yet gathered data, correct? You are asking how many participants you need in your experiment. For this you need to conduct a power analysis. For a linear regression that is a little tricky to do, but for a 1-way ANOVA it's pretty easy. If you do not have access to statistical software there are a number of online calculators. One I found quickly is: https://www.anzmtg.org/stats/PowerCalculator/PowerANOVA

Your third question is just a variation on the second question. When you do the power analysis you will find how many subjects you need for your desired power level.

To do the power analysis you will need to specify the standard deviation you expect and the difference in mean score which you are testing for. The difference in mean score is easier because you should be able to know what type of difference in your scale is meaningful. If you don't know the variation, it would be a good idea for you to try out the scale on some test subjects first (a mix of educational levels) and calculate the mean and standard deviation. You can use this for the power calculation. Alternatively you can just use an effect size in the calculation, meaning that for a strong effect (effect of 1.0 or larger) you need the change in mean to be proportional to the standard deviation. A moderate effect is 0.5, meaning the change in mean is half the standard deviation. Effect sizes of <0.5 are usually considered small.

Good luck!