Is this treatment for a very small sample size appropriate?

Question

I am doing a lit review and I am wondering about a study with 12 participants (repeated measures ANOVA with Posh-hoc Bonferroni CI adjustment). There was no control group due to the small n. The authors state:

"Given the exploratory nature of the study with a very small sample size, a nominal error rate of 0.05 instead of the Bonferroni-adjusted Type I error rate 0.007 was used to identify any trends to guide further research investigations with a much larger sample. It was also due to these constraints that multivariate analysis of variance was not used"

Significance was detected at this level, and I just wonder if this was an appropriate action to take. Thanks in advance.

Answer 1

If I am understanding correctly, you are making sure it was appropriate to use an alpha level (error rate) at 0.05.

If this is the case, I will say that it does appear to be completely appropriate. The reason why is because in Statistics, using an alpha level of .05 is typically the standard, and it makes computations relatively simple.

Now an alpha level (in your words error rate), is the probability of rejecting the null hypothesis when the null hypothesis is true (also known as Type 1 error rate). The smaller your alpha level is in your test, the smaller the area where you would reject the null hypothesis. Obviously the smaller your level, the larger the chance that you will not reject the null hypothesis. So in the case of your question, I see that this was completely appropriate!

Hopefully I answered your question!

Answer 2

As long as they state accurately what they did and as long as it doesn't violate any of the assumptions, it's fine.

The use of 0.05 is arbitrary and, in any case, effect size is usually more interesting than a significance test.