3
$\begingroup$

Let's say I have an experiment with a 2x2 repeated-measures design. Let's call the four cells A1B1, A1B2, A2B1, A2B2, for the factors A and B, each with two conditions, labeled 1 and 2.

I want to compute the power needed for a main effect of A, and for an interaction, reflecting a moderation of the effect of A in different levels of B.

I plan to analyze the data using a repeated measures ANOVA.

My question is about the implications of using a power analysis for t-test in order to determine the sample size. (I am putting aside the question of how to determine the desired effect size; let's say I know it is d=0.2).

To be specific, for the main effect of factor A, I will run a power analysis for a dependent-samples t-test that will compare the average of A1 and the average of A2. For the interaction, I will run a power analysis for a dependent-samples t-test that will compare the average of A2B1-A1B1 and the average of A2B2-A1B2.

Is there any risk of underestimation of the required sample-size?

The more general question is about this method for other designs as well. Any effect in an ANOVA can be tested in a t-test. That is not recommended, of course, but for the power-analysis, is there any risk of underestimating the required sample size in computing the power for that t-test?

Improve this question
$\endgroup$
2
  • $\begingroup$ ANOVA uses an F-test, so the two are not compatible. $\endgroup$
    – user2974951
    Commented Feb 15, 2019 at 8:30
  • 1
    $\begingroup$ Given that $t^2 = F$, it's not clear why "uses an F-test" would be dispositive of itself.. I think a deeper argument would be needed here. $\endgroup$
    – Glen_b
    Commented May 13, 2022 at 2:52

2 Answers 2

Reset to default
1
$\begingroup$

You want to analyse the data using a repeated measures ANOVA because you think that fits the structure of your data. Two paired-sample t-tests would ignore some of the structure in your design - and therefore some of the information in your data.

A paired t-test only tests for one effect at once (are the differences significantly different from 0), whereas the ANOVA takes all of the information that you have about the variance into account when testing for the main effects and the interaction. So I don't think that the t-tests are equivalent to the ANOVA. The power analysis using t-tests is therefore unlikely to give you an accurate answer.

I'm not sure of the risk of underestimating, but it seems far preferable to use a power analysis for the test that you are actually planning to use. There is an example here of estimating sample size for a repeated measures ANOVA using simulation.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for the answer. I am still interested in the answer to my specific question. Does anyone know whether there is any risk of underestimation of the sample size? $\endgroup$
    – YBA
    Commented Feb 15, 2019 at 15:02
  • $\begingroup$ Please can you provide a dummy ANOVA results table showing the degrees of freedom... including how many residual degrees of freedom you will have? I think that might answer your own question, as I think the individual paired t-tests will overestimate the residual degrees of freedom for a given sample size (therefore giving you an underestimate of the required sample size). $\endgroup$
    – Izy
    Commented Feb 15, 2019 at 17:13
  • $\begingroup$ Also, I'm not sure how you plan to carry out the power analysis, but I might use something like this in R: power.t.test(delta=0.2,sd=0.1,sig.level=0.05,power=0.8,type="paired"). Note that you need to include an estimate of the standard deviation (the default in R is sd=1, if you don't set it you're just assuming that sd=1). Do you think that the change in factor B might increase the variance of your differences for A1-A2? If it might do, and you fail to account for that when you estimate the sd, then it is very likely that you will underestimate the required sample size. $\endgroup$
    – Izy
    Commented Feb 15, 2019 at 17:20
1
$\begingroup$

I would suggest running the numbers for both. You will see that a t-test asks for a higher n and therefore if you have the numbers right for a t-test you will certainly have them right for Anova.

library(pwr)

pwr.anova.test(k = 2,
               n = NULL,
               f = 0.2, 
               sig.level = 0.05, 
               power = 0.8)

Note the above is for a one way anova power calculator. The Pwr2 library has a two way anova calculator.

pwr.t.test(n = NULL,
           d = 0.2,
           sig.level = 0.05,
           type = "two.sample",       # options are: 
                         # "two.sample", "one.sample", "paired" 
           alternative = "two.sided"  # options are: "two.sided", 
                                      # "less", "greater"
           power = 0.8)
```
Improve this answer
$\endgroup$
1
  • $\begingroup$ In the power test for the ANOVA, it should be f = 0.1. Cohen's f is half of Cohen's d. This shows that for both these examples n = 393.4057 (or basically n = 393). $\endgroup$
    – Geraldine
    Commented Sep 20, 2022 at 19:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.