2
$\begingroup$

I have a question that may be fairly easy to answer but it is making my head spin. If I do a paired t test on two groups of data with sample size 3 each AND I assume that the pairs are correlated with at least an correlation coefficient of 0.5, then I should have about 90% power to detect an effect size of 2SD.

Ok, so lets say that after the experiment, I check my sample correlation and it turns out that it is only 0.2 for that sample and not 0.5. I am wondering how this would affect the power of my test since, I believe, the sample correlation value influences my p-value. Would I then have to say that I actually had less than 90% power? Or did my test still have 90% power regardless of how the sample correlation turned out? I am thinking it is the latter rather than the former but I would appreciate some help as this whole business of calculating power after a test is very confusing. Hopefully my question makes sense.

$\endgroup$
6
  • $\begingroup$ With a correlation of 0.5 and d of 2 then the n for 90% power is 5. It's unclear what you would mean by effect size. Perhaps it's better to state the mean and sd of each condition along with the correlation. Or, just change n in your example to 5. $\endgroup$ – John Nov 1 '13 at 13:15
  • $\begingroup$ A sample correlation of 0.2 is not very far away from the assumed true correlation of 0.5 given that it is based on just three observations. @John: I suppose that here, the effect size is measured in standard deviations of the values within group, not of the differences between groups. That's why correlations are relevant. $\endgroup$ – Michael M Nov 1 '13 at 13:15
  • $\begingroup$ @ John: Are you sure about that value of n? This example seems to say otherwise. stats.stackexchange.com/questions/71525/…. Although I guess I didn't really say that my value of alpha is 0.1. Were you thinking 0.05? $\endgroup$ – Jimj Nov 1 '13 at 13:25
  • $\begingroup$ Typically an unstated alpha is assumed to be 0.05. With an alpha of 0.1 and n of 3 the power is still only 0.7. You are not looking at the table in your linked post carefully. You can't run half a subject so you need at least 4 to guarantee a minimum power of 0.9. However, if alpha were the typical 0.05 then the N would be 5. $\endgroup$ – John Nov 1 '13 at 22:29
  • $\begingroup$ I think the table in the link uses an alpha of 0.05 though. $\endgroup$ – Jimj Nov 1 '13 at 22:40
5
$\begingroup$

Power is an attribute of a test that depends upon the true parameter values, not the estimated parameter values. It may be that the assumptions about the true parameter values that you made when calculating power are contraindicated by the data, but that does not affect the power as such, since that is specific to the null/alternative hypotheses used in the calculation. It is important to remember that power calculations take into account the randomness of the sample; in your case, for example, a power calculation based upon a true correlation of 0.5 does not assume that the sample correlation will always be 0.5.

Consequently, your thinking is correct. If your assumptions are correct, the test still has 90% power. As Michael Meyer has pointed out in comments, given your sample size, it is not even the case that your sample correlation indicates that your assumptions are false, which means there isn't any reason to revisit your power calculations.

$\endgroup$
4
$\begingroup$

Power is the probability of finding a significant effect in a sample if what you believed true about the population is true. Correlation, effect, and variance all change from sample to sample. That's why power is expressed as a probability. It's sensitive to these variations.

Here's a simulation in R that demonstrates what I'm saying. It generates data from the population you expect in your question. But each sample has a different correlation, variance in groups, and effect.

library(MASS)
# mvrnorm requires a variance, covariance matrix
# with variances of 1 then the covariances will equal the correlation
sigma <- matrix(c(1.0, 0.5,
                  0.5, 1.0), 2, byrow = TRUE)
y <- replicate(1000, {
     mat <- mvrnorm(3, c(0, 2*sqrt(2)), sigma, empirical = FALSE) # get a sample
     tt <- t.test(mat[,1], mat[,2], paired = TRUE, var.equal = TRUE)
     tt$p.value })  #return p-value of t-test

sum( y < 0.1 ) / 1000

The typical result is that about 90% of the tests pass a significance test.

[The alpha used is 0.1 from the comments and the effect size is diff / (pooledSD / sqrt(1-r)) based on comments as well. None of that's particularly germaine to the basic story but I note it because there are arguments to use different effect size calculations and this is an atypical significance cutoff.]

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.