Is the post hoc power of a dependant t test related to the sample correlation or actual correlation

Question

I have a question that may be fairly easy to answer but it is making my head spin. If I do a paired t test on two groups of data with sample size 3 each AND I assume that the pairs are correlated with at least an correlation coefficient of 0.5, then I should have about 90% power to detect an effect size of 2SD.

Ok, so lets say that after the experiment, I check my sample correlation and it turns out that it is only 0.2 for that sample and not 0.5. I am wondering how this would affect the power of my test since, I believe, the sample correlation value influences my p-value. Would I then have to say that I actually had less than 90% power? Or did my test still have 90% power regardless of how the sample correlation turned out? I am thinking it is the latter rather than the former but I would appreciate some help as this whole business of calculating power after a test is very confusing. Hopefully my question makes sense.

Answer 1

Power is an attribute of a test that depends upon the true parameter values, not the estimated parameter values. It may be that the assumptions about the true parameter values that you made when calculating power are contraindicated by the data, but that does not affect the power as such, since that is specific to the null/alternative hypotheses used in the calculation. It is important to remember that power calculations take into account the randomness of the sample; in your case, for example, a power calculation based upon a true correlation of 0.5 does not assume that the sample correlation will always be 0.5.

Consequently, your thinking is correct. If your assumptions are correct, the test still has 90% power. As Michael Meyer has pointed out in comments, given your sample size, it is not even the case that your sample correlation indicates that your assumptions are false, which means there isn't any reason to revisit your power calculations.

Answer 2

Power is the probability of finding a significant effect in a sample if what you believed true about the population is true. Correlation, effect, and variance all change from sample to sample. That's why power is expressed as a probability. It's sensitive to these variations.

Here's a simulation in R that demonstrates what I'm saying. It generates data from the population you expect in your question. But each sample has a different correlation, variance in groups, and effect.

library(MASS)
# mvrnorm requires a variance, covariance matrix
# with variances of 1 then the covariances will equal the correlation
sigma <- matrix(c(1.0, 0.5,
                  0.5, 1.0), 2, byrow = TRUE)
y <- replicate(1000, {
     mat <- mvrnorm(3, c(0, 2*sqrt(2)), sigma, empirical = FALSE) # get a sample
     tt <- t.test(mat[,1], mat[,2], paired = TRUE, var.equal = TRUE)
     tt$p.value })  #return p-value of t-test

sum( y < 0.1 ) / 1000

The typical result is that about 90% of the tests pass a significance test.

[The alpha used is 0.1 from the comments and the effect size is diff / (pooledSD / sqrt(1-r)) based on comments as well. None of that's particularly germaine to the basic story but I note it because there are arguments to use different effect size calculations and this is an atypical significance cutoff.]