5
$\begingroup$

My question in short:

What is the current practice regarding Power Analysis in a multiple regression framework when we are interested in a single coefficient (which may or may not be dummy / interaction term)? In particular, how I do calculate required sample sizes?

Lengthy version:

When we test equality of a simple mean with a standard t-test, the corresponding formula contains $n$, which makes it easy to calculate the power or to solve for $n$ for a given power. In a regression framework, I have $\hat\beta$ and I have $\hat\sigma_\beta$, the estimated standard error of the coefficient. In a homoskedastic case, $\hat\sigma^2_\beta = \hat\sigma^2_\epsilon (X'X)^{-1} $. If $X$ contains a constant, then there will be a division by $n$ involved somewhere, so this could be worked out. However, this seems less obvious in case we adjust the variance-covariance matrix for potential heteroskedasticity or clustering. Googling around the standard reference appears to be Cohen (1988), and there is even a nice R package called pwr that has implemented a couple of power calculations based on this reference. Chapter 9 of Cohen (1988) is framed in terms of the F-test, and the power in the regression framework is ascertained in terms of the $R^2$. For instance, effect sizes ("$f^2$") are defined as $f^2 = \frac{R^2}{1-R^2}$ or $f^2 = \frac{R_{AB}^2-R_{B}^2}{1-R_{B}^2}$, where in the latter $A,B$ indicate different sets of regressors, and everything is framed in terms of variance explained.

My main concern: Since 1988, statistical practice at least in economics and also in other fields has changed. The major differences, as far as I can tell, are these: first, we compute robust or cluster-robust standard errors by default. This naturally inflates standard errors, which makes it harder to reject the null in standard t-tests. Second, in small samples, we are typically worried about distributional assumptions, and violations thereof. Third, many people now work with quasi-experimental methods, where variance explained or $R^2$-values are typically tiny, and $R_{AB}^2-R_{B}^2$ is typically close to zero.

I'd imagine that this has implications for Power Analysis and the determination of the required sample size. I think this is a methodologically really interesting and also important question. I wonder what people who are more up to date are would do, which is why I start a bounty.

Illustration of the question:

Imagine you ran the following regression:

library(sandwich)
library(lmtest)
mod <- lm(mpg ~ disp + drat + wt*qsec, data=mtcars)
coeftest(mod, vcov. = vcovHC(mod))

t test of coefficients:

               Estimate  Std. Error t value Pr(>|t|)
(Intercept) -13.3238114  51.2643074 -0.2599   0.7970
disp          0.0026224   0.0113389  0.2313   0.8189
drat          1.5662444   1.4498325  1.0803   0.2899
wt            3.1612129  16.6582389  0.1898   0.8510
qsec          2.3617536   2.8229593  0.8366   0.4104
wt:qsec      -0.4402128   0.8979944 -0.4902   0.6281

You see that the coefficient on, say, wt:qsec is insignificant, but imagine you had a strong prior that it would be important. Imagine you wondered if there truly is no effect, or if the sample size is merely too small. How can we calculate the power of this test, or, correspondingly how can we calculate the sample size required to detect an effect of similar size?

Importantly, notice that standard errrors in the above regression are computed with a variance-covariance matrix robust to heteroskedasticity of unknown form, which reflects the standard practice in many social science fields nowadays. This is markedly different from homoskedastic standard errors. You can run summary(mod) yourself to verify this.

$\endgroup$
  • 1
    $\begingroup$ you could try simulation. $\endgroup$ – kjetil b halvorsen Nov 5 '16 at 20:12
  • $\begingroup$ Is your choice variable simply $n$, the number of observations? Or are the regressors $X$ (or some subset of the regressors) chosen by you? $\endgroup$ – Matthew Gunn Nov 6 '16 at 0:55
  • $\begingroup$ The choice variable is $n$. The regressors are needed to control for confounding factors and will stay the same no matter what, i.e. there is no variable selection involved. $\endgroup$ – coffeinjunky Nov 6 '16 at 0:58
  • $\begingroup$ Something to consider is block matrix inversion - you can show the variance for a coefficient will look like $var (\beta_j)=\frac {\sigma_e^2}{\sum_ix_{ij}^2-a_j^T (X_{-j}^TX_{-j})^{-1}a_j} $ where $X_{-j} $ is design matrix without column $j $ and $a_j $ is the jth column of $ X^TX$ with row j removed. $\endgroup$ – probabilityislogic Nov 7 '16 at 21:35
  • $\begingroup$ "[...] wt:qsec is insignificant, but imagine you had a strong prior that it would be important. Imagine you wondered if there truly is no effect, or if the sample size is merely too small." Thinking like this can mislead you. Gelman et al, link quote "After data have been collected, and a result is in hand, statistical authorities commonly recommend against per forming power calculations (see, e.g., Goodman & Berlin, 1994; Lenth, 2007; Senn, 2002). Hoenig and Heisey (2001) " $\endgroup$ – CloseToC Nov 7 '16 at 22:42
1
+100
$\begingroup$

There are two main approaches to power analysis:

When your design conforms to "classical standards" with regard to estimators used and distributional assumptions, then the formulae from Cohen (most of which are much older than that reference) are mathematically correct, provably so.

When your design starts to depart from these standards, either because you are using nonstandard estimators (for whatever reason) or because there are other wrinkles with your data generation or selection process, theory generally breaks down quickly. Whilst there do exist formulae for a few cases which are very close to the classical paradigm, the normal approach is simulation. If you believe your effect is of a certain magnitude then simulate, say, 10000 datasets of a given sample size, with this magnitude of effect. Apply your chosen estimator to each of these datasets, and see how many return a significant result. Then, adjust the sample size to suit your needs (if not enough of the replicates are significant, you should increase the sample size. If more were significant than were required, you can get away with reducing it.)

$\endgroup$
  • $\begingroup$ FWIW, I had to talk a little about power analysis/sample size estimation during teaching earlier this year and suggested there were three main approaches: Formulas for "classical standards", simulation, and rules of thumb (e.g., 10 observations per predictor in a linear regression, etc.). These "experiential" approaches are obviously of dubious merit, but it's not uncommon for them to be used. $\endgroup$ – Ian_Fin Nov 8 '16 at 14:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.