Practical power analysis on linear model

Question

Power analysis for linear model test asks for an analytical expression of the power of a linear model. The answer claims that

Note that this depends on the correlation between $X$ and $T$, $r_{TX}$.

In practice, when running the power analysis of an experiment, prior to the experiment, you don't really know this (the treatment has not been assigned yet, you don't have data labelled on treatment and control). At most, you have historical data with the covariate X, the outcome Y and a way to randomize the treatment.

How do we calculate $r_{TX}$ in practice? Do we simulate a specific randomization and calculate power with that randomization? Do we simulate many and average the power? Is there a standard way of doing this?

Also, how do we estimate $\sigma$ and $\sigma_T$ in practice?

Answer 1

As Stephan Kolassa said in a comment, simulation is the best way to proceed. With modern computing technology it's reasonably straightforward to use your understanding of the subject matter to evaluate a wide range of scenarios.

Analytical formulas for power size date back to times when data crunching was done by hand on mechanical calculators. That was before digital computers were commercially available, long before electronic calculators like those produced by Wang or Hewlett-Packard became available to individuals in the 1960s, and even longer before personal computers extended major computing capacity to a wide audience. At that time those analytical power formulas were developed, that was about the best one could do.

Although analytical power formulas provide general guidance, the large level of uncertainty about the values of the parameters in the formula means that it's wise to evaluate a range of possibilities in study design. See this page, for example, about the difficulty simply in evaluating the value of a linear-regression error $\sigma^2$ from a limited data set. You want to be covered in case your initial estimate is incorrect.

Simulation allows you to handle more complicated situations like the crossover, stratified, and clustered designs that your mention in a comment. If your outcome isn't a simple continuous value as in a linear regression, say a binomial or survival outcome, simulation is even more likely to prevent you from being led astray.

The simulations provide information about the types of results that you might end up finding. You then have to apply judgment to choose the sample size based on that information.