Is the variance of the residuals of a linear regression useful for estimating experimental sample sizes?

Question

I have a data set of $y$ values that is not particularly normally distributed. However, the $y$s do partially depend on several other parameters. A linear regression model $y=c+\mathbf{ \beta x}+\epsilon$ fits the data decently and the residuals look like they're normally distributed, have uniform variance vs. fitted values, etc.

[Edited to answer whuber's question from comments]

The data I have is just happenstance historical data. Say I think another parameter $q$ might affect $y$, but I don't have any data where $q$ varies. I will have to collect more data, hopefully this time with an actual experiment where I do my best to control for everything that isn't $q$, including the $x$s. I could observe some $y$ values for "low" $q$ and some for "high" $q$, while holding $\mathbf{x}$ constant and in general doing whatever else is necessary to make a valid experiment. I could then test the null hypothesis that the mean of $y$ given low $q$ is the same as the mean of $y$ given high $q$. In order to decide what statistical test to use or how many observations I need to make, I need to assume some things like the distribution of my $y$ values, how much random variation I expect in $y$, how much I expect $q$ to affect $y$, etc.

My question is whether or not the fact that I have historical data and some regression results helps me fill in those assumptions. Does the fact that the residuals are approximately normally distributed mean that it's reasonable to plan to use a test that assumes normal distributions, such as the t-test? (My reasoning would be that I expect the distribution of $y$ given some $\mathbf{x}$ to be normal if the model residuals appear normal.) Would it be reasonable to use the variance of the residuals when doing a power analysis for such an experiment? It seems like the variance of the model residuals a more reasonable number to use for "expected variance" that would the variance of $y$ in the raw data set, because some of that variation is explained by $\mathbf{x}$. But I don't know if there is an even more reasonable number to use.

Finally, if I knew exactly which values of $\mathbf{x}$ I was going to hold constant, would it be better to use the standard error of the forecast at $\mathbf{x}$ - i.e. what I would get from Stata with predict std_err_fcast, stdf? (In this particular case, the forecast standard errors for the existing observations are all quite close to the standard deviation of the residuals, so the distinction might not matter much for scoping out roughly how many new observations I need when there are still other unknowns as well.)

Answer 1

For this part of your question: 'Would it be reasonable to use the variance of the residuals when doing a power analysis for such an experiment? It seems like the variance of the model residuals a more reasonable number to use for "expected variance" that would the variance of y in the raw data set, because some of that variation is explained by x.'

I've been reviewing literature to try and answer this same question. It turns out that sample size prediction/power analysis for multivariate regression (which your problem would become with the addition of your q predictor variable) is known to be complicated. I have a couple of leads for you:

1. Yes, one paper did use the variation of residuals within a power analysis (an exciting find!): "The degree of dispersion of the response values about the regression line affects power and sample size calculations. A parameter that quantifies this dispersion is [sigma], the standard deviation of the regression errors." However, they go on to list other parameters that are also important, and the resulting "n=" equation became complicated. It also seems to be intended for univariate linear regressions. Unfortunately, the program they wrote to carry it out no longer exists. (Dupont & Plummer, 1998)
2. If you have a lot of time on your hands and/or understand what it means to iteratively calculate "principal diagonal elements" of population correlation matrices (I don't), have fun with Kelley & Maxwell 2003.
3. Since your covariate X is normal (my X's sadly aren't), you could go down the path recommended by Hsieh, Bloch & Larson 1998. It does also get sort of complicated, though less so than Kelley & Maxwell. (I can't use more than 2 links per post, so search for the authors + "A simple method of sample size calculation")
4. If you use SAS, they have a Proc Power procedure for multivariate regression that uses R-squared values, power, and alpha to determine N. Unfortunately for me, it also assumes normality, but you could try it out.

Personally, I'm considering hand-calculating a standard power equation, which is Z-score based and therefore assumes normality and symmetrical variance; however, instead of using the pilot Y variable's mean and standard deviation, I'll input the mean and SD of my normally-distributed residuals from my pilot multivariate model. Maybe that's what you were thinking. I haven't seen it done before, but it does seem better justified than using Y values' mean and SD only, and the idea is backed up in basic concept by Dupont & Plummer.