# Is the variance of the residuals of a linear regression useful for estimating experimental sample sizes?

I have a data set of $y$ values that is not particularly normally distributed. However, the $y$s do partially depend on several other parameters. A linear regression model $y=c+\mathbf{ \beta x}+\epsilon$ fits the data decently and the residuals look like they're normally distributed, have uniform variance vs. fitted values, etc.

The data I have is just happenstance historical data. Say I think another parameter $q$ might affect $y$, but I don't have any data where $q$ varies. I will have to collect more data, hopefully this time with an actual experiment where I do my best to control for everything that isn't $q$, including the $x$s. I could observe some $y$ values for "low" $q$ and some for "high" $q$, while holding $\mathbf{x}$ constant and in general doing whatever else is necessary to make a valid experiment. I could then test the null hypothesis that the mean of $y$ given low $q$ is the same as the mean of $y$ given high $q$. In order to decide what statistical test to use or how many observations I need to make, I need to assume some things like the distribution of my $y$ values, how much random variation I expect in $y$, how much I expect $q$ to affect $y$, etc.

My question is whether or not the fact that I have historical data and some regression results helps me fill in those assumptions. Does the fact that the residuals are approximately normally distributed mean that it's reasonable to plan to use a test that assumes normal distributions, such as the t-test? (My reasoning would be that I expect the distribution of $y$ given some $\mathbf{x}$ to be normal if the model residuals appear normal.) Would it be reasonable to use the variance of the residuals when doing a power analysis for such an experiment? It seems like the variance of the model residuals a more reasonable number to use for "expected variance" that would the variance of $y$ in the raw data set, because some of that variation is explained by $\mathbf{x}$. But I don't know if there is an even more reasonable number to use.

Finally, if I knew exactly which values of $\mathbf{x}$ I was going to hold constant, would it be better to use the standard error of the forecast at $\mathbf{x}$ - i.e. what I would get from Stata with predict std_err_fcast, stdf? (In this particular case, the forecast standard errors for the existing observations are all quite close to the standard deviation of the residuals, so the distinction might not matter much for scoping out roughly how many new observations I need when there are still other unknowns as well.)

• You have described a lot of useful things--except for the most important, which is the purpose of your additional experiment. Presumably it will be testing some hypothesis. What is it? – whuber Jun 5 '14 at 19:16
• @whuber, I tried to answer your question with an edit to my own. Basically, my new observations would be meant to test a hypothesis that another variable affects y. I phased this as a new variable q, but it could also be one of my xs at a values outside the range it's been seen historically. I know that in order to design any experiment that I need to know or assume something about the random variation that will be present. What I'm wondering is how my regression results help me fill in the necessary assumptions. Thanks. – jtolle Jun 5 '14 at 21:25