Logistic regression: the standard deviation used in: GLMPOWER

Question

I attended a training class from SAS about experimental design in marketing. They advocated the use of their GLMPOWER proc for power analysis for designing experiments.

GLMPOWER is a power analysis procedure for ascertaining the required sample size for a general linear model (main effects, interactions and/or specific contrasts between factor levels).

For this class, interest rests on designing experiments with regards to binary outcomes (response or no response) analyzed using logistic regression. Given the large sample sizes typically witnessed in marketing, they said an approximation to this problem could be handled by GLMPOWER, which assumes the response is a Gaussian distributed continuous variable.

One of the parameters in the GLMPOWER procedure is a standard deviation STDDEV defined as "the error standard deviation, or root MSE" of the model being postulated.

Questions:

1. This error standard deviation is not the same as the standard deviation of "Y" is it? They state that it is.

2. Further, after stating this, they use SQRT(p(1-p)) as an estimate of this value, where p is the pooled response rate (number of responses / number of attempts) of the experiment. Where does this come from and does it sound like the right value to use?

It was well stated that these are approximations only given the assumptions of GLMPOWER for a binary outcome, but can anyone help with the reasoning?

Answer 1

Personally I dislike the method suggested (though @Michael Chernick did a good job of describing why it gives an approximation). In my mind this requires to many assumptions and approximations. In a logistic regression the variance varies with the mean, but in a gaussian regression the variance is assumed constant.

Instead I would suggest using simulations, then you can use the exact same methods that will be used in the analysis and you know exactly what assumptions you are making about the data (including predictor variables). Basically you simulate data under the conditions you expect to see, then analyze it and see if you get significance for the primary test of interest. Repeat this a bunch of times and the proportion of times that the null is rejected is your power.

I think this would be easiest in R, but could be done in SAS using macros to do the multiple replications, the data step to generate the data, and proc glm to analyze. Or it may be easier to use proc IML for parts. There may be other tools in SAS to make this even easier, I just don't use SAS that much recently.

Answer 2

The probability model they are assuming for the binary outcome is the Bernoulli distRibution with success proportion p. The Bernoulli random variable has

variance = p(1-p).

The sum of n independent identically distributed random variables is the binomial with parameters n and p. The average of the n Bernoulli variable is an unbiased and maximum likelihood estimate for p. Its distribution is determined by the binomial. By the central limit theorem this estimate is approximately normal for large n because it is a sample mean whose distribution satisfies the necessary conditions. Now the theory tells us that the estimates variance is p(1-p)/n. So it makes sense to apply this formula with the estimate for p plugged in. This is better than simply using the sample standard error estimate which doesn't take account of the special form of the variance for the binomial. So a hypothesis test on the proportion can be made using the normal distribution approximation. That explains 2. Regarding 1, the error standard deviation is the standard deviation of the model residuals and not the standard deviation for the Bernoulli variable Y.