4
$\begingroup$

I attended a training class from SAS about experimental design in marketing. They advocated the use of their GLMPOWER proc for power analysis for designing experiments.

GLMPOWER is a power analysis procedure for ascertaining the required sample size for a general linear model (main effects, interactions and/or specific contrasts between factor levels).

For this class, interest rests on designing experiments with regards to binary outcomes (response or no response) analyzed using logistic regression. Given the large sample sizes typically witnessed in marketing, they said an approximation to this problem could be handled by GLMPOWER, which assumes the response is a Gaussian distributed continuous variable.

One of the parameters in the GLMPOWER procedure is a standard deviation STDDEV defined as "the error standard deviation, or root MSE" of the model being postulated.

Questions:

  1. This error standard deviation is not the same as the standard deviation of "Y" is it? They state that it is.

  2. Further, after stating this, they use SQRT(p(1-p)) as an estimate of this value, where p is the pooled response rate (number of responses / number of attempts) of the experiment. Where does this come from and does it sound like the right value to use?

It was well stated that these are approximations only given the assumptions of GLMPOWER for a binary outcome, but can anyone help with the reasoning?

$\endgroup$
4
$\begingroup$

Personally I dislike the method suggested (though @Michael Chernick did a good job of describing why it gives an approximation). In my mind this requires to many assumptions and approximations. In a logistic regression the variance varies with the mean, but in a gaussian regression the variance is assumed constant.

Instead I would suggest using simulations, then you can use the exact same methods that will be used in the analysis and you know exactly what assumptions you are making about the data (including predictor variables). Basically you simulate data under the conditions you expect to see, then analyze it and see if you get significance for the primary test of interest. Repeat this a bunch of times and the proportion of times that the null is rejected is your power.

I think this would be easiest in R, but could be done in SAS using macros to do the multiple replications, the data step to generate the data, and proc glm to analyze. Or it may be easier to use proc IML for parts. There may be other tools in SAS to make this even easier, I just don't use SAS that much recently.

$\endgroup$
2
$\begingroup$

The probability model they are assuming for the binary outcome is the Bernoulli distRibution with success proportion p. The Bernoulli random variable has

variance = p(1-p).

The sum of n independent identically distributed random variables is the binomial with parameters n and p. The average of the n Bernoulli variable is an unbiased and maximum likelihood estimate for p. Its distribution is determined by the binomial. By the central limit theorem this estimate is approximately normal for large n because it is a sample mean whose distribution satisfies the necessary conditions. Now the theory tells us that the estimates variance is p(1-p)/n. So it makes sense to apply this formula with the estimate for p plugged in. This is better than simply using the sample standard error estimate which doesn't take account of the special form of the variance for the binomial. So a hypothesis test on the proportion can be made using the normal distribution approximation. That explains 2. Regarding 1, the error standard deviation is the standard deviation of the model residuals and not the standard deviation for the Bernoulli variable Y.

$\endgroup$
  • $\begingroup$ They use p(1-p) (omitting the n). Would this likely be because when analyzing the data in a logistic regression, each "unit" or "record " in the data is an individual (so n=1) and the data is distributed Bernoulli? $\endgroup$ – B_Miner Sep 8 '12 at 15:50
  • $\begingroup$ But, even so, as you state, that the error standard deviation of the model residuals is NOT the same as the standard deviation of the Bernoulli r.v. Y so....does it make sense to use p(1-p) for this purpose? $\endgroup$ – B_Miner Sep 8 '12 at 15:53
  • $\begingroup$ Note: I should have said they use SQRT(p(1-p)). I corrected in the question. $\endgroup$ – B_Miner Sep 8 '12 at 16:04
  • $\begingroup$ GLMPOWER determines the minimum sample size to achieve say 80% power for a test that the group means differ. This involves comparing group proportions in your case. For the Gaussian general linear model that is being used to approximate the model for the proportions the pooled variance is a key parameter in determining what degrees of freedom (group sample sizes) are needed for the F statistic. That would be the pooled variance of the Bernoulli variables which would have variance p(1-p) where p is your presumed common value for the proportion as SAS stated to you. $\endgroup$ – Michael Chernick Sep 8 '12 at 16:10
  • 1
    $\begingroup$ Thanks Michael. I think for me (and I am sure it is just my lack of knowledge) the remaining difficulty is that it seems we are using the variance of the independent Bernoulli r.v. (the Y) for an estimate of the average error from the model. It seems this would be the same as using the variance of the Y when Y is truly continuous - which we said is NOT the same as the root MSE - which is the parameter called for by the procedure. Do you see what I am struggling with? $\endgroup$ – B_Miner Sep 8 '12 at 16:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.