Multiple logistic regression power analysis

Question

I have a logistic regression model and output an $R^2$ value. I then go and add another predictor variable to fit a second model. I can output a new $R^2$ value associated with the second model. When I run an ANOVA test, I see no significant improvement in the second model, but I want to assess the power associated with including the additional variable in model 2.

I have found an example for linear regression that uses an $F$-Test. I want to do something similar for a logistic regression using G*Power.

But there appears to be very little documentation on multiple logistic regression models like my situation. I don't know how to do a more detailed power analysis for multiple logistic regression.

From what I understand, in G*Power I set Test Family == z tests and statistical test == logisitic regression. But I am not sure what to set R² other X equal to. Is that the improvement in $R^2$?

Reading the tutorial in 27.4 from the software manual makes no variation of $R^2$, whereas this example, does not discuss the improvements made from $R^2$.

Answer 1

The problem is that there isn't really a $R^2$ for logistic regression. Instead there are many different "pseudo-$R^2$s" that may be similar to the $R^2$ from a linear model in different ways. You can get a list of some at UCLA's statistics help website here.

In addition, the effect (e.g., odds ratio) of the added variable, $x_2$, isn't sufficient to determine your power to detect that effect. It matters how $x_2$ is distributed: The more widely spread the values are, the more powerful your test, even if the odds ratio is held constant. It further matters what the correlation between $x_2$ and $x_1$ is: The more correlated they are, the more data would be required to achieve the same power.

As a result of these facts, the way I try to calculate the power in these more complicated situations is to simulate. In that vein, it may help you to read my answer here: Simulation of logistic regression power analysis - designed experiments.

Looking at G*Power's documentation, they use a method based on Hsieh, Bloch, & Larsen (1998). The idea is that you first regress $x_2$ on $x_1$ (or whatever predictor variables went into the first model) using a linear regression. You use the regular $R^2$ for that. (That value should lie in the interval $[0,\ 1]$.) It goes in the R² other X field you are referring to. Then you specify the distribution of $x_2$ in the next couple of fields (X distribution, X parm μ, and Z parm σ).

• Hsieh, F.Y., Bloch, D.A., & Larsen, M.D. (1998). A simple method of sample size calculation for linear and logistic regression. Statistics in Medicine, 17, 1623-1634.

Answer 2

The excellent book Regression Methods in Biostatistics: Linear, Logistic, Survival, and Repeated Measures Models have a treatment of power analysis for logistic regression, with some simple useful (approximate) formulas, very possibly the formulas used by GPower referred in another answer (in section 5.7.) If those approximations are not good enough, probably simulation will be needed.

Two-sided testing of $H_0\colon \beta_j=0$ (log-odds scale) versus $H_1\colon \beta_j=\beta_j^a$ with level $\alpha$ and power $\gamma$, standard deviation of predictor $x_j$ is $\sigma_{x_j}$, $p$ the marginal prevalence of the outcome and $\rho_j^2$ is the multiple correlation of $x_j$ with all the other predictors (this is the R-squared reported by a linear multiple regression of $X_j$ as response on all the other predictors, and do not involve the response in the logistic regression at all.)

The minimum sample size is then $$ n=\frac{(z_{1-\alpha/2}+z_\gamma)^2}{(\beta_j^a \sigma_{x_j})^2 p(1-p) (1-\rho_j^2)} $$ where $z_{1-\alpha/2}$ and $z_\gamma$ are quantiles of the standard normal distribution corresponding to level and power. Note the use in this formula of the variance inflation factor $\text{vif}_j=\frac1{1-\rho_j^2}$.

A graph showing minimum sample size as function of alternative coefficient $\beta_j^a$:

For completeness some related formulas from the same source:

If sample size $n$ is decided then power is $$ \gamma=1-\Phi\left(z_{1-\alpha/2}-|\beta_j^a| \sigma_x\sqrt{np(1-p)(1-\rho_j^2)}\right)$$ where $\Phi$ is the standard normal cumulative distribution function. The minimum detectable effect (on log-odds scale) is $$ \pm \beta_j^a = \frac{z_{1-\alpha/2}+z_\gamma}{\sigma_{x_j}\sqrt{np(1-p)(1-\rho_j^2)}} $$ The references given for this approximate formulas is A SIMPLE METHOD OF SAMPLE SIZE CALCULATION FOR LINEAR AND LOGISTIC REGRESSION which in turn for much of the theory refers to Sample Size for Logistic Regression With Small Response Probability which bases its result on approximations to the Fisher information matrix, so this is really based on normal approximations. It is known that normal approximations can be bad for logistic regression, so the results from this formulas should probably be checked by simulation.

Code in R:

min_n <- function(alpha=0.05, power=0.8, beta_a, sigma_x, p, R2) {
    z1 <- qnorm(1-alpha/2); z2 <- qnorm(power)
    n <-  (z1 + z2)^2 / ( (beta_a*sigma_x)^2 * p*(1-p) * (1-R2) )
    ceiling(n)
}

A simple test:

min_n(beta_a=0.2, sigma_x=1, p=0.5, R2=0.5)
[1] 1570

Code for the plot:

ypos <- c(30, 100, 300, 1000, 3000, 10000, 30000, 100000)
plot(function(b) min_n(beta_a=b, sigma_x=1, p=0.5, R2=0.5), from=0.0, to=1.0, n=101, log="y", xlab="alternativ coefficient", ylab="n", yaxt="n", lwd=3)
axis(2, at=ypos, labels=sprintf("%i", ypos), las=1)
plot(function(b) min_n(beta_a=b, sigma_x=1, p=0.1, R2=0.5), from=0.0, to=1.0, n=101, log="y", add=TRUE, col="blue", lwd=3)
legend("topright", legend=c("p=0.5", "p=0.1"), col=c("black", "blue"), lwd=3)
title(expression(paste("Sample size for logistic regression ", "with ", sigma[x]==1, " and ", "R2=0.5")))

Answer 3

I dig a little and found that GPower does have a section on logistic regression in the supporting manual (GPower 3.1 Correlation & Regression, 2009)) Here are explanations for the parameters based on that manual. This will help to get the conceptual part but you may still struggle with specific numbers to enter.

Odds ratio & Pr(Y=1 | X=1)H0: These two are to calculate the effect size. "Given the conditional probability p1=p(Y=1|X=1) under H0, we may define the effect size either by specifying p2=(Y=1|X=1)under H1 or by specifying the odds ratio OR=[p2/(1-p2)]/[p1(1-p1)]."

R2 other X: "The sample size N is multiplied by (1-R2), where R2 is the squared multiple correlation coefficient when the predictor of interest is regressed on the other predictors," i.e., "The proportion of variance of Xj explained by additional predictors in the model (R2 other X)."

X distribution, X parm mu, & X parm sigma: These are about the predictor distribution. "The power of the test also depends on the distribution of the predictor X, the 'X distribution' and its parameters need to be specified. Users may choose between six predefined distributions or select a manual imput mode. Depending on this selection, additional parameters must be specified" (see the original manual for details).

(p.1157, Faul, Erdfelder, Buchner, & Lang, 2009)