Interpretation of coefficients in a poorly performing GLM

Question

Suppose that I have trained a logistic regression model on a certain dataset, and I wish to interpret the coefficients of this model.

Does it make any difference on the validity of the interpretation if the model is poor?

What I mean by this is that imagine that we measure the performance of the model by a ROC curve and we get a low value (but still better than 0.5) for the area under the curve. This model is not terribly accurate, but does the performance of it influence the interpretation of the coefficients?

Answer 1

The statistical interpretation of the coefficients doesn't depend on how the model was fit. I could make completely random guesses of the coefficients and they would have the same interpretation as they would had I estimated them with maximum likelihood. For two units identical on all measured variables except that they differed on $X_1$ by one unit, the difference in the log odds of success is $\beta_1$. That interpretation comes directly from simply writing down the regression equation and has nothing to do with the fitting process.

To interpret the coefficients as consistent estimates of some "true" association, or as total effects rather than direct effects, or as causal effects rather than mere conditional assocations, requires more assumptions, far more than whether the model fit well in your sample.

For example, let's say the true data-generating (i.e., structural causal) model was

$$P(Y=1|X_1,X_2) = expit(\gamma_0 + \gamma_1 X_1 + \gamma_2 X_2)$$

Let's say I'm considering the model

$$P(Y=1|X_1) = expit(\beta_0 + \beta_1 X_1)$$

which excludes $X_2$. $\beta_1$ doesn't have a causal interpretation, but it's the regression slope you would get if you were to fit that model to the population data (i.e., so there is no sampling error). The interpretation of $\beta_1$ in this model is: For two units that differed on $X_1$ by one unit, the difference in the log odds of success is $\beta_1$.

Let's say I collect a sample and then pull an estimate of $\beta_1$ out of a hat and call it $\hat \beta_1^{guess}$. Even though that value is completely unconnected to the sample, it still has the same interpretation as any other estimate of $\beta_1$, which is as an estimate of the difference in the log odds of success for two units that differed on $X_1$ by one unit. It's not a valid or consistent estimate, but it's an estimate of a quantity that has a clear interpretation. The quantity ($\beta_1$) does not have a causal interpretation, but it's still meaningfully interpretable as an associational quantity.

If I were to estimate $\beta_1$ with maximum likelihood, and call the estimate $\hat \beta_1^{MLE}$, it has the same interpretation as $\hat \beta_1^{guess}$, which is that it is an estimate of $\beta_1$, which, again, has a clear interpretation. $\hat \beta_1^{MLE}$ is a consistent estimate of $\beta_1$, so if I were to want to know what $\beta_1$ was I would be inclined to say it's closer to $\hat \beta_1^{MLE}$ than it is to $\hat \beta_1^{guess}$. $\hat \beta_1^{MLE}$ could result from a terribly fitting model, and that would say nothing of its interpretation. A terribly fitting model might result because we failed to include $X_2$ in it. That doesn't change how $\beta_1$, and thus how $\hat \beta_1^{MLE}$ and $\hat \beta_1^{guess}$, are interpreted.

If you wanted to interpret a regression coefficient as causal, then you want to estimate $\gamma_1$, not $\beta_1$. The interpretation of $\gamma_1$ is the change in the log odds of success caused by intervening on $X_1$ by one unit while holding $X_2$ constant. Any estimate of $\gamma_1$, regardless of how it came to be, could be interpreted as an estimate of the change in the log odds of success caused by intervening on $X_1$ by one unit while holding $X_2$ constant. You could even use $\hat \beta_1^{guess}$ as an estimate of $\gamma_1$ and it would still have this interpretation. It would likely be a bad estimate that you shouldn't trust, but that doesn't change its interpretation. Even if you estimated $\gamma_1$ using maximum likelihood estimation of a model that included both $X_1$ and $X_2$, its interpretation would be the same; it would likely just be a better estimate (but it doesn't mean it's a good estimate!).

All this is to say that the interpretation of coefficients comes from the model as it is written, not the way they are estimated or how well the estimated model fits. These may serve as indicators as to whether the estimated coefficients might be close to the population versions they are trying to approximate, but not how they should be interpreted. For example, a poorly fitting model resulting from regressing $Y$ on $X_1$ may indicate that $\hat \beta_1$ is a poor estimate of $\gamma_1$, but it may be a good estimate of $\beta_1$. The interpretations of $\beta_1$ and $\gamma_1$ are unrelated to how the estimates were generated, and the interpretation of the estimates is simply as estimates of those quantities.

Answer 2

We do something like this all the time when we do t-testing of means.

Remember that a t-test of means is a two-sample ANOVA, meaning that we do a regression like:

$$\hat{y}_i = \hat{\beta}_0 + \hat{\beta}_1x_i$$

where $x_i$ is a $0/1$ indicator variable for group membership.

When you do a t-test, you often leave lots of variance unexplained.

set.seed(2020)
N <- 250
x <- c(rep(0, N), rep(1, N))
y <- c(rnorm(N, 0, 1), rnorm(N, 0.5, 1))
tt <- t.test(y[x==0], y[x==1], var.equal=T)$p.value
L <- lm(y~x)
summary(L)
tt

The p-value is tiny, $8.48\times 10^{-5}$, and the correct value of $\beta_1=0.5$ is within the $95\%$ confidence interval, yet the $R^2 = 0.03057$.

So yes, it can be acceptable to do the same when you do a logistic regression instead of a linear regression. It might be a terrible idea, but poor fit alone is not a reason to keep from interpreting the coefficients. Consider the situation where the true conditional probabilities are all around $0.5$. You shouldn't be able to do much better than guessing.

Finally, be leery of using improper scoring rules like AUCROC. There are many posts on here about this topic, some of which are mine. This linked post has an excellent answer with some links. The "Frank Harrell" I mention says that ROCAUC can be used for diagnostics of a model on its own---does it perform well at all---but is not for model comparisons.

Answer 3

My advice on how to gain some guidance in a particular context of a poor regression model, is to proceed to construct a model where, if the correct model specification is provided, along with its random error structure, it actually performs well. The latter is determined based on parameter estimation routines as commonly employed over repeated simulation runs. This exercise also assists in interpreting the coefficients of a particular model when the model's underlying assumptions are theoretically accurate.

The next step requires specific knowledge of the context so as to introduce a reasonable occurring model misspecification error (by say lacking availability to a significant contributing variable, or having to employ a less than perfect correlated variable). Re-estimate and now compare observed coefficients over repeated trials to the actual known values for the correct theoretical model.

If the particular analysis you are employing is, say, highly sensitive to such misspecifications, you will be quantifiably educated and may wish to investigate other robust alternatives.

You may also find modeling approaches that a surprisingly robust.

Also, it may be the case, that the estimation routine itself is not particularly robust based on the particular parameter values, and not, per se, the model itself.