1
$\begingroup$

I have data $y$ which is the rate of success in $n$ trials. I also have covariates $X$ that I want to regress against $y$ to understand the relationship between them. I tried 2 different approaches. The first is to take the log of $y$ and apply OLS regression. The second is to apply logistic regression to $y$ directly.

I noticed that one theoretically important coefficient, call it $x_0$, is highly significant in both models but in OLS it is estimated as a strong negative effect and in logistic regression it is a small but positive effect.

How is this possible? The variable is binary and I am comparing the standardized coefficients from both models. My assumption is that $x_0$ either increases the rate or not and it should be of the same sign in both models.

UPDATE

When my only regressor is $x_0$ this doesn't happen, it's only when I add the other covariates (of which there is about $200$). I have pasted the output of the full models below, excluding the estimates of the other covariates.

OLS

Call:
lm(formula = fx, data = fit_data, weights = weights)

Weighted Residuals:
         Min           1Q       Median           3Q          Max 
-5.044177447 -0.616615480  0.070124542  0.692928438  4.539252302 

Coefficients
                 Estimate      Std. Error   t value   Pr(>|t|)    
(Intercept)     -1.33734283e+04  4.75932218e+03  -2.80994 0.00495879 ** 
x0              -1.06636833e-01  2.63420263e-02  -4.04816 5.1773e-05 ***

---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 1.01868241 on 25500 degrees of freedom
Multiple R-squared:  0.59443022,    Adjusted R-squared:  0.591090233 
F-statistic: 177.973843 on 210 and 25500 DF,  p-value: < 2.220446e-16

GLM

Call:
glm(formula = fx, family = "binomial", data = fit_data, weights = weights)

Deviance Residuals: 
          Min             1Q         Median             3Q            Max  
-103.51742894    -3.06309844    -0.24575206     2.27364818   165.10350046  

Coefficients:

                 Estimate      Std. Error    z value   Pr(>|z|)    
(Intercept)      2.83201590e+03  1.31990079e+03    2.14563 0.03190268 *  
x0               1.77428670e-02  1.90814767e-03    9.29848 < 2.22e-16 ***

---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Dispersion parameter for binomial family taken to be 1)

Null deviance: 9542439.111  on 25710  degrees of freedom
Residual deviance: 2224838.019  on 25500  degrees of freedom
AIC: 2340850.46

Number of Fisher Scoring iterations: 8
```
$\endgroup$
8
  • $\begingroup$ Probably not this one, but I have to ask: you are not comparing unstandardized $\beta$ from OLS with odds ratio from logistic regression, right? $\endgroup$
    – T.E.G.
    Mar 1, 2021 at 18:48
  • 1
    $\begingroup$ Please include the output of both models. The logistic model may be reporting an odds ratio $\endgroup$ Mar 1, 2021 at 18:48
  • $\begingroup$ I am comparing the standardized $\beta$ coefficients from both models. My assumption is that $x_0$ either pushes $y$ up or down, so it should be of the same sign in both models. $\endgroup$
    – badmax
    Mar 1, 2021 at 18:49
  • $\begingroup$ @RobertLong the output is too long, the model contains several hundred variables. I am using R and I am looking at the summary of lm() for the OLS model and glm() for the logistic model. $\endgroup$
    – badmax
    Mar 1, 2021 at 18:51
  • 2
    $\begingroup$ The short answer is that these are very different models, especially when the numbers of trials vary and/or the values of $y$ approach $0$ and $1,$ so there's no a priori reason to expect every one of hundreds of variables to play similar roles. We of course would like to go further and diagnose your particular case, but for that we need specific information. You can start by looking at and reporting on much smaller models using subsets of your data and tiny numbers of variables (including $x_0$) to see whether you can reproduce this behavior. $\endgroup$
    – whuber
    Mar 1, 2021 at 18:54

2 Answers 2

6
$\begingroup$

I don't think an entire data presentation is needed to give some intuition behind this phenomenon. While we would expect that a logistic regression and OLS model will, on average, produce parameter estimates (slopes or log-odds ratios) that are similar sign, it's entirely possible they will disagree for a given dataset and analysis.

The most likely issue is the influence of observations in the tails. Supposing we have an X that follows a standard normal density, and a modest positive trend in the risk of outcome for greater values of X. The influence of a single observation at (X=5, Y=0) will be far, far greater in a logistic regression model than in a linear regression model. That is because the influence of such an observation in a logistic model can be arbitrarily high. If the fitted value in the logistic model is 0.9, then the Pearson residual will be $(0-0.9)^2/\left( 0.9*(1-0.9) \right)\approx 9$ compared to the linear model where the residual is $0.81$.

The other issue is overadjustment. Due to non-collapsibility of the odds ratio, if (as you say) "several hundreds" of variables are input to the linear model, and suppose several of those variables are unrelated to the outcome, the tendency will be for the OR of effect to attenuate with larger adjustment. Then it becomes likely that the OR will spontaneously flip effect due to random perturbations along the lines of what's pointed out above - again, in multivariate adjustment, it can be difficult to understand the contribution of a single variable, so using the DF-beta diagnostic is a prudent choice.

$\endgroup$
4
  • 2
    $\begingroup$ (+1) nice answer Adam :) it is also noteworthy, that, based on the output added to the OP, in both models, the estimate for x0 is around 4 orders of magnitude smaller than the intercept term. $\endgroup$ Mar 1, 2021 at 20:45
  • $\begingroup$ You got it! The most influential DF-beta points for $x_0$ were 8 times larger in the logistic model than in the linear model. Once I removed the outliers (0.5% of them) everything lined up as expected (and the estimate went up an order of magnitude in the logistic model). $\endgroup$
    – badmax
    Mar 1, 2021 at 22:40
  • $\begingroup$ Can you please clarify the part about the sum-of-squared error? Why can it be at most 1, when the target ($\log y$) is unbounded on the left? Why is the Pearson residual important for the logistic model rather than the same sum-of-squares? $\endgroup$
    – badmax
    Mar 1, 2021 at 22:43
  • 1
    $\begingroup$ @badmax well noted. The fitted values for the linear model do indeed exceed 0, 1. In the diagnostics I made for these kinds of models, I have typically truncated predictions at 0 or 1. The DF betas tend to reveal the truth about the real influence of outlying observations. A word of caution however: removing the outliers to make linear and logistic agree is a biased form of data cleaning because you are using statistical results to make decisions. $\endgroup$
    – AdamO
    Mar 1, 2021 at 23:36
0
$\begingroup$

This is an artifact of the model used for logistic regression, which is model for one of the two probabilities, which add to 1. If your two values are coded, eg, 0 and 1, then linear regression assigns a positive coefficient to any variable, x, that increases in value as y shifts from 0 to 1. So regression assumes that y increases from 0 to 1.

The logistic transformation can model either the probability that y = 1 or that y = 0, eg logit(P(Y=1)) = b0 + b1x or logit(P(Y=0)) = b0 + b1x.

These two models give opposite signs for b1, but they are the same in absolute value and they have the same estimated variance or standard error so they also have the same p-value. Confidence intervals also have their signs switched. The inferences you make about b1 are the same.

The choice of models, for P(y=1) or P(y=1), is arbitrary as far as the logistic method is concerned, when there are two categories.

The person who wrote the software probably wasn't concerned about whether users would want to compare linear regression with logistic regression, only with having a program that worked well.

I seem to recall that the model SAS originally programmed did just what you have noticed. For many years they refused to make any change. After a lot of complaints they included an option that permitted the user to switch the roles of the two categories that resulted in switching the signs of all the variables.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.