# Why Logistic and Linear Regressions P-value results are so different?

I have a data:

A    B    C    D    Successes    Trials    Rate
0    0    0    0    19    19000    0,100000
0    0    1    1    21    19000    0,110526
0    1    0    1    17    19000    0,089474
0    1    1    0    21    19000    0,110526
1    0    0    1    15    19000    0,078947
1    0    1    0    22    19000    0,115789
1    1    0    0    17    19000    0,089474
1    1    1    1    21    19000    0,110526


So if I run Linear Regression where outputs are Number of Successes or Success Rate, I get results with C variable as significant. However, if I run Logistic Regression the variable C is not significant.

**Regression Analysis: Successes versus C**

Method

Categorical predictor coding  (1; 0)

Analysis of Variance

Regression   1  36,125  36,125    24,77    0,003
C          1  36,125  36,125    24,77    0,003
Error        6   8,750   1,458
Total        7  44,875

Model Summary

1,20761  80,50%     77,25%      65,34%

Coefficients

Term        Coef  SE Coef  T-Value  P-Value   VIF
Constant  17,000    0,604    28,15    0,000
C
1        4,250    0,854     4,98    0,003  1,00

Regression Equation

Successes = 17,000 + 0,0 C_0 + 4,250 C_1

**Regression Analysis: Rate versus C**

Method

Categorical predictor coding  (1; 0)

Analysis of Variance

Regression   1  0,001001  0,001001    24,77    0,003
C          1  0,001001  0,001001    24,77    0,003
Error        6  0,000242  0,000040
Total        7  0,001243

Model Summary

0,0063560  80,50%     77,25%      65,33%

Coefficients

Term         Coef  SE Coef  T-Value  P-Value   VIF
Constant  0,08947  0,00318    28,15    0,000
C
1       0,02237  0,00449     4,98    0,003  1,00

Regression Equation

Rate = 0,08947 + 0,0 C_0 + 0,02237 C_1

**Binary Logistic Regression: Successes versus C**

Method

Categorical predictor coding  (1; 0)
Rows used                     8

Response Information

Event
Variable   Value       Count  Name
Successes  Event         153  Event
Non-event  151847
Trials     Total      152000

Deviance Table

Regression   1   1,8947   1,89470        1,89    0,169
C          1   1,8947   1,89470        1,89    0,169
Error        6   0,5072   0,08453
Total        7   2,4019

Model Summary

Deviance   Deviance
78,88%     37,25%  2419,72

Coefficients

Term        Coef  SE Coef   VIF
Constant  -7,018    0,121
C
1        0,223    0,163  1,00

Odds Ratios for Categorical Predictors

Level A  Level B  Odds Ratio       95% CI
C
1      0            1,2503  (0,9088; 1,7201)

Odds ratio for level A relative to level B

Regression Equation

P(Event)  =  exp(Y')/(1 + exp(Y'))

Y' = -7,018 + 0,0 C_0 + 0,223 C_1

Goodness-of-Fit Tests

Test             DF  Chi-Square  P-Value
Deviance          6        0,51    0,998
Pearson           6        0,51    0,998
Hosmer-Lemeshow   0        0,00        *


As I understood one of the reason that Linear Regression and ANOVA could be used for binomial output is described in the chapter "DOE with Categorical Inputs and Outputs" (p. 399 Introduction to Engineering Statistics and Lean Sigma by Theodore Allen):

" In general, none of the design of experiments and regression methods in this and previous chapters are appropriate if the response is categorical, e.g., conforming or non-conforming to specifications. Logistic regression and neural nets described in the next chapter are relevant when outputs are categorical.

However, if each experimental run is effectively a batch of “b” successes or failures, then the fraction non-conforming can be treated as a continous response.

Moreover, if the batch size and true fraction non-conforming satisfies the following, then it is reasonable to expect that the residuals in regression will be normally distributed:

b × p0 > 5 and b × (1 – p0) > 5. (15.16)

This is the condition such that binomial distributed random probabilities can be approximated using the “normal approximation” or normal probability distribution functions."

• Why should they be similar? Commented Oct 27, 2017 at 14:44
• You're estimating different models. The real question is, "Why would you expect them to be the same?"
– Sycorax
Commented Oct 27, 2017 at 14:44

The linear regression of success rate against categorical variable C includes no information about the number of successes or number of trials. In your particular set of 4 runs, it happened that all 4 of the runs with C=1 had a higher success rate than the 4 runs with C=0. As the numbers of successes were fairly similar within each group of 4 runs, you found a "significant" p-value in regression.

Now step back and take into account the actual numbers of successes: 85 among the 4 runs with C=1, and 68 among the 4 runs with C=0. Poisson-distributed counts (as effectively assumed here in the logistic regression with such a low probability of success) have standard deviations equal to their means, so the value of 85 is $\pm$ 9.2 while the value of 68 is $\pm$ 8.2; there's a lot of overlap between the 2 distributions. So the p-value doesn't meet the threshold of "significance."

So is this "significant" or not? You have to use your knowledge of the subject matter, not just some quote from a textbook, to figure out which test is most appropriate for your application. Are the assumptions of the logistic regression or of the linear regression most appropriate? If these were all runs of 19000 independent trials then the logistic regression would be more appropriate. At first glance the numbers of successes seem more tightly distributed than you might expect from Poisson statistics, but it's easy to get misled by small numbers of cases. You could get tightly-distributed numbers of successes if there were some intra-run correlations: for example, if there were consistently runs of about 1000 failures followed by a single success. In that case, however, you probably should be looking at some time-series analysis within the sets of 19000 trials instead of a simple linear regression of overall success rates.

One quick comment, using a simple linear regression to do number of successes is inappropriate. What you want to do is use a general linear regression fit onto a poisson distribution.

• I've added clarification in the question. Commented Oct 27, 2017 at 14:38

There could be several reasons:

1. You may have a coding error; next time please include both input and output. (I would have mentioned this in a comment, but I don't have the repuation.)
2. Even though a logistic model is typically better than a linear model for binary response data, you can still have a data generating process such that a linear fit is a better fit than a logistic fit. Edit: In general, one should expect smaller standard errors with a smaller model fit. Of course, the parameters in different models have different interpretations and therefore there is no reason to expect them to be the same and therefore no reason to expect them to have the same p-value, even if the standard errors are the same.
3. Logistic regression is subject to rare events bias. See, for example, King and Zeng 2001: https://gking.harvard.edu/files/0s.pdf This may also bias coefficient estimates and cause standard error problems.
• None of the points seem to be related to the question. As noted by others in the comments, there is no reason to assume that the models will be similar at the first place. Moreover, the question is not about model fit, but p-values for parameters. Linear model for such data is wrong by definition (wrong output format), so I can't see what "data generating process" you have in mind -- yet, I agree that in some cases linear regression may be a better tool for making predictions for many different reasons.
– Tim
Commented Nov 3, 2017 at 9:10
• "As noted by others in the comments, there is no reason to assume that the models will be similar at the first place." Of course not, but in general a better fit will lead to smaller standard errors and therefore smaller p-values. Perhaps I should state this explicitly in my answer. "Linear model for such data is wrong by definition (wrong output format), so I can't see what "data generating process" you have in mind" The linear model being wrong does not imply the logistic model is better. Moreover, it's possible for the linear model to be correct over a restricted domain. Commented Nov 4, 2017 at 9:44