7
$\begingroup$

I am conducting a logistic regression I created the following test-data (the two predictors and the criterion are binary variables):

   UV1 UV2 AV
1    1   1  1
2    1   1  1
3    1   1  1
4    1   1  1
5    1   1  1
6    1   1  1
7    1   1  1
8    0   0  1
9    0   0  1
10   0   0  1
11   1   1  0
12   1   1  0
13   1   0  0
14   1   0  0
15   1   0  0
16   1   0  0
17   1   0  0
18   0   0  0
19   0   0  0
20   0   0  0

AV = $\frac{dependent variable}{criterion}$

$\frac{UV1}{UV2} = \frac{both independant variables}{predictors}$

For measuring the UVs effect on the AV a logistic regression is necessary, as the AV is a binary variable. Hence I used the following code

> lrmodel <- glm(AV ~ UV1 + UV2, data = lrdata, family = "binomial")

including "family = "binomial"". Is this correct?

Regarding my test-data, I was wondering about the whole model, especially the estimators and sigificance:

> summary(lrmodel)


Call:
glm(formula = AV ~ UV1 + UV2, family = "binomial", data = lrdata)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-1.7344  -0.2944   0.3544   0.7090   1.1774  

Coefficients:
              Estimate Std. Error z value Pr(>|z|)
(Intercept) -4.065e-15  8.165e-01   0.000    1.000
UV1         -1.857e+01  2.917e+03  -0.006    0.995
UV2          1.982e+01  2.917e+03   0.007    0.995

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 27.726  on 19  degrees of freedom
Residual deviance: 17.852  on 17  degrees of freedom
AIC: 23.852

Number of Fisher Scoring iterations: 17
  1. Why is UV2 not significant. See therefore that for group AV = 1 there are 7 cases with UV2 = 1, and for group AV = 0 there are only 3 cases with UV2 = 1. I was expecting that UV2 is a significant discriminator.

  2. Despite the not-significance of the UVs, the estimators are - in my opinion- very high (e.g. for UV2 = 1.982e+01). How is this possible?

  3. Why isn't the intercept 0,5?? We have 5 cases with AV = 1 and 5 cases with AV = 0.

Further: I created UV1 as a predictor I expected not to be significant: for group AV = 1 there are 5 cases withe UV1 = 1, and for group AV = 0 there are 5 cases withe UV1 = 1 as well.

The whole "picture" I gained from the logistic is confusing me...

What was consuming me more: When I run a "NOT-logistic" regression (by omitting "family = "binomial")

> lrmodel <- glm(AV ~ UV1 + UV2, data = lrdata,)

I get the expected results

Call:
glm(formula = AV ~ UV1 + UV2, data = lrdata)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-0.7778  -0.1250   0.1111   0.2222   0.5000  

Coefficients:
            Estimate Std. Error t value Pr(>|t|)   
(Intercept)   0.5000     0.1731   2.889  0.01020 * 
UV1          -0.5000     0.2567  -1.948  0.06816 . 
UV2           0.7778     0.2365   3.289  0.00433 **
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for gaussian family taken to be 0.1797386)

    Null deviance: 5.0000  on 19  degrees of freedom
Residual deviance: 3.0556  on 17  degrees of freedom
AIC: 27.182

Number of Fisher Scoring iterations: 2
  1. UV1 is not significant! :-)
  2. UV2 has an positive effect on AV = 1! :-)
  3. The intercept is 0.5! :-)

My overall question: Why isn't logistic regression (including "family = "binomial") producing results as expected, but a "NOT-logistic" regression (not including "family = "binomial") does?

Update: are the observations described above because of the correlation of UV1 and UV 2. Corr = 0.56 After manipulating the UV2's data

AV: 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0

UV1: 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0

UV2: 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0

(I changed the positions of the three 0s with the three 1s in UV2 to gain a correlation < 0.1 between UV1 and UV2) hence:

UV1 UV2 AV
1    1   0  1
2    1   0  1
3    1   0  1
4    1   1  1
5    1   1  1
6    1   1  1
7    1   1  1
8    0   1  1
9    0   1  1
10   0   1  1
11   1   1  0
12   1   1  0
13   1   0  0
14   1   0  0
15   1   0  0
16   1   0  0
17   1   0  0
18   0   0  0
19   0   0  0
20   0   0  0

to avoid correlation, my results come closer to my expectations:

Call:
glm(formula = AV ~ UV1 + UV2, family = "binomial", data = lrdata)

Deviance Residuals: 
     Min        1Q    Median        3Q       Max  
-1.76465  -0.81583  -0.03095   0.74994   1.58873  

Coefficients:
            Estimate Std. Error z value Pr(>|z|)  
(Intercept)  -1.1248     1.0862  -1.036   0.3004  
UV1           0.1955     1.1393   0.172   0.8637  
UV2           2.2495     1.0566   2.129   0.0333 *
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 27.726  on 19  degrees of freedom
Residual deviance: 22.396  on 17  degrees of freedom
AIC: 28.396

Number of Fisher Scoring iterations: 4

But why does the correlation influence the results of the logistic regression and not the results of the "not-logistic" regression?

$\endgroup$
  • 2
    $\begingroup$ When you don't know how the response was generated from the regressors, your data will be almost useless for testing. Why not generate AV according to a specified model based on UV1 and UV2 (with known coefficients)? Then you will have data worth looking at and learning from. $\endgroup$ – whuber Aug 26 '16 at 17:05
  • $\begingroup$ thx for this suggestion, but: I know how to generate the values for AV according to known coefficients for UV1 and UV2 for a linear/multiple regression. It would be easy to define, for example, y = 5+10*UV1+20*UV2 (for linear regression`s equation y = b0+b1*UV1+b2*UV2) and, based on this equation, to create a dataset for X and Y. But here is all about a logistic regression. $\endgroup$ – flobrr Aug 29 '16 at 9:19
  • $\begingroup$ How can i adapt this procedure to a logistic regression, especially considering a logistic regression ist calculating an exact y-value but an value which can be defined as the probability to be part of group AV = 1 (and not part of group AV = 0); and this value isnt excact 0 or 1 but is between 0 and 1 and you have to choice a threshold for your model, to assign a case to group AV = 1, when the threshold is exceeded. Hence, there is a more "abstract" value in logistic regression and not a clear y-value as it is in linear regression. How to handle with this? $\endgroup$ – flobrr Aug 29 '16 at 9:19
  • $\begingroup$ Logistic regression uses a specific mathematical model that describes exactly how to generate data. I posted a working R example in a solution at stats.stackexchange.com/a/40609/919: see the line of code below the "Conduct a simulation" comment. $\endgroup$ – whuber Aug 29 '16 at 13:13
1
$\begingroup$

My overall question: Why isn't logistic regression (including "family = "binomial") producing results as expected, but a "NOT-logistic" regression (not including "family = "binomial") does?

You get different results as the linear models minimizes

$$ \sum_{i = 1}^n (y_i - \eta_i)^2 $$ whereas the logistic regression minimizes:

$$ \sum_{i = 1}^n y_i \log(\frac{1}{1 + \exp(-\eta_i)}) + (1 - y_i) \log(1 - \frac{1}{1 + \exp(-\eta_i)}) $$ where $$ \eta_i = \beta_0 + \beta_1 UV_1 + \beta_2 UV_2 $$

There is no reason the results should be the same.

But why does the correlation influence the results of the logistic regression and not the results of the "not-logistic" regression?

It is going to affect both. I presume you also see lower Std. Errors in the linear model. This is an issue with multicollinearity though you might not call this multicollinearity when you only have two variables with correlation $.56$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.