# Logistic Regression with R

I am conducting a logistic regression I created the following test-data (the two predictors and the criterion are binary variables):

   UV1 UV2 AV
1    1   1  1
2    1   1  1
3    1   1  1
4    1   1  1
5    1   1  1
6    1   1  1
7    1   1  1
8    0   0  1
9    0   0  1
10   0   0  1
11   1   1  0
12   1   1  0
13   1   0  0
14   1   0  0
15   1   0  0
16   1   0  0
17   1   0  0
18   0   0  0
19   0   0  0
20   0   0  0


AV = $\frac{dependent variable}{criterion}$

$\frac{UV1}{UV2} = \frac{both independant variables}{predictors}$

For measuring the UVs effect on the AV a logistic regression is necessary, as the AV is a binary variable. Hence I used the following code

> lrmodel <- glm(AV ~ UV1 + UV2, data = lrdata, family = "binomial")


including "family = "binomial"". Is this correct?

Regarding my test-data, I was wondering about the whole model, especially the estimators and sigificance:

> summary(lrmodel)

Call:
glm(formula = AV ~ UV1 + UV2, family = "binomial", data = lrdata)

Deviance Residuals:
Min       1Q   Median       3Q      Max
-1.7344  -0.2944   0.3544   0.7090   1.1774

Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -4.065e-15  8.165e-01   0.000    1.000
UV1         -1.857e+01  2.917e+03  -0.006    0.995
UV2          1.982e+01  2.917e+03   0.007    0.995

(Dispersion parameter for binomial family taken to be 1)

Null deviance: 27.726  on 19  degrees of freedom
Residual deviance: 17.852  on 17  degrees of freedom
AIC: 23.852

Number of Fisher Scoring iterations: 17

1. Why is UV2 not significant. See therefore that for group AV = 1 there are 7 cases with UV2 = 1, and for group AV = 0 there are only 3 cases with UV2 = 1. I was expecting that UV2 is a significant discriminator.

2. Despite the not-significance of the UVs, the estimators are - in my opinion- very high (e.g. for UV2 = 1.982e+01). How is this possible?

3. Why isn't the intercept 0,5?? We have 5 cases with AV = 1 and 5 cases with AV = 0.

Further: I created UV1 as a predictor I expected not to be significant: for group AV = 1 there are 5 cases withe UV1 = 1, and for group AV = 0 there are 5 cases withe UV1 = 1 as well.

The whole "picture" I gained from the logistic is confusing me...

What was consuming me more: When I run a "NOT-logistic" regression (by omitting "family = "binomial")

> lrmodel <- glm(AV ~ UV1 + UV2, data = lrdata,)


I get the expected results

Call:
glm(formula = AV ~ UV1 + UV2, data = lrdata)

Deviance Residuals:
Min       1Q   Median       3Q      Max
-0.7778  -0.1250   0.1111   0.2222   0.5000

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)   0.5000     0.1731   2.889  0.01020 *
UV1          -0.5000     0.2567  -1.948  0.06816 .
UV2           0.7778     0.2365   3.289  0.00433 **
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for gaussian family taken to be 0.1797386)

Null deviance: 5.0000  on 19  degrees of freedom
Residual deviance: 3.0556  on 17  degrees of freedom
AIC: 27.182

Number of Fisher Scoring iterations: 2

1. UV1 is not significant! :-)
2. UV2 has an positive effect on AV = 1! :-)
3. The intercept is 0.5! :-)

My overall question: Why isn't logistic regression (including "family = "binomial") producing results as expected, but a "NOT-logistic" regression (not including "family = "binomial") does?

Update: are the observations described above because of the correlation of UV1 and UV 2. Corr = 0.56 After manipulating the UV2's data

AV: 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0

UV1: 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0

UV2: 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0

(I changed the positions of the three 0s with the three 1s in UV2 to gain a correlation < 0.1 between UV1 and UV2) hence:

UV1 UV2 AV
1    1   0  1
2    1   0  1
3    1   0  1
4    1   1  1
5    1   1  1
6    1   1  1
7    1   1  1
8    0   1  1
9    0   1  1
10   0   1  1
11   1   1  0
12   1   1  0
13   1   0  0
14   1   0  0
15   1   0  0
16   1   0  0
17   1   0  0
18   0   0  0
19   0   0  0
20   0   0  0


to avoid correlation, my results come closer to my expectations:

Call:
glm(formula = AV ~ UV1 + UV2, family = "binomial", data = lrdata)

Deviance Residuals:
Min        1Q    Median        3Q       Max
-1.76465  -0.81583  -0.03095   0.74994   1.58873

Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept)  -1.1248     1.0862  -1.036   0.3004
UV1           0.1955     1.1393   0.172   0.8637
UV2           2.2495     1.0566   2.129   0.0333 *
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

Null deviance: 27.726  on 19  degrees of freedom
Residual deviance: 22.396  on 17  degrees of freedom
AIC: 28.396

Number of Fisher Scoring iterations: 4


But why does the correlation influence the results of the logistic regression and not the results of the "not-logistic" regression?

• When you don't know how the response was generated from the regressors, your data will be almost useless for testing. Why not generate AV according to a specified model based on UV1 and UV2 (with known coefficients)? Then you will have data worth looking at and learning from. – whuber Aug 26 '16 at 17:05
• thx for this suggestion, but: I know how to generate the values for AV according to known coefficients for UV1 and UV2 for a linear/multiple regression. It would be easy to define, for example, y = 5+10*UV1+20*UV2 (for linear regressions equation y = b0+b1*UV1+b2*UV2) and, based on this equation, to create a dataset for X and Y. But here is all about a logistic regression. – flobrr Aug 29 '16 at 9:19
• How can i adapt this procedure to a logistic regression, especially considering a logistic regression ist calculating an exact y-value but an value which can be defined as the probability to be part of group AV = 1 (and not part of group AV = 0); and this value isnt excact 0 or 1 but is between 0 and 1 and you have to choice a threshold for your model, to assign a case to group AV = 1, when the threshold is exceeded. Hence, there is a more "abstract" value in logistic regression and not a clear y-value as it is in linear regression. How to handle with this? – flobrr Aug 29 '16 at 9:19
• Logistic regression uses a specific mathematical model that describes exactly how to generate data. I posted a working R example in a solution at stats.stackexchange.com/a/40609/919: see the line of code below the "Conduct a simulation" comment. – whuber Aug 29 '16 at 13:13

My overall question: Why isn't logistic regression (including "family = "binomial") producing results as expected, but a "NOT-logistic" regression (not including "family = "binomial") does?

You get different results as the linear models minimizes

$$\sum_{i = 1}^n (y_i - \eta_i)^2$$ whereas the logistic regression minimizes:

$$\sum_{i = 1}^n y_i \log(\frac{1}{1 + \exp(-\eta_i)}) + (1 - y_i) \log(1 - \frac{1}{1 + \exp(-\eta_i)})$$ where $$\eta_i = \beta_0 + \beta_1 UV_1 + \beta_2 UV_2$$

There is no reason the results should be the same.

But why does the correlation influence the results of the logistic regression and not the results of the "not-logistic" regression?

It is going to affect both. I presume you also see lower Std. Error`s in the linear model. This is an issue with multicollinearity though you might not call this multicollinearity when you only have two variables with correlation $.56$.