0
$\begingroup$

I am trying to use logistic regression to make predictions in R. I am confused as to why a model is predicting TRUE for 90% of predictions, when the training data had only a 50% probability of being TRUE.

Here is some test data:

set.seed(10)

x1 <- rnorm(10000, 0, 15)
x2 <- rnorm(10000, 10, 4)
x3 <- rnorm(10000, 3, 10)
x4 <- rnorm(10000, -8, 3)

b0 <- -4
b1 <- 2
b2 <- 15
b3 <- 90
b4 <- -23

e <- rnorm(10000, mean = 0, sd=25)

y <- b0 + x1*b1 + x2*b2 + x3*b3 + x4*b4 + e

If I model y as continuous, the mean prediction is identical to the mean value of y. For example:

cont_mod <- lm(y ~ x1 + x2 + x3 + x4)
cont_pred <- predict(cont_mod , data.frame(x1, x2, x3, x4))

> mean(y)
[1] 602.8305
> mean(cont_pred)
[1] 602.8305

Thus, my intuition is that in a logistic regression the probability of the model predicting a TRUE value should be equal to the probability of a TRUE value in the training dataset. However, when I use logistic regression to model y as a binary outcome y_bin, the model predicts TRUE values at a very high rate.

y_bin <- y < median(y)
bin_mod <- glm(y_bin ~ x1 + x2 + x3 + x4)

logodds <- predict(bin_mod, data.frame(x1, x2, x3, x4))
odds <- exp(logodds)
prob <- odds / (1 + odds)

> mean(y_bin)
[1] 0.5
> mean(prob > 0.5)
[1] 0.896

After doing some searching here and on the internet, I found some people suggesting that a logistic regression model will make such predictions when it is fit/trained on unbalanced data. But it is making these predictions even when I ensure that 50% of the values of y_bin are true!

So, why does logistic regression predict 90% of outcomes as TRUE, when only 50% of the training data had a TRUE outcome? Additionally, if I should use a different value besides 0.5 as the cutoff between a TRUE and a FALSE prediction, how do I go about selecting the cutoff value?

$\endgroup$
1
  • 4
    $\begingroup$ glm does not fit a logistic regression unless you specify family=binomial. Your glm call is generating the same results as your lm call, due to the defaults for glm. Furthermore, your call won't work as written, since yb isn't defined. $\endgroup$
    – jbowman
    Feb 24, 2020 at 22:50

1 Answer 1

4
$\begingroup$

First, you would need to change your code to something like this:

set.seed(10)

x1 <- rnorm(10000, 0, 15)
x2 <- rnorm(10000, 10, 4)
x3 <- rnorm(10000, 3, 10)
x4 <- rnorm(10000, -8, 3)

b0 <- -4
b1 <- 2
b2 <- 15
b3 <- 90
b4 <- -23

e <- rnorm(10000, mean = 0, sd=25)

y <- b0 + x1*b1 + x2*b2 + x3*b3 + x4*b4 + e

y_bin <- y < median(y)
bin_mod <- glm(y_bin ~ x1 + x2 + x3 + x4, family = "binomial")

But this gives an error because numerical 0 and 1 occur. Your coefficients are very high which leads to perfect separation.

Change it to

b0 <- -4
b1 <- 2
b2 <- 3
b3 <- 4
b4 <- -2

and that error disappears.

and

head(bin_mod$fitted.values)

yields a wide range.

and

table(bin_mod$fitted.values > 0.5)

shows very close to a 50-50 split.

$\endgroup$
1
  • $\begingroup$ Getting fitted probabilities equal to 0 or 1 is not actually an "error". R simply issues a Warning message. $\endgroup$ Mar 13, 2020 at 7:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.