5
$\begingroup$

Logistic regression essentially means taking the logit of your response proportions, p, and then doing standard regression. Consider the case where one p is 1.

logit(1) = log(1/(1-1)) = infinity. How can you do regression with infinity?

That is, if one of your observed proportions p is 1, then you are trying to find the line that minimises the sum of squared differences from a set of points that include infinity. How does this not drive the regression line to always have infinite slope?

$\endgroup$
  • 11
    $\begingroup$ You have mischaracterized logistic regression: one does not take the logits of the data but instead models the data as outcomes of random variables whose parameters are expressed in terms of logits. For more information see inter alia the Wikipedia article on generalized linear models. $\endgroup$ – whuber Sep 12 '13 at 5:44
  • $\begingroup$ OK, thanks! I misunderstood how the link function is used, and thus misunderstood glms. I'll be looking for a more basic tutorial than what I see in wikipedia. $\endgroup$ – Alex Holcombe Sep 12 '13 at 6:16
  • 1
    $\begingroup$ I have offered an answer, Alex, to help bridge that gap. $\endgroup$ – whuber Sep 12 '13 at 16:03
8
$\begingroup$

The question characterizes logistic regression as

$$\text{logit}(y) = \beta_0 + \beta_1 x + \varepsilon$$

and proposes to fit this model using least squares. It points out that because $y$ is a binary ($0$-$1$) variable, $\text{logit}(y)$ is undefined (or should be considered infinite), which is--to say the least--problematic!

The resolution of this conundrum is to avoid taking the logit of $y$ but instead apply its inverse, the logistic function

$$f(x) = \frac{1}{1 + \exp(-x)},$$

to the right hand side. Because $y$ on the left hand side still is a random variable with possible outcomes $0$ and $1$, it must be a Bernoulli variable: that is, what we need to know about it is the chance that $y=1$, written $\Pr(y=1).$ Therefore we make another attempt in the form

$$\Pr(y=1) = f(\beta_0 + \beta_1 x).$$

This is an example of a generalized linear model. Its parameters $\beta_0$ and $\beta_1$ are typically (but not necessarily) found using Maximum Likelihood.

To understand this better, many people find it instructive to create synthetic datasets according to this model (instead of analyzing actual data, where the true model is unknown). We will look at how that might be coded in R, which is well suited to expressing and simulating statistical models. First, though, let's inspect its results.

Figure

The data are shown as jittered points (they have been randomly shifted slightly in the horizontal direction to resolve overlaps). The true underlying probability function is plotted in solid red. The probability function fit using Maximum Likliehood is plotted in dashed gray.

You can see that where the red curve is high--which means the chance of $y=1$ is high--most of the data are $1$'s, whereas where the red curve drops to low levels, most of the data are $0$'s. The height of the curve stipulates the chance that the response will be a $1$. In logistic regression, the curve usually has the sigmoidal shape of the logistic function, while the data are always either at $y=1$ or $y=0$.

Reading over the code, which is written for expressive clarity, will help make these descriptions precise.

#
# Synthesize some data.
#
set.seed(17)                        # Allows results to be reproduced exactly
n <- 8                              # Number of distinct x values
k <- 4                              # Number of independent obs's for each x
x <- rep(1:n, 4)                    # Independent values
beta <- c(3, -1)                    # True parameters
logistic <- function(x) 1 / (1 + exp(-x))
probability <- function(x, b) logistic(b[1] + b[2]*x)
y <- rbinom(n*k, size=1, prob=probability(x, beta))   # Simulated data
#
# Fit the data using a logistic regression.
#
summary(fit <- glm(y ~ x, family=binomial(link="logit")))
#
# Plot the data, the true underlying probability function, and the fitted one.
#
jitter <- runif(n*k, -1/3, 1/3)     # Displaces points to resolve overlaps
plot(x+jitter, y, type="p", xlab="x", ylab="y", main="Data with true and fitted models")
curve(probability(x, beta), col="Red", lwd=2, add=TRUE)
curve(probability(x, coef(fit)), col="Gray", lwd=2, lty=2, add=TRUE)
$\endgroup$
-2
$\begingroup$

You got logistic regression kind of backwards (see whuber's comment on your question). True, the logit of 1 is infinity. But that's ok, because at no stage do you take the logit of the observed p's.

Consider standard regression, for simplicity using a single x variable:

y = b0 + b1*x

Logistic regression means that y is not interpreted as the response proportions (the p's), but rather as the logit of the response proportions. So let's say a p = 1. Because its logit is infinity, with finite b0 and b1, the logistic regression will never exactly predict it except asymptotically as x approaches infinity.

With the observed proportion of 1, the logistic regression for that x can however predict a very high probability (but again, never as high as 1), say .99. Then, the probability of observing a proportion of 1 for a limited number of trials can easily be large. So when logistic regression corresponds to maximizing the likelihood, the probability of that data point given the model can be high. So, a proportion of 1 need not be more influential (to "drive the regression to always have infinite slope") than any other proportion.

This is also true for other sigmoidal-link regression types such as probit.

$\endgroup$
  • 3
    $\begingroup$ @This seems unclear. Maximum-likelihood logistic regression certainly can give infinite odds ratio estimates & hence predicted probabilities of 1 or 0 - the phenomenon is called separation. $\endgroup$ – Scortchi - Reinstate Monica Sep 12 '13 at 8:42
  • $\begingroup$ It also seems that this answer misunderstands the question, which was about one response, not the whole set of responses. $\endgroup$ – Peter Flom - Reinstate Monica Sep 12 '13 at 10:54
  • $\begingroup$ This answer starts well but by the second paragraph it makes exactly the same mistake about logistic regression as the OP. $\endgroup$ – whuber Sep 12 '13 at 17:20
  • 4
    $\begingroup$ @whuber, I believe this is the OP. $\endgroup$ – gung - Reinstate Monica Sep 12 '13 at 19:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.