9
$\begingroup$

Logistic regression essentially means taking the logit of your response proportions, p, and then doing standard regression. Consider the case where one p is 1.

logit(1) = log(1/(1-1)) = infinity. How can you do regression with infinity?

That is, if one of your observed proportions p is 1, then you are trying to find the line that minimises the sum of squared differences from a set of points that include infinity. How does this not drive the regression line to always have infinite slope?

$\endgroup$
3
  • 11
    $\begingroup$ You have mischaracterized logistic regression: one does not take the logits of the data but instead models the data as outcomes of random variables whose parameters are expressed in terms of logits. For more information see inter alia the Wikipedia article on generalized linear models. $\endgroup$
    – whuber
    Commented Sep 12, 2013 at 5:44
  • $\begingroup$ OK, thanks! I misunderstood how the link function is used, and thus misunderstood glms. I'll be looking for a more basic tutorial than what I see in wikipedia. $\endgroup$ Commented Sep 12, 2013 at 6:16
  • 1
    $\begingroup$ I have offered an answer, Alex, to help bridge that gap. $\endgroup$
    – whuber
    Commented Sep 12, 2013 at 16:03

3 Answers 3

12
$\begingroup$

The question characterizes logistic regression as

$$\text{logit}(y) = \beta_0 + \beta_1 x + \varepsilon$$

and proposes to fit this model using least squares. It points out that because $y$ is a binary ($0$-$1$) variable, $\text{logit}(y)$ is undefined (or should be considered infinite), which is--to say the least--problematic!

The resolution of this conundrum is to avoid taking the logit of $y$ but instead apply its inverse, the logistic function

$$f(x) = \frac{1}{1 + \exp(-x)},$$

to the right hand side. Because $y$ on the left hand side still is a random variable with possible outcomes $0$ and $1$, it must be a Bernoulli variable: that is, what we need to know about it is the chance that $y=1$, written $\Pr(y=1).$ Therefore we make another attempt in the form

$$\Pr(y=1) = f(\beta_0 + \beta_1 x).$$

This is an example of a generalized linear model. Its parameters $\beta_0$ and $\beta_1$ are typically (but not necessarily) found using Maximum Likelihood.

To understand this better, many people find it instructive to create synthetic datasets according to this model (instead of analyzing actual data, where the true model is unknown). We will look at how that might be coded in R, which is well suited to expressing and simulating statistical models. First, though, let's inspect its results.

Figure

The data are shown as jittered points (they have been randomly shifted slightly in the horizontal direction to resolve overlaps). The true underlying probability function is plotted in solid red. The probability function fit using Maximum Likliehood is plotted in dashed gray.

You can see that where the red curve is high--which means the chance of $y=1$ is high--most of the data are $1$'s, whereas where the red curve drops to low levels, most of the data are $0$'s. The height of the curve stipulates the chance that the response will be a $1$. In logistic regression, the curve usually has the sigmoidal shape of the logistic function, while the data are always either at $y=1$ or $y=0$.

Reading over the code, which is written for expressive clarity, will help make these descriptions precise.

#
# Synthesize some data.
#
set.seed(17)                        # Allows results to be reproduced exactly
n <- 8                              # Number of distinct x values
k <- 4                              # Number of independent obs's for each x
x <- rep(1:n, 4)                    # Independent values
beta <- c(3, -1)                    # True parameters
logistic <- function(x) 1 / (1 + exp(-x))
probability <- function(x, b) logistic(b[1] + b[2]*x)
y <- rbinom(n*k, size=1, prob=probability(x, beta))   # Simulated data
#
# Fit the data using a logistic regression.
#
summary(fit <- glm(y ~ x, family=binomial(link="logit")))
#
# Plot the data, the true underlying probability function, and the fitted one.
#
jitter <- runif(n*k, -1/3, 1/3)     # Displaces points to resolve overlaps
plot(x+jitter, y, type="p", xlab="x", ylab="y", main="Data with true and fitted models")
curve(probability(x, beta), col="Red", lwd=2, add=TRUE)
curve(probability(x, coef(fit)), col="Gray", lwd=2, lty=2, add=TRUE)
$\endgroup$
3
  • $\begingroup$ I do not understand "Because y on the left hand side", which left hand side? There is above only a logit(y) and logit(1) or logit(0) is infininity. However in the shown plot there is only a range from 0 to 1. Still confusing, how the logit is regressed. $\endgroup$ Commented Dec 21, 2020 at 18:30
  • $\begingroup$ @granular the only equation preceding that remark has a left side and a right side. $\endgroup$
    – whuber
    Commented Dec 21, 2020 at 18:46
  • $\begingroup$ Nothertheless, still do not understand the answer and a new posed question that is similar was deleted. And there is a bad equation style in commentaries. So we try to fulfill $1=\frac{1}{1+\text{exp}(-(\beta_0+\beta_1x))}$ if the dependent variable is 1 and $0=\frac{1}{1+\text{exp}(-(\beta_0+\beta_1x))}$ if the dependent variable is 0 for all corresponding independent variables? $\endgroup$ Commented Dec 21, 2020 at 19:02
0
$\begingroup$

The answer is that you do use the log odds ratio for the regression, but the equation for "p" instead which has values of 0 or 1 in the sample. The regression equation is estimated via iteratively reweighted least squares (IRLS).

The method can be found here Manually calculating logistic regression coefficient

$\endgroup$
-2
$\begingroup$

You got logistic regression kind of backwards (see whuber's comment on your question). True, the logit of 1 is infinity. But that's ok, because at no stage do you take the logit of the observed p's.

Consider standard regression, for simplicity using a single x variable:

y = b0 + b1*x

Logistic regression means that y is not interpreted as the response proportions (the p's), but rather as the logit of the response proportions. So let's say a p = 1. Because its logit is infinity, with finite b0 and b1, the logistic regression will never exactly predict it except asymptotically as x approaches infinity.

With the observed proportion of 1, the logistic regression for that x can however predict a very high probability (but again, never as high as 1), say .99. Then, the probability of observing a proportion of 1 for a limited number of trials can easily be large. So when logistic regression corresponds to maximizing the likelihood, the probability of that data point given the model can be high. So, a proportion of 1 need not be more influential (to "drive the regression to always have infinite slope") than any other proportion.

This is also true for other sigmoidal-link regression types such as probit.

$\endgroup$
4
  • 3
    $\begingroup$ @This seems unclear. Maximum-likelihood logistic regression certainly can give infinite odds ratio estimates & hence predicted probabilities of 1 or 0 - the phenomenon is called separation. $\endgroup$ Commented Sep 12, 2013 at 8:42
  • $\begingroup$ It also seems that this answer misunderstands the question, which was about one response, not the whole set of responses. $\endgroup$
    – Peter Flom
    Commented Sep 12, 2013 at 10:54
  • $\begingroup$ This answer starts well but by the second paragraph it makes exactly the same mistake about logistic regression as the OP. $\endgroup$
    – whuber
    Commented Sep 12, 2013 at 17:20
  • 5
    $\begingroup$ @whuber, I believe this is the OP. $\endgroup$ Commented Sep 12, 2013 at 19:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.