1
$\begingroup$

I have read that there are multiple ways to look at residuals for logistic regression. I am not necessarily that interested in analyzing the residuals for a logit model, but my main question is rather:

Where are residuals in the specification of the model?

For linear models, it's common to write $$y=X\beta+\varepsilon \ \ \text{ or } \ \ y = \beta_0+\beta_1x_1 + \dots + \beta_kx_k+\varepsilon$$

However, for the logit model I keep finding $$l=\log\frac{p}{1-p}=\beta_0+\beta_1x_1 + \dots + \beta_kx_k \ \ \text{ or } \ \ p = \frac{1}{1+e^{-(\beta_0+\beta_1x_1 + \dots + \beta_kx_k)}}$$

How come the disturbances are not part of the model in this case?

Edit:

I have thought of something but am not sure if this is the complete reasoning behind it. In my understanding, a linear model assumes that data have been generated according to $y=X\beta$ with random disturbances, as nothing is perfectly linear.

And for a logit model this disturbance is not needed, as the data are assumed to be generated(?) according to $p$: at every point $x_i$ there is a certain probability $p$ that the outcome will be $1$, and the points $y=0,1$ are generated according to this, rather than a set linear relation with disturbances that make it random.

$\endgroup$
2
  • 1
    $\begingroup$ For logistic regression, the role of $R^2$ is taken by "Pseudo $R^2$", for which different definitions are in use. All have in common that they are not based on residuals but on ratios between likelihood values. If you search for "pseudo r squared" and "logistic regression", you should find some possible definitions. $\endgroup$
    – cdalitz
    Jun 27, 2021 at 19:43
  • 1
    $\begingroup$ You should distinguish error i.e. $\epsilon_i = Y_i-\mu_i(X_i, \beta)$ (where $\mu_i$ is a conditional mean that depends on $X_i$ and parameter $\beta$ and is unique), from residuals i.e. $Y_i-\mu_i(X_i, \hat{\beta})$ (with estimated $\beta$, which will be different each data set). Both always exist, but $var(\epsilon) = \sigma^2$ won't be parameterized when it's already determined by the mean e.g. in logistic regression's Bernoulli distrib. assumption. That linear regression requires Normality and therefore $\mu_i$ and $\sigma^2$ makes it necessary to write $\epsilon$ into the model. $\endgroup$ Jun 27, 2021 at 20:47

1 Answer 1

1
$\begingroup$

You can have residuals, or not, in either case. But to start with, you don't need residuals at all to specify the model: In either case, you can write $$Y|X=x \ \ \sim \ \ p(y|x),$$ where $p(y|x)$ is some conditional distribution. In the case of logistic regression, $p(y|x)$ is a Bernoulli distribution. In the case of classical regression, $p(y|x)$ is a normal distribution, but this can be relaxed to allow other forms, like Poisson, negative binomial, multinomial, non- or semi-parametric variants, etc.

If you want residuals, you can also have them, in either case: For an observation where $X=x$, the residual is $\epsilon = Y - E(Y|X=x)$. In the case of standard regression, under the usual linearity assumption, this gives you $$\epsilon = Y - (\beta_0 + \beta_1 x).$$ In the case of logistic regression, under the usual linearity (of logit) assumption, this gives you $$\epsilon = Y - \frac{\exp{(\beta_0 + \beta_1 x)}}{1+\exp{(\beta_0 + \beta_1 x)}},$$ where $Y$ is either 0 or 1.

It is just not common to use this form in the case of logistic regression because the salient feature of the model is the form of the conditional Bernoulli distribution of $Y|X=x$.

$\endgroup$
3
  • 1
    $\begingroup$ Although this answer is correct, it is formulated in such abstract terms that the original poster presumably won't understand it. I would suggest to add an explanation how $E(Y|X=x)$ is computed from the estimated coefficients $\beta_i$. $\endgroup$
    – cdalitz
    Jun 27, 2021 at 19:40
  • $\begingroup$ Thanks; edited. $\endgroup$ Jun 27, 2021 at 19:47
  • $\begingroup$ The last equation is used as the summand for calculating the Brier score, which measures the accuracy of probabilistic predictions. $\endgroup$ Jun 27, 2021 at 20:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.