6
$\begingroup$

I understand the outlier impact for linear regression with squared loss. @gung had a beautiful answer in this post to explain the concept of leverage and residual.

My question is How does outlier impact logistic regression? Does the same concept apply? (that we want to have a closer look at high leverage/residual points?)

For example, R, plot(glm(am~wt,mtcars,family="binomial")) is telling me Toyota Corona has high leverage and residual, should I take a closer look?

enter image description here

I found this post that says logistic regression is robust to outliers but did not discuss leverage and residual. Is it correct?

enter image description here

Improve this question
$\endgroup$
1
  • 8
    $\begingroup$ The second illustration is extremely confusing--in some instructive ways. First, it does not exhibit any outlying responses. We might understand the rightmost point to be a (somewhat) high-leverage one, but that's all. Second, the fit is obviously wrong: this is a case of complete separation. As such (a) there is no unique fit and (b) no matter what solution is chosen (among the correct ones), the regressor value for that rightmost point has no influence on the solution whatsoever! $\endgroup$
    – whuber ♦
    May 16, 2017 at 13:43

2 Answers 2

Reset to default
7
$\begingroup$

Outliers may have the same essential impact on a logistic regression as they have in linear regression: The deletion-diagnostic model, fit by deleting the outlying observation, may have DF-betas greater than the full-model coefficient; this means that the sigmoid-slope of association may be of opposite direction. Separately, the inference may not agree in the two models, suggesting one commits a type II error, or the other commits a type I error.

This point underscores the problem of suggesting that, when outliers are encountered, they should summarily be deleted.

The implication for logistic regression data analysis is the same as well: if there is a single observation (or a small cluster of observations) which entirely drives the estimates and inference, they should be identified and discussed in the data analysis. DF-beta residual diagnostics is an effective numerical and graphical tool for either type of model which is easy to interpret by statisticians and non-statisticians alike.

There are some differences to discuss. In linear regression, it is very easy to visualize outliers using a scatter plot. The scaled vertical displacement from the line of best fit as well as the scaled horizontal distance from the centroid of predictor-scale X together determine the influence and leverage (outlier-ness) of an observation. For a logistic model, the mean-variance relationship means that the scaling factor for vertical displacement is a continuous function of the fitted sigmoid slope. Farther out in the tails, the mean is closer to either 0 or 1, leading to smaller variance so that seemingly small perturbations can have more substantial impacts on estimates and inference. However, whereas a Y value in linear regression may be arbitrarily large, the maximum fitted distance between a fitted and observed logistic value is bounded. Does that mean that a logistic regression is robust to outliers? Absolutely not.

Improve this answer
$\endgroup$
0
$\begingroup$

Logistic regression is robust to concordant outliers, (with an extreme X value but an outcome that accords with the X value), in a way that the linear probability model is not. See http://teaching.sociology.ul.ie:3838/logitinfl for a simple illustration.

It might be better to say that such observations are outliers for the linear probability model, but not for logistic.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.