# Residuals for logistic regression and Cook's distance

1. Are there any particular assumptions regarding the errors for logistic regression such as the constant variance of the error terms and the normality of the residuals?

2. Also typically when you have points that have a Cook's distance larger than 4/n, do you remove them? If you do remove them, how can you tell if the model with the removed points is better?

I don't know if I can give you a complete answer, but I can give you some thoughts that may be helpful. First, all statistical models / tests have assumptions. However, logistic regression very much does not assume the residuals are normally distributed nor that the variance is constant. Rather, it is assumed that the data are distributed as a binomial, $\mathcal{B}(n_{x_i},p_{x_i})$, that is, with the number of Bernoulli trials equal to the number of observations at that exact set of covariate values and with the probability associated with that set of covariate values. Remember that the variance of a binomial is $np(1-p)$. Thus, if the $n$'s vary at different levels of the covariate, the variances will as well. Further, if any of the covariates are at all related to the response variable, then the probabilities will vary, and thus, so will the variances. These are important facts about logistic regression.

Second, model comparisons are usually performed between models with different specifications (for example, with different sets of covariates included), not over different subsets of the data. To be honest, I am not sure how that would properly be done. With a linear model, you could look at the 2 $R^2$s to see how much better the fit is with the aberrant data excluded, but this would only be descriptive, and you should know that $R^2$ would have to go up. With logistic regression, the standard $R^2$ cannot be used, however. There are various 'pseudo-$R^2$s' that have been developed to provide similar information, but they are often considered to be flawed and are not often used. For an overview of the different pseudo-$R^2$s that exist, see here. For some discussion, and criticism, of them, see here. Another possibility might be to jackknife the betas with and without the outliers included to see how excluding them contributes to stabilizing their sampling distributions. Once again, this would only be descriptive (i.e., it wouldn't constitute a test to tell you which model--er, subset of your data--to prefer) and the variance would have to go down. These things are true, for both pseudo-$R^2$s and the jackknifed distributions, because you selected those data to exclude based on the fact that they appear extreme.

1) Are there any particular assumptions regarding the errors for logistic regression such as the constant variance of the error terms and the normality of the residuals?

Logistic regression models don't have "errors" in the traditional sense. It is both counter-intuitive and methodologically inconsistent. The model outputs are fitted probabilities or risks whereas the observed outcomes are 0/1 event indicators. Methodologically, you would tend to under-emphasize domains of very high or very low fitted probabilities (contributing very small amounts to the residual distance) whereas the model fitting algorithm places considerably higher importance on such regions. Squared distance is generally a poor way of calibrating a logistic regression model.

An alternative goodness of fit test is the Hosmer-Lemeshow test in which the fitted values are used to create binned partitions based on deciles of fitted risk. You can read about this test in Alan Agresti's Categorical Data Analysis or the book Logistic Regression by Hosmer and Lemeshow. Another process is to use the Studentized Residuals where the mean variance relationship is used to reweight residuals by their fitted inverse variance. For logistic regression this is

$$r_{stud} = \frac{Y - \mu}{\sqrt{\mu(1-\mu)}}$$

2) Also typically when you have points that have a Cook's distance larger than 4/n, do you remove them? If you do remove them, how can you tell if the model with the removed points is better?

I never remove points based on sensitivity analyses. If I do a random sample of 100 people and their income and 1 person happens to be a billionaire, then my safest assumption is that that 1 billionaire represents 1/100th of the population.

• Why would you assume thet the 1 billionaire represents 1/100th of the population? You would probably be able to get an outside estimate of the proportion of billionaires in the population! – kjetil b halvorsen Mar 5 '17 at 16:19

I agree with AdamO's comment above in general—assuming that 1 billionaire represents 1/100th of the population is totally fine. However, if the presence of the 1 billionaire skews the data so much that the prediction for the other 99 people is affected, I would remove the 1 billionaire. I would rather be wrong with prediction of an outlier than everyone else.

Having said that, if you remove data points using Cook's D values (i.e., anything > 4/d.f.), then you could use area under ROC curves for both the models to check for improvement.

• (+1) Modelling the relation between the log-odds of response & income with a natural spline, perhaps transforming income beforehand, is another way to avoid the billionaire's affecting predictions for the others overly much. Removing him suggests you're happy not to make predictions for other billionaires (fair enough) rather than happy to make wrong predictions about them. – Scortchi - Reinstate Monica Jan 30 '15 at 11:43
• Ironically, when it comes to predicting binary events, it is true that excluding influential observations may lead to better calibration of risk predictions. However, excluding influential observations will reduce discrimination of risk predictions. The latter is arguably more important. When it comes to predicting the risk of a certain event (which is either 0 or 1, not continuously valued) the best kind of prediction will push cases' predictions closer to 1 and controls' predictions closer to 0. High influence points are often effective at doing this. – AdamO Jan 30 '15 at 19:16