Using the Hat Matrix to detect influential observations in logistic regression I'm currently running residual diagnostics for a logistic regression model, aiming to identify possible influential record could influence the parameters estimate.
I wonder about if it is possible to detect such record on the base of the diagonal hat matrix (Pregibon, 1981) and estimate beta parameters by deleting influential record in the hope to get better model performance in terms of accuracy (e.g. concordance statistic, AUROC).
By running a logistic regression model with SAS PROC LOGISTIC option INFLUENCE, I should get the diagonal value of the Hat Matrix (is it correct?), underlined in the image attached below:
Now, according to Pregibon, the hat matrix diagonal should help to identify influential points; from the SAS PROC LOGISTIC documentation I cannot understand what I can do with such values.
Could you provide a brief description of what one can do with such values and what they are? Alternatively, could you suggest a rule of thumb to detect influential points given the table reported in the image?
Thanks in advance for your help!
 A: In the context of standard linear (ridge) regression, the diagonal entries of the 'hat' matrix correspond to the (ridge) leverage scores. These can be interpreted as the influence that the corresponding input point has on the prediction at the training input locations.
Suppose we condition (train) on data $X\in\mathbb{R}^{n\times d}$, $y\in\mathbb{R}^n$ with a regulariser $\lambda\geq0$ (taking care of rank considerations with $\lambda = 0$) and obtain weights $\beta = (X^TX + \lambda I)^{-1}X^Ty$ such that
$$ \hat{y} = X\beta = X(X^TX+\lambda I)^{-1} X^T y = Py.$$
The matrix $P$ is then the 'hat' matrix you refer to. Now to see how much influence a point $X_i$ has, with a corresponding output $y_i$, we could see how much the prediction at $X_i$ changes if we change the observation $y_i$. This is the derivative $d\hat{y}_i/d y_i$, and it is immediate from the above that it is equal to $[P]_{ii}$.
The above might need to be modified in the context of logistic regression, but the intuition should carry over. As to what you can do with such values, they're used throughout both theoretical and applied statistics for selecting 'informative subsets'. Their sum (the trace of $P$) is a quantity known as the 'effective dimension' of the regressor, and characterises the difficulty of the regression problem (roughly the number of near-orthogonal directions).
