1
$\begingroup$

I'm currently running residual diagnostics for a logistic regression model, aiming to identify possible influential record could influence the parameters estimate.

I wonder about if it is possible to detect such record on the base of the diagonal hat matrix (Pregibon, 1981) and estimate beta parameters by deleting influential record in the hope to get better model performance in terms of accuracy (e.g. concordance statistic, AUROC).

By running a logistic regression model with SAS PROC LOGISTIC option INFLUENCE, I should get the diagonal value of the Hat Matrix (is it correct?), underlined in the image attached below:enter image description here

Now, according to Pregibon, the hat matrix diagonal should help to identify influential points; from the SAS PROC LOGISTIC documentation I cannot understand what I can do with such values.

Could you provide a brief description of what one can do with such values and what they are? Alternatively, could you suggest a rule of thumb to detect influential points given the table reported in the image?

Thanks in advance for your help!

$\endgroup$
1
$\begingroup$

In the context of standard linear (ridge) regression, the diagonal entries of the 'hat' matrix correspond to the (ridge) leverage scores. These can be interpreted as the influence that the corresponding input point has on the prediction at the training input locations.

Suppose we condition (train) on data $X\in\mathbb{R}^{n\times d}$, $y\in\mathbb{R}^n$ with a regulariser $\lambda\geq0$ (taking care of rank considerations with $\lambda = 0$) and obtain weights $\beta = (X^TX + \lambda I)^{-1}X^Ty$ such that $$ \hat{y} = X\beta = X(X^TX+\lambda I)^{-1} X^T y = Py.$$ The matrix $P$ is then the 'hat' matrix you refer to. Now to see how much influence a point $X_i$ has, with a corresponding output $y_i$, we could see how much the prediction at $X_i$ changes if we change the observation $y_i$. This is the derivative $d\hat{y}_i/d y_i$, and it is immediate from the above that it is equal to $[P]_{ii}$.

The above might need to be modified in the context of logistic regression, but the intuition should carry over. As to what you can do with such values, they're used throughout both theoretical and applied statistics for selecting 'informative subsets'. Their sum (the trace of $P$) is a quantity known as the 'effective dimension' of the regressor, and characterises the difficulty of the regression problem (roughly the number of near-orthogonal directions).

$\endgroup$
2
  • $\begingroup$ thanks for the answer! In your opinion, could I delete record by selecting them in the base of some rule in the hope to increase the model performance? if yes, could you suggest some rule, possibly supported by some reference? $\endgroup$
    – Quantopik
    Apr 29 at 13:01
  • 1
    $\begingroup$ Sorry for the late reply. Unfortunately I'm only familiar with the theoretical side of things, and not statistical practice. I don't however think you could 'detect outliers' using leverage scores only -- they are based on the input space locations, not on the output values! $\endgroup$
    – Oxonon
    May 12 at 23:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.