3
$\begingroup$

In order to visually compare two models (logistic regression, in case that it matters) I thought of plotting the contribution of the individual observations to the AIC of the respective model. The plot looks like this:

AIC per observation plot

One can see that, although for some observations the full model makes larger errors than the simple one (i.e. a larger contribution to the AIC), for the majority of the observations the per-observation AIC is lower for the full model. The linear regression (the orange line) reflects that by being below the diagonal dotted line. This, I hope, gives some visual support for using the full model---assuming that the model selection is based on the AIC in the first place.

Are plots like these actually used in practice and do they have an established name?

If not, why? Is this plot in any way misleading or unclear?

Improve this question
$\endgroup$
7
  • $\begingroup$ Linear regression is not symmetrical. If you regressed X on Y, would the orange line be above the diagonal? Moreover, what is the justification for treating X as fixed & Y as (partly) random? $\endgroup$
    – gung - Reinstate Monica
    Mar 20 at 16:27
  • $\begingroup$ If the idea is to argue for the full model, what is the advantage of this method over just reporting that the full model's AIC is lower? Is someone arguing that the lower AIC is due to just some of the data or something? $\endgroup$
    – gung - Reinstate Monica
    Mar 20 at 16:29
  • $\begingroup$ @gung-ReinstateMonica (1): Yes, since the regression line must pass through the $(\bar{x}, \bar{y})$. Regarding randomness, if it were for predictive modelling, I'd worry more about the non-normality and heteroskedacticity of the errors. But I'm just interested in an intuitive visualisation. $\endgroup$
    – Igor F.
    Mar 21 at 7:55
  • $\begingroup$ @gung-ReinstateMonica (2): This is just a general example. In my case at hand I have the opposite situation: The full model has a slightly lower AIC than a simpler one (467 vs. 469, less than 0.5% difference). I want to argue that the simpler model is sufficient; that the difference in AIC is likely random. The graph shows the observations neatly aligned along the diagonal. $\endgroup$
    – Igor F.
    Mar 21 at 7:57
  • $\begingroup$ Good point regarding $(\bar x, \bar y)$, but I bet it would cross the diagonal. Is the simpler model nested within the full model (e.g., the full model has X1-5, & the simpler model only has X1)? $\endgroup$
    – gung - Reinstate Monica
    Mar 21 at 11:33

1 Answer 1

Reset to default
1
$\begingroup$

I've never heard of that plot. It may already exist and have a name, however, I suspect it would not be persuasive to a reasonable skeptic, if it's sufficiently esoteric. Simply based on human psychology, people will find things they're familiar with persuasive and be skeptical of someone trying to persuade them of something based on something they've never heard of. For what it's worth, I do think it's interesting—I like plots and thoroughly exploring data to ensure I understand them fully.

In your actual case, I gather you want to argue for the simpler model even though the AIC for the full model is ever so slightly lower. I have heard various times that an AIC that isn't more than 1% lower shouldn't be trusted, but I certainly don't have a citation for that.

Instead, I think there may be a simpler solution. You state that the models are nested. You can perform a nested model test / simultaneous test of all added variables in the full model. For logistic regression, that would be a likelihood ratio test. In R, the code would be something like:

anova(simple.model, full.model, test="LRT")

@Glen_b has shown that a lower AIC corresponds to a p-value of approximately .16, so such a test is unlikely to be significant by conventional standards.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Different tools are meant for different goals: AIC versus Likelihood Ratio Test. I do not find it quite reasonable to contemplate the choice between a LR test and AIC once the goal of the analysis has been determined. $\endgroup$
    – Richard Hardy
    Mar 21 at 15:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.