1
$\begingroup$

I have a model comparison I'm doing using AIC, only the residuals of my models are not normally distributed. I know that in this case, the simple formula for AIC does not work.

I'm thinking I may need to use bootstrapping to calculate the likelihood of various models. However, I see several pitfalls in this approach, the biggest being that the bootstrapped likelihood will, in essence, be coming up with a different distribution of residuals for each model fitted, totally invalidating the comparison. How do I do this bootstrapping properly?

$\endgroup$
3
  • $\begingroup$ What simple formula? The definition of AIC still works, for example. $\endgroup$ Nov 16, 2017 at 13:35
  • $\begingroup$ $\textrm{AIC} = 2k- 2\hat{L}$, where $\hat{L}$ is the maximum value of of the log-likelihood of the model. In least squares with gaussian IID residuals, we have $\hat{L} = \frac{n}{2} \ln(\frac{\textrm{RSS}}{n}) + C$. Substitution leads directly to $\textrm{AIC} = 2k-n\cdot \ln(\frac{\textrm{RSS}}{n}) + C$. This is the simple formula I was talking about. $\endgroup$
    – Lewis Hein
    Nov 16, 2017 at 16:59
  • $\begingroup$ The first formula works all the time, the last one is indeed only suitable for the Gaussian case (or maybe also some related cases, I am not sure). $\endgroup$ Nov 16, 2017 at 18:04

1 Answer 1

1
$\begingroup$

the bootstrapped likelihood will, in essence, be coming up with a different distribution of residuals for each model fitted, totally invalidating the comparison.

This will not on its own invalidate the comparison of AICs. Consider this, you can compare AICs of normal and Student-t residual models.

$\endgroup$
2
  • $\begingroup$ I believe this will cause problems in my case. Suppose that my data is generated by a process $y(x) = a\cdot x+b+e(x)$ where $e(x)$ is IID and sampled from a symmetrical distribution, but not any familiar distribution, even in approximation, and certainly not a normal distribution. Suppose further that I fit two models to my data, $y(x) = a\cdot x + b+e(x)$ and $y(x) = b+e(x)$, and then try to compare them with AIC. $e(x)$ is unknown so I must bootstrap the model likelihood, but the two models will generate different bootstrap distributions for $e(x)$ and non-comparable model likelihoods $\endgroup$
    – Lewis Hein
    Nov 16, 2017 at 17:20
  • $\begingroup$ I still don't see how is this a problem, and don't know what is "non-comparable model likelihoods." The fact that the error distributions are different is not an issue. $\endgroup$
    – Aksakal
    Nov 16, 2017 at 19:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.