9
$\begingroup$

I'm a little puzzled by a formula presented in Hastie's "Introduction to Statistical Learning". In Chapter 6, page 212 (sixth printing, available here), it is stated that:

$AIC = \frac{RSS}{n\hat\sigma^2} + \frac{2d}{n} $

For linear models with Gaussian noise, $d$ being the number of predictors and $\hat\sigma$ being the estimate of error variance. However,

$\hat\sigma^2 = \frac{RSS}{(n-2)}$

Which is stated in Chapter 3, page 66.

Which would imply:

$AIC = \frac{(n-2)}{n} + \frac{2d}{n} $

Which can't be right. Can someone point out what I'm doing incorrectly?

$\endgroup$
  • $\begingroup$ Unless I miss something, I don't think the book can be right. $\endgroup$ – Glen_b Nov 15 '15 at 3:43
3
$\begingroup$

I think that you are confusing the two residual sum of squares that you have. You have one RSS to estimate the $\hat{\sigma}^2$ in the formula, this RSS is in some sense independent of the number of parameters, $p$. This $\hat{\sigma}^2$ should be estimated using all your covariates, giving you a baseline unit of error. You should call the RSS in the formula for AIC: $\text{RSS}_{p_i}$, meaning that it corresponds to model $i$ with $p$ parameters, (There may be many models with $p$ parameters). So the RSS in the formula is calculated for a specific model, while the RSS for $\hat{\sigma}^2$ is for the full model.

This is also noted in the page before, where $\hat{\sigma}^2$ is introduced for $C_p$.

So the RSS for the formula in AIC is not indepednent of $p$, it is calculated for a given model. Introducing $\hat{\sigma}^2$ to all of this is just to have a baseline unit for the error, such that there is a "fair" comparison between the number of parameters and the reduction in error. You need to compare the number of parameters to something that is scaled w.r.t. the magnitude of the error.

If you would not scale the RSS by the baseline error, it might be that the RSS is dropping much more than the number of variables introduced and thus you become more greedy in adding in more variables. If you scale it to some unit, the comparison to the number of parameters is independent of the magnitude of the baseline error.

This is not the general way to calculate AIC, but it essentially boils down to something similar to this in cases where it is possible to derive simpler versions of the formula.

$\endgroup$
  • $\begingroup$ Would you be able to provide some reference where I can read more about the reasoning behind estimating error variance in these models with a total set of available predictors as opposed to the RSS of some subset? I see how your response answers this question but I'm not sure why it is legitimate to do so in the first place. $\endgroup$ – Sue Doh Nimh Nov 13 '15 at 21:44
  • $\begingroup$ @SueDohNimh These slides provide a good start. Note that the best estimate for $\sigma^2$ is using the full model, introduced for $C_p$. The AIC you have, is the one where $\sigma^2$ is known, but you just use the best estimate that you can get. Estimating $\sigma^2$ can be very difficult. This discussion is also relevant. This is also relevant. $\endgroup$ – Gumeo Nov 14 '15 at 15:02
  • 2
    $\begingroup$ You should also read Akaike's original paper, I think that is the best source, it has over 15K citations as of now. Here it is, you should be able to find it somewhere online or access it from a university. $\endgroup$ – Gumeo Nov 14 '15 at 15:03
5
$\begingroup$

Unfortunately this will be a rather unsatisfying answer...

First of all usually for the AIC calculation you will use the Maximum Likelihood estimate of $\sigma^2$ which would be biased. So that would reduce to $\sigma^2 = \frac{RSS}{n}$ and ultimately the calculation you do would reduce to $1+2\frac{d}{n}$. Second I would refer you to the Wikipedia article on AIC in particular in the equivariance cases section. As you see there it is clear that most derivations omit a constant $C$. This constant is irrelevant for model comparison purposes so it is omitted. It is somewhat common to see contradictory derivations of AIC because exactly of that issue. For example Johnson & Wichern's Applied Multivariate Statistical Analysis, 6th edition give AIC as: $n \log(\frac{RSS}{N}) + 2d$ (Chapt. 7.6), which clearly does not equate the definition of James et al. you are using. Neither book is wrong per se. Just people using different constants. In the case of the James et al. book it seems they do not allude this point. In other books eg. Ravishanker and Dey's A First Course in Linear Model Theory this is even more profound as the authors write:

\begin{align} AIC(p) &= -2l(y; X, \hat{\beta}_{ML}, \hat{\sigma}_{ML}^2) + 2p \\ &= -N \log(\hat{\sigma}_{ML}^2)/2 - N/2 + 2p \qquad (7.5.10) \end{align}

which interestingly it cannot be concurrently true either. As Burnham & Anderson (1998) Chapt 2.2 write: "In the special case of least squares (LS) estimation with normally distributed errors, and apart from an arbitrary additive constant, AIC can be expressed as a simple function of the residual sum of squares."; B&A suggest the same AIC variant that J&W use. What messes you up is that particular constant (and the fact you were not using the ML estimate for the residuals.) Looking at M. Bishop's Pattern Recognition and Machine Learning (2006) I find an even more contradictory definition as:

\begin{align} AIC &= l(D|w_{ML}) - M \qquad (1.73) \end{align}

which is funny because it not only omits the multiplier from the original paper but also goes ahead to tumble the signs so it can use AIC based selection as a maximization problem...

I would recommend sticking with the old fashioned definition $−2\log(L)+2p$ if you want to do theoretical derivations. This is the one Akaike states in his original paper. All the other intermediate formulas tend to be messy and/or make some implicit assumptions. If it is any consolation, you "did nothing wrong".

$\endgroup$
  • $\begingroup$ Ah! Well that indeed is a little anticlimactic but thank you. However by implication Hastie's AIC is both linearly increasing in d and not a function of the sum of squared residuals at all! The other definitions you provided at the very least vary with training set errors, whereas the Hastie's AIC would imply that the optimal model would just be one with 0 predictors. Is there any way to marry that up? $\endgroup$ – Sue Doh Nimh Nov 13 '15 at 9:16
  • 1
    $\begingroup$ Sorry I do not know why they give that formula. Usually there is a $\log$ involved somewhere. In the text they do not labour around AIC a lot and they focus on Mallow's $C_p$ so I won't be surprised if they made some simplifying assumptions. In general that chapter appears to use a bit odd conventions. Marrying it up seems to equate to dropping the logarithms more or less. They also seem to favour an $\frac{1}{N}$multiplier; I suspect this is done to make thing more similar to $C_p$. BTW, it is James's book. Hastie is an amazing academic but he is the 3rd author. $\endgroup$ – usεr11852 Nov 13 '15 at 18:34
  • $\begingroup$ I came to the same conclusions as you, reading the famous Hastie/Tibshirani book "The elements of statistical learning" (p.230-233), where the definitions of AIC/BIC are very similar to their definitions given in "Introduction to statistical learning in R". So, Hastie is amazing academic, but he is not so good at defining AIC/BIC =). $\endgroup$ – Rodvi Dec 24 '19 at 10:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.