9
$\begingroup$

In time series models, like ARMA-GARCH, to select appropriate lag or order of the model different information criterion, like AIC, BIC, SIC etc, are used.

My question is very simple, why donot we use adjusted $R^2$ to choose appropriate model? We can select model which lead to higher value of adjusted $R^2$. Because both adjusted $R^2$ and information criterion penalize for additional number of regressors in the model, where former penalize $R^2$ and later penalize likelihood value.

$\endgroup$
  • $\begingroup$ I may be missing something in the answers (below) but R-squares as well as Adjusted R-squares are appropriate for the relatively limited class of OLS estimated models whereas AICs, BICs, etc., are appropriate for the broader class of generalized linear models estimated, perhaps, with ML or a variant. $\endgroup$ – Mike Hunter Feb 18 '16 at 13:59
12
$\begingroup$

I would argue that at least when discussing linear models (like AR models), adjusted $R^2$ and AIC are not that different.

Consider the question of whether $X_2$ should be included in $$ y=\underset{(n\times K_1)}{X_1}\beta_1+\underset{(n\times K_2)}{X_2}\beta_2+\epsilon $$ This is equivalent to comparing the models \begin{eqnarray*} \mathcal{M}_1&:&y=X_1\beta_1+u\\ \mathcal{M}_2&:&y=X_1\beta_1+X_2\beta_2+u, \end{eqnarray*} where $E(u|X_1,X_2)=0$. We say that $\mathcal{M}_2$ is the true model if $\beta_2\neq0$. Notice that $\mathcal{M}_1\subset\mathcal{M}_2$. The models are thus nested. A model selection procedure $\widehat{\mathcal{M}}$ is a data-dependent rule that selects the most plausible of several models.

We say $\widehat{\mathcal{M}}$ is consistent if \begin{eqnarray*} \lim_{n\rightarrow\infty}P\bigl(\widehat{\mathcal{M}}=\mathcal{M}_1|\mathcal{M}_1\bigr)&=&1\\ \lim_{n\rightarrow\infty}P\bigl(\widehat{\mathcal{M}}=\mathcal{M}_2|\mathcal{M}_2\bigr)&=&1 \end{eqnarray*}

Consider adjusted $R^2$. That is, choose $\mathcal{M}_1$ if $\bar{R}^2_1>\bar{R}^2_2$. As $\bar{R}^2$ is monotonically decreasing in $s^2$, this procedure is equivalent to minimizing $s^2$. In turn, this is equivalent to minimizing $\log(s^2)$. For sufficiently large $n$, the latter can be written as \begin{eqnarray*} \log(s^2)&=&\log\left(\widehat{\sigma}^2\frac{n}{n-K}\right) \\ &=&\log(\widehat{\sigma}^2)+\log\left(1+\frac{K}{n-K}\right) \\ &\approx&\log(\widehat{\sigma}^2)+\frac{K}{n-K} \\ &\approx&\log(\widehat{\sigma}^2)+\frac{K}{n}, \end{eqnarray*} where $\widehat{\sigma}^2$ is the ML estimator of the error variance. Model selection based on $\bar{R}^2$ is therefore asymptotically equivalent to choosing the model with the smallest $\log(\widehat{\sigma}^2)+K/n$. This procedure is inconsistent.

Proposition: $$\lim_{n\rightarrow\infty}P\bigl(\bar{R}^2_1>\bar{R}^2_2|\mathcal{M}_1\bigr)<1$$

Proof: \begin{eqnarray*} P\bigl(\bar{R}^2_1>\bar{R}^2_2|\mathcal{M}_1\bigr)&\approx&P\bigl(\log(s^2_1)<\log(s^2_2)|\mathcal{M}_1\bigr) \\ &=&P\bigl(n\log(s^2_1)<n\log(s^2_2)|\mathcal{M}_1\bigr) \\ &\approx&P(n\log(\widehat{\sigma}^2_1)+K_1<n\log(\widehat{\sigma}^2_2)+K_1+K_2|\mathcal{M}_1) \\ &=&P(n[\log(\widehat{\sigma}^2_1)-\log(\widehat{\sigma}^2_2)]<K_2|\mathcal{M}_1) \\ &\rightarrow&P(\chi^2_{K_2}<K_2) \\ &<&1, \end{eqnarray*} where the 2nd-to-last line follows because the statistic is the LR statistic in the linear regression case that follows an asymptotic $\chi^2_{K_2}$ null distribution. QED

Now consider Akaike's criterion, $$ AIC=\log(\widehat{\sigma}^2)+2\frac{K}{n} $$ Thus, the AIC also trades off the reduction of the SSR implied by additional regressors against the "penalty term," which points in the opposite direction. Thus, choose $\mathcal{M}_1$ if $AIC_1<AIC_2$, else select $\mathcal{M}_2$.

It can be seen that the $AIC$ is also inconsistent by continuing the above proof in line three with $P(n\log(\widehat{\sigma}^2_1)+2K_1<n\log(\widehat{\sigma}^2_2)+2(K_1+K_2)|\mathcal{M}_1)$. The adjusted $R^2$ and the $AIC$ thus choose the "large" model $\mathcal{M}_2$ with positive probability, even if $\mathcal{M}_1$ is the true model.

As the penalty for complexity in AIC is a little larger than for adjusted $R^2$, it may be less prone to overselect, though. And it has other nice properties (minimizing the KL divergence to the true model if that is not in the set of models considered) that are not addressed in my post.

$\endgroup$
  • 1
    $\begingroup$ Great answer: not too heavy but still exact! If it had been there yesterday, I would have not posted mine. $\endgroup$ – Richard Hardy Feb 18 '16 at 14:37
  • $\begingroup$ What about for the ARMA-GARCH case? How would $R^2_{adj} $ do at selecting amung MA and GARCH terms? $\endgroup$ – Zachary Blumenfeld Feb 18 '16 at 17:46
  • $\begingroup$ I wouldn't dare to say. As you explain it's not even clear what R2 means for the fit of such a model. $\endgroup$ – Christoph Hanck Feb 18 '16 at 18:31
5
$\begingroup$

The penalty in $R^2_{adj}$ does not yield the nice properties in terms of model selection as posessed by the AIC or BIC. The penalty in $R^2_{adj}$ is enough to make $R^2_{adj}$ an unbiased estimator of the population $R^2$ when none of the regressors actually belongs to the model (as per Dave Giles' blog posts "In What Sense is the "Adjusted" R-Squared Unbiased?" and "More on the Properties of the "Adjusted" Coefficient of Determination"); however, $R^2_{adj}$ is not an optimal model selector.

(There could be a proof by contradiction: if AIC is optimal in one sense and BIC is optimal in another, and $R^2_{adj}$ is not equivalent to either of them, then $R^2_{adj}$ is not optimal in either of these two senses.)

$\endgroup$
  • $\begingroup$ How many GARCH parameters do I have to add before the $R^2$ increases? :)....I believe a similar argument could be made for the assumption of correlated errors (as in an MA model), A GLS model does not reduce the sum of squared residuals over ordinary least squares. In both MA and GARCH, parameters (not explanatory variables, which $R^2{adj}$ is adjusted for) are added to the model. MA and GARCH parameters are not added to reduce $SSR$, rather they re added to increase the likelihood and/or decreases a weighted sum of squared residuals to reflect the lack of iid error terms. $\endgroup$ – Zachary Blumenfeld Feb 17 '16 at 21:07
  • $\begingroup$ Does this actually address the original post or my answer? In any case, I agree with your points. $\endgroup$ – Richard Hardy Feb 17 '16 at 21:15
  • $\begingroup$ What I was trying to point out is that $R^2_{adj}$ can't really be used to select upon GARCH components (and possibly MA components too) since it is based on the fraction of $SST-SSR$ over $SST$ which are biased estimators of variance when the error terms are not iid. (this is just a specific case of the bias you where talking about). In the case of ARMA-GARCH, you would never select a model with GARCH components, even if there was stochastic volatility in the data, because it does not increase $R^2$. Basically, I am agreeing with you by trying to give specific examples. $\endgroup$ – Zachary Blumenfeld Feb 17 '16 at 21:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.