1
$\begingroup$

Consider the following $AR(p)$ model $$Y_{t}=\sum_{j=1}^{p}\phi_{j}Y_{t-j}+\epsilon_{t},$$ where $\epsilon_{t}\sim i.i.d\ \mathcal{N}(0,\sigma^{2}).$ Say we have samples $t=1,\dots, T$. The true lag is denoted by $p_{0}$. We don't know about the true lag, but we know that it is a bounded finite number $p_{0}\leq \overline{p}<\infty$. Normally, we use information criteria to select the proper lag. The information criteria, in general, is $$S_{p}:=\log\dfrac{SSR_{p}}{T}+pg(T),$$ where $g(T)$ is a deterministic function of $T$ such that $g(T)\rightarrow 0$ and $Tg(T)\rightarrow\infty$ when $T\rightarrow\infty$.

Then information criteria selection process is to choose $$\hat{p}:=\arg\min_{p\leq\overline{p}}S_{p}.$$

My question is, why does $\hat{p}=\hat{p}(T)$ provide consistent estimation of the real $p_{0}$? In other words, how to prove that $$\mathbb{P}(\hat{p}=p_{0})\rightarrow 1,\ \text{when}\ T\rightarrow\infty?$$


I tried a little but I can never connect this probability with $S_{p}$. For example, we know that $\hat{p}$ is the minimizer of $S_{p}$, and thus we must have $$S_{\hat{p}}\leq S_{p_{0}},$$ which then can be written as $$\log\dfrac{SSR_{\hat{p}}}{SSR_{p_{0}}}\leq (p_{0}-\hat{p})g(T).$$ But then i don't know what to do.

I am also happy to know if there is any reference of the formal proof. Thank you!


Edit 1: One Idea

I found a way to connect $p$ to $S_{p}$. If we consider the set $\{p:p\leq \overline{p}\}$, it is just bunch of integers $\{0,2,\dots, p_{0}-1,p_{0},p_{0}+1,\dots, p\}.$ We denote this set as $A$, and $\hat{p}$ is somewhere in this set. We can view $\hat{p}\geq p_{0}$ as $\hat{p}$ is positioned to the right of $p_{0}$ or equals to $p_{0}$.

We know that $\hat{p}$ is the minimizer of $S_{\hat{p}}$. Let us assume that it is the unique minimizer. Then, $$\hat{p}\geq p_{0}\iff S_{\hat{p}}< S_{0},\dots, S_{\hat{p}}< S_{p_{0}-1}.$$

I am still thinking of how to go from here.

$\endgroup$
2
  • $\begingroup$ Related to stats.stackexchange.com/questions/197112/…, in which you can the penalty to your conditions to see consistency. $\endgroup$ Commented Feb 20 at 10:09
  • $\begingroup$ @ChristophHanck Thx for the link! I read this post before I posted this question, and I believe that this post is more about the R-sqaure. I don't see how to adapt the proof. $\endgroup$ Commented Feb 20 at 11:54

2 Answers 2

1
$\begingroup$

Building on Why information criterion (not adjusted $R^2$) are used to select appropriate lag order in time series model?, we may write information criteria with $K$ parameters as (for general regression models and not just AR(p)) $$ IC=\log(\widehat{\sigma}^2)+g(n)K, $$ where $K$ denotes the number of parameters and $\widehat{\sigma}^2$ the error variance estimate $SSR_K/T$.

E.g., the AIC has $g(n)=2/n$ and thus $ng(n)=2$, so that the condition is not satisfied (and the link shows that we do not have consistency for AIC).

In turn, the BIC, which is consistent, has $g(n)=\ln (n)/n$, so that the condition is satisfied.

We choose, as usual, $\mathcal{M}_1$ if $IC_1<IC_2$, else select $\mathcal{M}_2$.

We will show that the probability that the IC for the true order (model $\mathcal{M}_1$ with $p=p_0$ in the AR(p) case) is less than the IC for any larger lag order (a model $\mathcal{M}_2$ with $p>p_0$ in the AR(p) case) with probability tending to one, so that the probability that an overly large order is chosen goes to zero.

From the link, we choose the smaller model $\mathcal{M}_1$ with $K_1$ parameters over the larger with $K_1+K_2$ parameters with probability $$ P\bigl(IC_1<IC_2\bigr)=P(n\log(\widehat{\sigma}^2_1)+ng(n)K_1<n\log(\widehat{\sigma}^2_2)+ng(n)(K_1+K_2)) $$ Continuing the linked proof, we may rewrite this, if the smaller model $\mathcal{M}_1$ is indeed the correct one as (in the notation, we "condition on" the smaller model) \begin{eqnarray*} P\bigl(IC_1<IC_2|\mathcal{M}_1\bigr)&=&P(n[\log(\widehat{\sigma}^2_1)-\log(\widehat{\sigma}^2_2)]<ng(n)K_2|\mathcal{M}_1) \\ &\rightarrow&P(\chi^2_{K_2}<ng(n)K_2) \\ &=&1 \end{eqnarray*} if $ng(n)\to\infty$. Again, the second-to-last line follows because the statistic is the LR statistic in the linear regression case that follows an asymptotic $\chi^2_{K_2}$ null (which it is because we consider the case in which the smaller model is the correct one) distribution.

The last line follows because the $\chi^2_{K_2}$ random variable is $O_p(1)$, and hence smaller than the r.h.s w.p.a 1 if the latter diverges.

Why do we not asymptotically pick the lag order too small? This case heuristically can be covered by noting that a relevant lag would then "go into" the error term, which will then have a larger variance $\hat\sigma^2$ and hence a large IC, so that the argmin will be at the true lag order.

More specifically, we may then reverse the role of the correct and false model in the above derivations, with $\mathcal{M}_1$ the smaller, incorrect and $\mathcal{M}_2$ the larger correct model. We then have that, under the "alternative", the LR statistic $$n[\log(\widehat{\sigma}^2_1)-\log(\widehat{\sigma}^2_2)]=O_p(n),$$ so that the probability that the larger model will be chosen now is $$ \begin{eqnarray*} P\bigl(IC_1>IC_2|\mathcal{M}_2\bigr)&=&P(O_p(n)>ng(n)K_2|\mathcal{M}_2) \\ &\to&1, \end{eqnarray*} $$ as $g(n)\to0$ implies that $ng(n)=o(n)$, i.e., the lhs in the inequality diverges faster that the rhs, implying consistency.

A little Monte Carlo for $\bar p=3$ and $p_0=2$ illustrating that the selection $\hat p$ centers on the correct $p_0$ for larger $n$:

enter image description here

ARMA.BIC <- function(n){
  y <- arima.sim(list(ar=c(0.4,0.3)), n=n)
  which.min(sapply(1:3, function(i) BIC(arima(y, order=c(i,0,0)))))
} 

par(mfrow=c(2,2))
n <- c(50, 100, 250, 500)
lapply(n, function(i) barplot(table(replicate(2000, ARMA.BIC(i))), col="darkgreen", main=paste0("n=",i)))
$\endgroup$
4
  • $\begingroup$ How does this connect to $\mathbb{P}(\hat{p}= p_{0})$? Under this notation, $IC_{1}$ is the $S_{\hat{p}}$. By definition it is less than everyone because it is the minimizer. $\endgroup$ Commented Feb 20 at 12:22
  • 1
    $\begingroup$ Do you mind me leaving this post open for a while so that it can get more exposed? Nice answer! And I will try to work out a proof based on your work. Thank you very much. $\endgroup$ Commented Feb 20 at 14:15
  • 1
    $\begingroup$ Sure. I added some further detail $\endgroup$ Commented Feb 20 at 15:45
  • $\begingroup$ I think that you were correct but it is hard to connect to the desired probability $\mathbb{P}(\hat{p}=p_{0})$. Based on your proof, especially the $O_{P}(1)$-argument, I believe that I figured it out. Thanks for the fruitful discussion! $\endgroup$ Commented Feb 24 at 17:28
1
$\begingroup$

Ok, I figured it out.

Note that \begin{align*} \mathbb{P}(\hat{p}<p_{0})&=\mathbb{P}(S_{j}<\min\{S_{p_{0}},\dots, S_{\overline{p}})\ \text{for some}\ j<p_{0})\\ &\leq \mathbb{P}(S_{j}<S_{p_{0}})\\ &=\mathbb{P}\Big(T\log\dfrac{\text{SSR}_{j}}{\text{SSR}_{p_{0}}}< (p_{0}-j)Tg(T)\Big) \end{align*} But note that $j$ is the under-specified model so we always have $\text{SSR}_{j}>\text{SSR}_{p_{0}}$. Hence, $T\log\frac{\text{SSR}_{j}}{\text{SSR}_{p_{0}}}\rightarrow \infty$ with $\sim T$ speed. However, since $p_{0}-j>0$, $(p_{0}-j)Tg(T)\rightarrow\infty$ with less than $T$ speed because $g(T)\rightarrow 0$. Hence, $$\mathbb{P}(\hat{p}<p_{0})\rightarrow\mathbb{P}(\infty<\text{finite})=0.$$


Conversely, \begin{align*} \mathbb{P}(\hat{p}>p_{0})&=\mathbb{P}(S_{j}<\min\{S_{0},\dots, S_{p_{0}})\ \text{for some}\ j>p_{0})\\ &\leq \mathbb{P}(S_{j}<S_{p_{0}})\\ &=\mathbb{P}\Big(T\log\dfrac{\text{SSR}_{j}}{\text{SSR}_{p_{0}}}< (p_{0}-j)Tg(T)\Big) \end{align*} As $T\rightarrow\infty$, $(p_{0}-j)Tg(T)\rightarrow -\infty$ because $Tg(T)\rightarrow\infty$ and $j>p_{0}$. On the other hand $T\log\dfrac{\text{SSR}_{j}}{\text{SSR}_{p_{0}}}=O_{p}(1),$ so it is bounded in probability, which means that $$\mathbb{P}(\hat{p}>p_{0})\rightarrow\mathbb{P}(\text{bounded}<-\infty)=0.$$

The desired convergence $\mathbb{P}(\hat{p}=p_{0})\rightarrow 1$ follows immediately.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.