3
$\begingroup$

Question Does the Gaussian $AR(1)$ model, with a fixed sample size $T$, have nontrivial sufficient statistics?

The model is given by $$ y_t = \rho y_{t-1}, \, t = 1, \cdots, T, \; \epsilon_i \stackrel{i.i.d.}{\sim} \mathcal{N}(0, \sigma^2), $$ parametrized by $(\rho, \sigma^2)$. Let $y = (y_1, \cdots, y_T)'$. Conditional on $y_0$, the likelihood function is \begin{align*} L(y) &= \frac{1}{ ( \sqrt{2 \pi \sigma^2} )^n } e^{-\frac{1}{2\sigma^2} ( \sum_{t = 1}^T (y_t - \rho y_{t-1})^2)} \\\\ &= \frac{1}{ ( \sqrt{2 \pi \sigma^2} )^n } e^{-\frac{1}{2\sigma^2} y' P(\rho) y}, \end{align*} where $P(\rho)$ is the $T \times T$ tridiagonal Toeplitz matrix given by $$ P(\rho) = \begin{bmatrix} 1 + \rho^2 & -\rho & 0 & 0 & & & \\ -\rho & 1 + \rho^2 & -\rho & 0 & & & \\ 0 &-\rho & 1 + \rho^2 & -\rho & & & \\ & & & & \ddots& & \\ & & & & -\rho & 1 + \rho^2 & -\rho\\ & & & & 0 & -\rho & 1\\ \end{bmatrix}. $$

My guess is that, if a nontrivial sufficient statistic exists, it would involve factorizing $P(\rho)$.

Comment

This situation is a little different with the case of the Gaussian linear model $$ Y = X \beta + \epsilon, \;\; \epsilon \stackrel{d}{\sim} \mathcal{N}(0, \sigma^2 I_T) $$ parametrized by $\beta \in \mathbb{R}^p$ and $\sigma^2 > 0$. The design matrix $X, T \times p$ is considered fixed. The likelihood function is \begin{align*} L(y) &= \frac{1}{ ( \sqrt{2 \pi \sigma^2} )^n } e^{-\frac{1}{2\sigma^2} ( Y - X \beta)'( Y - X \beta) } \\\\ &= \frac{1}{ ( \sqrt{2 \pi \sigma^2} )^n } e^{-\frac{1}{2\sigma^2} \left[ ( Y - X \hat{\beta})'( Y - X \hat{\beta}) + (\hat{\beta} - \beta)' X' X (\hat{\beta} - \beta) \right]}. \end{align*} This makes $(\hat{\beta}, s^2)$ sufficient (and minimal), where $\hat{\beta}$ is the OLS estimate $\hat{\beta}$ and $s^2 = \frac{1}{T-1} ( Y - X \hat{\beta})'( Y - X \hat{\beta})$. But in the $AR(1)$ case, it doesn't make sense to consider the covariates $y_{t-1}$ as fixed.

$\endgroup$
0
$\begingroup$

Turns out the answer is simpler than expected.

Consider the conditional form of the likelihood function is (still conditioning on $y_0$) \begin{align*} L(y) &= \frac{1}{ ( \sqrt{2 \pi \sigma^2} )^n } e^{-\frac{1}{2\sigma^2} ( \sum_{t = 1}^T (y_t - \rho y_{t-1})^2)}. \\\\ \end{align*}

As in the case of linear model $$ \sum_{t = 1}^T (y_t - \rho y_{t-1})^2 = \sum_{t = 1}^T (y_t - \hat{\rho} y_{t-1})^2 + (\hat{\rho} - \rho)^2 \sum_{t = 1}^T y_{t-1}^2 $$ where $\hat{\rho} = \frac{\sum_{t = 1}^T y_{t-1} y_t}{\sum_{t = 1}^T y_{t-1}^2}$ is the OLS estimate, and $\sum_{t = 1}^T (y_t - \hat{\rho} y_{t-1})^2$ is the residual sum of squares.

Evidently, $$ T_1 = (\sum_{t = 1}^T (y_t - \hat{\rho} y_{t-1})^2,\, \hat{\rho},\, \sum_{t = 1}^T y_{t-1}^2) $$ is a minimal sufficient statistic.

The only difference with the linear model case is the additional term $\sum_{t = 1}^T y_{t-1}^2 = X'X$. This is not surprising. Inspecting the calculation for the linear model, we see that taking the design matrix $X$ as fixed is unnecessary. One only needs to take $X'X$ as fixed. In the time series case, the statement "taking $\sum_{t = 1}^T y_{t-1}^2$ as fixed" doesn't make sense. Instead it appears as part of a sufficient statistic.

Equivalently, \begin{align*} \sum_{t = 1}^T (y_t - \rho y_{t-1})^2 &= \sum_{t = 1}^T y_t^2 - 2 \rho \sum_{t = 1}^T y_{t-1} y_t + \rho^2 \sum_{t = 1}^T y_{t-1}^2 \\\\ &= (1 + \rho^2) \sum_{t = 1}^T y_{t-1}^2 + y_T^2 - 2 \rho \sum_{t = 1}^T y_{t-1} y_t \end{align*} means $$ T_2 = (\sum_{t = 1}^T y_{t-1}^2,\, y_T^2,\, \sum_{t = 1}^T y_{t-1} y_t) $$ is another minimal sufficient statistic. Geometrically, it is clear that $T_2$ contains the same information as $$ T_1 = \mbox{ (sum of OLS square residuals, OLS estimate, squared norm of $(y_{t-1})$). } $$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.