In this AR1 model, if we fit y with diff(y), regardless the true coefficient, when N is large, it seems the model will converge to (0,-0.5), why?

> set.seed(0)
> ts.sim <- arima.sim(list(order = c(1,0,0), ar = c(0.1)), n = 1e6)
> x= diff(ts.sim)
> y=ts.sim[-1e6]
> lm(y~x)

lm(formula = y ~ x)

(Intercept)            x  
 -4.376e-05   -5.000e-01  

> ts.sim <- arima.sim(list(order = c(1,0,0), ar = c(0.9)), n = 1e6)
> x= diff(ts.sim)
> y=ts.sim[-1e6]
> lm(y~x)

lm(formula = y ~ x)

(Intercept)            x  
  -0.003551    -0.500002  
  • $\begingroup$ Are you sure you want y[-1e6] rather than y[-1]? This will hardly make a numerical difference, but the interpretation of the model will be different. Now you are fitting $y_t\sim\Delta y_{t+1}$, after the correction you would be fitting $y_t\sim\Delta y_{t}$. $\endgroup$ – Richard Hardy Jan 21 at 13:27
  • $\begingroup$ Your results can easily be explained (from the well known autocovariance function of the AR(1) process). But it's unclear why are you regressing $y_t$ on $y_{t+1}-y_t$ so I vote to close. To recover the autoregressive coefficient via OLS you should regress $y_t$ on $y_{t-1}$. $\endgroup$ – Jarle Tufto Jan 21 at 13:53
  • 4
    $\begingroup$ @JarleTufto, you probably misunderstood the OP. Based on his recent posts, he is not trying to recover the autoregressive coefficient using the model here. He is wondering why the estimate is always 0.5 regardless of what AR coefficient is used when the underlying time series is generated. And could you please explain the result that is easy for you? $\endgroup$ – Richard Hardy Jan 21 at 13:56
  • $\begingroup$ @RichardHardy Then what is he trying to do? $\endgroup$ – Jarle Tufto Jan 21 at 13:56
  • 1
    $\begingroup$ @JarleTufto, please see my comment above. $\endgroup$ – Richard Hardy Jan 21 at 13:58

This is a law of large numbers in action.

Let $\rho$ be the parameter (the lag-1 correlation) and let $\varepsilon_i$ be a sequence of iid standard Normal variables, so that for $i=1, 2, \ldots,$ the model is

$$y_{i+1} = \rho y_i + \varepsilon_{i+1}$$

and $y_0=0.$ Therefore the first differences are

$$x_{i+1} = y_{i+1} - y_i = (\rho-1)y_i + \varepsilon_{i+1}.$$

Because $y_0=0,$ inductively the $x_i$ and $y_i$ all have expectations of zero.

Because $\rho \lt 1,$ notice that $x$ and $y$ move in opposite directions: they must be negatively correlated. You might (intuitively) think that the amount of correlation depends on $\rho.$ However, it also depends on the variances of the $x_i$ and $y_i.$ We need to examine these.

As in the question, stop the series at $n$ and regress $y$ against $x.$ The slope estimate is

$$\hat \beta = \frac{\sum_i (y_i-\bar y)(x_i - \bar x)}{\sum_i (x_i-\bar x)} \approx \frac{\sum_i y_i x_i}{\sum_i x_i^2}$$

because, for large $n,$ the means $\bar y$ and $\bar x$ will be close to their expectations, which are zero.

Again because $n$ is large, the empirical values in the fraction can be approximated by their expectations (a law of large numbers),

$$\frac{\sum_i y_i x_i}{\sum_i x_i^2} \approx \frac{E\left[\sum_i y_i x_i\right]}{E\left[\sum_i x_i^2\right]}.$$

Find those expectations from the definition of the series $(y_i).$ I will begin with a useful auxiliary calculation:

$$\begin{aligned} E\left[\sum_i y_i^2\right] &= \sum_i E\left[y_i^2\right] = \sum_i E\left[(\varepsilon_{i+1} + \rho y_i)^2\right]\\ &= \sum_i E\left[(\varepsilon_{i+1})^2\right] + 2\rho \sum_i E\left[\varepsilon_{i+1} y_i\right] + \rho^2 \sum_i E\left[ y_i^2\right]\\ &= n + 0 + \rho^2 E\left[\sum_i y_i^2\right] \end{aligned}$$

which is justified because the $\varepsilon_i$ have zero expectation, unit variance, and are independent of each other (whence, in particular, $\varepsilon_{i+1}$ is independent of $y_i$). Solving this equation gives

$$E\left[\sum_i y_i^2\right] = \frac{n}{1-\rho^2}.$$

With this (standard) result in hand we obtain, using similar calculations,

$$E\left[\sum_i y_i x_i\right] = -\frac{n}{1+\rho}$$


$$E\left[\sum_i x_i^2\right] = \frac{2n}{1+\rho},$$


$$\hat \beta \approx \left(-\frac{n}{1+\rho}\right)\,/\,\left(\frac{2n}{1+\rho}\right) = -\frac{1}{2}.$$

(You can carry out this analysis rigorously using the Delta method, which applies because the denominator of $\hat\beta,$ $\sum_i x_i^2,$ will stay away from zero.)

Finally, because the linear regression passes through the point of averages and the means of the $x_i$ and $y_i$ are each close to $0,$ the constant term will be close to $0,$ too.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.