19
$\begingroup$

In regular regression, the expected value of Y | X is allowed to change. In fact we generally use regression when we want to model this change in conditional mean.

I am not understanding why in time series, we want our series to be mean stationary. I get the stationary variance assumption as that's similar to the identically distributed assumption in regular regression. But why is mean stationarity so important?

$\endgroup$
  • $\begingroup$ Earlier, I posted a very basic response based on the title alone, which didn't take into account the details in your post about mean stationarity. After rereading your post, I have updated it with details more specific to your question, which I think it answers completely now. $\endgroup$ – Skander H. Jun 1 at 11:09
  • $\begingroup$ Take for example the levels of prices and levels of real GDP over time. Both tend to increase so are not stationary and so there is a positive correlation between them, sometimes a very high correlation as in the UK from 1993 to 2007. But it would be wrong to think this implies any sort of relationship between inflation and real GDP growth - the relationship between the time series for the levels is largely driven by the time series both change measures having positive means $\endgroup$ – Henry Jun 1 at 17:00
17
$\begingroup$

In the case of time series forecasting, first of all, you need to understand that stationarity is important mostly in the context of ARMA and related models (AR: Auto-Regressive, MA: Moving Average). There are other types of time series forecasting models where stationarity is not a requirement, such as Holt-Winters or Facebook Prophet.

Here are two intuitive, if not entirely mathematically rigorous, explanations of why mean stationarity is important in the ARMA case:

  • The AR component of ARMA models, treats time series modeling as a supervised learning problem, $Y_t = a_1Y_{t-1}+...a_nY_{t-n}+c+\sigma(t)$. A common rule of thumb in supervised learning is that the distribution of the training data and the distribution of the test data should be the same, otherwise your model will perform poorly on out-of-sample tests and on production data. Since for time series data, you train set is the past, and your test set is the future, the stationarity requirement is simply ensuring that the distribution stays the same over time. This way you avoid the problems that come with training your model on data that has a different distribution than the test/production distribution. And mean stationarity in particular is just saying that the mean of the train set and the mean of the test should stay the same.

  • An even simpler consideration: take the most basic ARMA model possible, an $AR(1)$ model: $$Y_t = aY_{t-1}+c+ \sigma$$ so the recursive relationship for estimating on step based on the previous one is: $$\hat{Y}_t = a\hat{Y}_{t-1}+c$$, $$\hat{Y}_t - c = a\hat{Y}_{t-1}$$ taking the expected value: $$E(\hat{Y}_t) - c = aE(\hat{Y}_{t-1})$$ meaning that: $$a = \frac{E(\hat{Y}_t) - c}{E(\hat{Y}_{t-1})}$$ so if we want $a$ to stay constant over time, which is the starting assumption of an $AR(1)$ model since we want it to be similar to a linear regression, then $E(\hat{Y}_t)$ has to stay the same for all $t$, i.e. you series has to be mean stationary.

The above considerations are applicable as well to the general ARMA case, with $AR(p)$ and $MA(q)$ terms, although the math is somewhat more complicated than what I describe, but intuitively, the idea is still the same. The 'I' in ARIMA stands for "Integrated" which refers to the differencing process that allows one to transform a more general time series into one that is stationary and can be modeled using ARMA processes.

I disagree with @Alexis characterization that "that time series are stationary is more or less embodying the worldview that the past does not matter" - if anything it is the other way around: Transforming a time series into a stationary one for modeling purposes is exactly about seeing whether there are any causal/deterministic structures in the time series beyond just trend and seasonality. I.e. does the past impact the present or the future in ways more subtle ways than just the large scale variations? (But I might simply misinterpreting what she is trying to say).

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ She, not he. I wonder if we are talking past each other? I agree about "beyond trends and beyond seasonality". My point is that if, say, you make a model, like $y_{ti} = \beta_0 + BX_{ti} + f(t,T) + \text{error}$ (i.e. "longitudinal models"), you are ignoring the dynamic/nonlinear nature of $y$'s past affect it at time $t$. $\endgroup$ – Alexis Jun 1 at 18:14
  • 1
    $\begingroup$ @Alexis my absolute deepest apologies. (Skander means "Alexander' in Arabic, so I tend to project myself onto anyone with a similar name pattern). And yes we do seem to be in agreement over the facts. One thing I would like to point out though it that in my experience, even a simple process that is "truly" AR and also stationary, is exceedingly rare. There are all sorts of non-stationary examples (e.g. population dynamics), but are the chances of a process being both stationary AND having a DGP that looks like $Y_t = a_1Y_{t-1}+a_2Y_{t-2}$ ? So ARIMA is a pretty strong assumption to make. $\endgroup$ – Skander H. Jun 1 at 18:29
  • $\begingroup$ No worries! Well, I did give two examples (consumption of addictive goods, and infectious disease prevalence) where the causal models must entail the past influencing the present. :) $\endgroup$ – Alexis Jun 1 at 20:15
  • $\begingroup$ "the causal models must entail the past influencing the present" but that's the point: They make perfect sense, but for the same reason that they make sense, they also can't possibly be stationary (e.g. the infectious disease example you gave shows exponential growth, not even linear or polynomial, which is the most ARIMA can handle with differencing). $\endgroup$ – Skander H. Jun 1 at 22:20
  • $\begingroup$ Well... you would need simultaneous equations, not a single equation, to for example, produce a compartmental model (and which can model the kinds of growth you are mentioning)... (Also: I am not arguing that ARIMA and attending to stationarity/non-stationarity are be-all and end all). Modeling (stationary) change is more important than modeling (non-stationary) level for making causal inferences. Stil: I like your point... it's chewy, and will be the kind of thing I think on, so thank you! $\endgroup$ – Alexis Jun 1 at 23:10
10
$\begingroup$

Stationarity is important because it is a mathematically strong assumption that's still much weaker than independence or finite-range dependence.

In some settings, it's important primarily for the mathematical tractability: it's easier to first find out what is true for stationary time series, then you can work on how to relax the assumptions. Perhaps you only need weak-sense stationarity, or mean stationarity plus some tail condition, or whatever. Or perhaps you need stationarity for a result to hold exactly, but it holds approximately under weaker assumptions.

In other settings stationarity is important because there are so many ways to be non-stationary that it would be hard to handle every one of them. If a problem can be approximated by a stationary series that's a big practical advantage. Here it's important to remember that the stationary series $X(t)$ that appears in the maths may not be your raw data. For example, traditional ARMA models are stationary, but you would typically want to remove season and trend relationships before fitting one. You might want to log-transform a series that has increasing mean and variance. And so on.

| cite | improve this answer | |
$\endgroup$
4
$\begingroup$

First, your mean estimates and your standard errors will be badly biased if you are using any of the inferential tools which assume i.i.d, meaning your results risk being spurious. This can even be true if your data are weakly stationary, but your study period is shorter than the time it takes your series to reach equilibrium after a disturbance.

Second, assuming that time series are stationary is more or less embodying the worldview that the past does not matter (e.g., the prevalence of COVID-19 today is completely independent of COVID-19 prevalence yesterday; the \$ per capita spent on addictive goods such as cigarettes this year is completely independent of the \$ per capita spent on them last year)… kinda unrealistic.

| cite | improve this answer | |
$\endgroup$
  • 4
    $\begingroup$ I respectfully disagree with your second statement. See my response. $\endgroup$ – Skander H. Jun 1 at 6:46
  • 6
    $\begingroup$ If the past did not matter, would collecting past data to make inference about the current properties of the process of interest or to predict future realization of the process make sense? $\endgroup$ – Richard Hardy Jun 1 at 16:08
  • 1
    $\begingroup$ @SkanderH. I do not think you understand my second statement. $\endgroup$ – Alexis Jun 1 at 18:15
  • 1
    $\begingroup$ @RichardHardy, "if the past did not matter" here means that a solution of the dynamic system is not path-dependent (except maybe in some trivial fashion), and there are fixed parameters (independent of the path) to be estimated by some suitable technique. $\endgroup$ – PatrickT Jun 2 at 9:20
  • 2
    $\begingroup$ @PatrickT, thank you, this is helpful. $\endgroup$ – Richard Hardy Jun 2 at 10:36
1
$\begingroup$

Stationary means that the statistics that describe the random process are constant. ‘A memoryless Markov process’ is another way to say stationary as is saying that the probability generating function has no “feedback” terms, but if you recognized those words you might not be asking this question. FWIW “weakly stationary” isn't quite the same, a constant or knowable rate of change of the stats would be weakly stationary, as would something that averages out, but it’s a little more involved so consider this fair warning that there’s more to know in case that’s part of the puzzle, but describing everything that isn’t stationary in detail would turn a simple answer a complex answer.

Why is stationary important? The commonly used statistical formulae are crafted to use a data set to extract an imprecise description with an estimable accuracy of an otherwise unknown random process. The formulae assume that adding more samples increases the accuracy of the description by reducing the uncertainty. For that the Mean Central tendency, i.e. ergodic in the mean, has to be true. If the random process itself is changing, e.g. the average value or the variance is changing, then an essential underlying assumption is invalid, you can’t make a better estimate.

As a general “what happens” if the mean is moving as a linear function of time, the computed mean will represent the mean at a weighted mean time, and the computed variance will be inflated. Is possible to compute an ‘optimal a posteriori” (after the fact) estimate of a non stationary process and then use that to extract meaningful stats because the best estimate of the time function minimizes the variance. It’s also easy to hypothesize some high order time function and create a complex model that appears to be valid and predictive that in fact has no predictive power because it modeled a snapshot of randomness, not an underlying time trend.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

Short and sweet:

The parameters need to be constant. If the series is not stationary, then the parameters that you estimate are going to be functions of time themselves. But the model assumes that they are constants, as such, you will estimate the average parameter value over the time-period. See Skander's answer for why, I won't dive into the math since he already did.

This presents at least 2 problems:

  1. Your estimates for the true parameter value are likely wrong, because at any moment in time the parameter value is likely to be different from its average value. Therefore, any inference that you make from the data is likely wrong. This leads to spurious regressions/correlations.
  2. You cannot use the model to predict the future. Since your parameter is now a function of time, and you don't know how it is evolving over time, any forecast that you make is complete (pardon my french) horseshit.

Getting to stationarity is actually pretty easy. We just need to difference until we have a stationary series. So just do that.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.