How do I detrend time series? Is it ok to just take first difference and run a Dickey Fuller test, and if it is stationary we are good?

I also found online that I can detrend the time series by doing this in Stata:

reg lncredit time
predict u_lncredit, residuals
twoway line u_lncredit time
dfuller u_lncredit, drift regress lags(0)

What is the best approach to detrend time series?

  • The code may be fairly transparent to non-Stata users, but note that the detrending is to work with the residuals from a linear regression on time. – Nick Cox Oct 16 '14 at 19:24

If the trend is deterministic (e.g. a linear trend) you could run a regression of the data on the deterministic trend (e.g. a constant plus time index) to estimate the trend and remove it from the data. If the trend is stochastic you should detrend the series by taking first differences on it.

The ADF test and the KPSS test can give you some information to determine whether the trend is deterministic or stochastic.

As the null hypothesis of the KPSS test is the opposite of the null in the ADF test, the following way to proceed can be determined beforehand:

  1. Apply the KPSS to test the null that the series is stationary or stationary around a trend. If the null is rejected (at a predetermined level of significance) conclude that the trend is stochastic, otherwise go to step 2.
  2. Apply the ADF test to test the null that a unit root exists. If the null hypothesis is rejected, then conclude that there is no unit root (stationarity), otherwise the result of the procedure is not informative since none of the tests rejected the corresponding null hypothesis. In that case it may be more cautions to consider the existence of a unit root and detrend the series by taking first differences.

In the context of structural time series models you could fit a local-level model or a local-trend model to the data to get an estimate of the trend and remove it from the series. The local-trend model is defined as follows (the local-level model is obtained with $\sigma^2_\zeta=0$):

\begin{eqnarray} \begin{array}{rll} \hbox{observed series:} & y_t = \mu_t + \gamma_t + \epsilon_t , & \epsilon_t \sim \hbox{NID}(0,\, \sigma^2_\epsilon) ; \\ \hbox{latent level:} & \mu_t = \mu_{t-1} + \beta_{t-1} + \xi_t , & \xi_t \sim \hbox{NID}(0,\, \sigma^2_\xi) ; \\ \hbox{latent drift:} & \beta_t = \beta_{t-1} + \zeta_t , & \zeta_t \sim \hbox{NID}(0,\, \sigma^2_\zeta) ; \\ \end{array} \end{eqnarray}

  • 2
    The ADF and KPSS tests have tons of assumptions which if not met yield false conclusions. Lack of Pulse Outliers etc , presence of ARIMA structure , presence of time varying error variance etc. are just some of the assumptions. In my opinion they should be studiously avoided and you second suggestion be implemented where a suitable combination of memory and dummy indicators is selected. – IrishStat Oct 17 '14 at 15:31
  • 1
    Not too mention structural breaks which might get the tests to indicate a unit root when in fact there is none! In that case a unit root test which allows for endogenous structural breaks could be used. – Plissken Oct 17 '14 at 16:34
  • I wouldn't say that unit root tests have tons of assumptions but I agree that we must be careful because the presence of level shifts or structural breaks may lead to wrong conlusions with these tests. For example, we already discussed here that the Nile time series does not require differencing despite it is the practice followed in many places. Since the paper by Perron (1989) published in Econometrica vol. 57 there has been a large concern about this issue, as witnessed by the number of papers published in this field. – javlacalle Oct 17 '14 at 17:54

You have several ways of detrending a time-series with the aim of making it stationary:

  • The linear detrending is what you copied. It may not give you what you desire as you arbitrarily fix a deterministic linear trend.

  • The quadratic detrending is in some ways similar to the linear detrending, except that you add a "time^2" and supposes a exponential-type behavior.

  • The HP-filter from Hodrick and Prescott (1980) allows you to extract the non-deterministic long-term component of the series. The residual series is thus the cyclical component. Be aware that, as it is an optimal weighted average, it suffers from endpoint bias (the first and last 4 observations are wrongly estimated.)

  • The Bandpass filter of Baxter and King (1995) which is essentialy a Moving Average filter where you exclude high and low frequencies.

  • The Christiano-Fitzgerald filter.

To sum up, it depends on what your intention is and some filters may be better suited to your needs than others.

  • "Whenever something can be done in two ways, someone will be confused." (That's a comment not on filters / spectral analyses, but on my own inadequacy.) See also why-so-many-methods-of-computing-psd on dsp.se . – denis Apr 6 '16 at 10:17

Perhaps there is more than one trend . Perhaps there is a level shift . Perhaps the error variance has changed over time In any case a simple de-trending might be inappropriate. Good exploratory analysis along the line of http://www.unc.edu/~jbhill/tsay.pdf should be used to discover the nature of the data/model.

I suggest to take a look at Singular Spectrum analysis. It is a nonparametric technique which can be very roughly seen as PCA for time series. One of useful properties is that it can effectively de-trend series.

You need to research this subject carefully and can start here.

http://www.stat.pitt.edu/stoffer/tsa3/

The key thing you are looking for is stationarity or non-stationarity because most statistical tests assume that data is distributed normally. There are different ways to transform data to make it stationary. Detrending is one of the methods but would be inappropriate for some kinds of non-stationary data.

If the data is a random walk with trend then you may have to use differencing.

If the data shows a deterministic trend with a seasonal or other deviation from the trend you should start with detrending.

You may have to experiment with different approaches.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.