Detrending a nice series with dummys and a break

I have a problem with a time series. In short: How do I detrend a time series with a break and some big outliers?

Long version: I want to detrend a time series into a trend and some cyclical components. The time series looks like this:

and the values can be found here

Using the augmented Dickey-Fuller-Test I get the following results:

Which means I can not reject the Null hypothesis that my time series has a unit root (the same is true when I use a constant without a trend or no constant and no trend). Even though unit root tests have a low power I don't think I should reject the Null hypothesis.

Looking at the time series again this is what I see. There are some huge outliers at t=18 and around t=100. Apart from that there seams to be a permanent shock (break) in the time series at around t=40.

Looking at the literature I found some reasonable explanations for the split in the time series as well as the big outliers at t=18. My problem is the following: How do I detrend the time series allowing for a break and without considering the outliers?

As far as I know there are only two possibilities:

(1) Using an OLS regression from t=1 to t=40 as well as some dummies for t=17, t=18 and t=19 and estimating another regression from t=41 to t=106.

Problem: the cyclical components for t=17, 18 and 19 will be zero so I loose effectively 3 observations. Another problem is that this method does not really make the series stationary.

or

(2) Replace the big outliers at t=17, t=18 and t=19 with some new values base in interpolation (maybe linear Interpolation) see for example this paper. After this I would estimate a quadratic regression or a moving average.

p. s. I know that there are many other methods to detrend time series: First difference (values are negative so I can't use this), moving average (requires a lot of observations), Hodrick-Prescott-Filter (I don't really want to use this one) and Bandpass Filter (not sure about this).

Sorry for the messy post.

Thank you for your answers.

p.s. (1) Using an AR(1) Model: $$y_t=\alpha_0 +\alpha_1 y_{t-1}+\varepsilon_t$$ I get stationary residuals that look like this:

There are some big outliers at t=18;19;20;21 that are the result of an exogenous shock that I don't want to consider in my considerations). My idea was to use the regression AR(1) with some dummy variables (I try different settings (alone and together) with dummys for t=18; t=19;t=20 and t=21). But when I do this the residuals become non-stationary again (For the Dickey-Fuller Test I can not reject the null Hypothesis while for the Phillips-Perron Test I can...). So I'm stuck again...

2 Answers

The question "How to detrend ...." begs the question "is there a trend that should be isolated and adjusted for " . Your unit of measurement required some non-intrusive magnification (10**3) in order to assess model structure. A reasonable model did not contain a trend to be excerpted and here is the actual,fit and forecast . The actual equation based upon a deterministic error variance reduction http://docplayer.net/12080848-Outliers-level-shifts-and-variance-changes-in-time-series.html at period 45 is here . The plot of the model errors provides reasonable (but not perfect) suggestion of model adequacy

Time series data ( and we have seen a few !) are like people .. some are easy to understand and require "simple tools" others can be a bit develish like this "nice series" warrant more nuanced approaches to sort out the underlying structure (the model !) . Recall all models are wrong but some are useful !

In summary there is no region in the 106 values that support a trended equation even the relatively stable period of 37-97 .

Ask not "what you can do to your data" but rather "ask the data what needs to be done to characterize it"

You say " First difference (values are negative so I can't use this), moving average (requires a lot of observations)," . I don't believe either of your assertions are corrrect. In this case first differences are quite appropriate and an Ma(1) model would be just as useful as the identified AR(1) model since ph1=.4

[1-.4B]-1 = [1+.4B + .16B2 + .064B**3 ... ]

EDITED TO PROVIDE A STEP-BY-STEP APPROACH TO DEAL WITH MODEL IDENTIFICATION:

A MASTER-CLASS IN TIME SERIES MODELLING

STEP 1: YOU SPECIFIED AN AR(1) . I estimated that model here

with residuals here with the following ACF

There are three Gaussian violations that possibly need to be addressed :

1. there are one-time pulses in the residuals suggesting the need to incorporate pulse indicators (0/1) predictors
2. there is significant autocorrelation in the residuals suggesting needed ARIMA augmentation
3. there is significant non-constant error variance in the residuals suggesting the need to do WEIGHTED LEAST SQUARES (Generalized Least Squares) by adjusting (reducing the volatility) of the first 44 observations

You didn't address any of these violations.

STEP 2: Specifying a first difference model with an AR(1) (0,1,0)(1,0,0) and 6 pulse indicators and a test for a breakpoint in error variance

Note that AUTOBOX found a breakpoint in the error variance suggesting that the first 44 values by down-weighted by .64 (square root of .406) culminating here

Not that the ACF of the residuals from this model indicate insufficiency as there appears to be some sort of a four period effect in the residuals suggesting further iterations. Peeling an onion comes to mind ! Is there any reason that you know of that would suggest this phenomenon ? It could represent an exogenous/unspecified predictor variable like an election cycle effect.

A plot of the residuals suggests that the error variance is much more homogeneous but with the 43-64 region still suggesting some non-constnt error variance.

• Thank you for your answer and the work you put into your analysis. I'm still trying to understand everything but I already have two questions. (1) You say (and show) that there is no trend in the time series. But all tests for stationarity show that the series is non-stationary. I know that all these tests have a very low power but still it's strange for me to think that all the tests are wrong (I don't want to say your statement is wrong - It's just that this would make all these tests pretty worthless) (2) What do you mean with "In this case first differences are quite appropriate"?
– PAS
Feb 13 '18 at 11:45
• The series is non-stationary in the mean. The remedy for non-stationarity in this case is to take first differences. See the (1-B) operator in the AUTOBOX model. For some reason you appear to think that the non-stationarity conclusion requires a trended model including a predictor series called "time" i.e. a series containing the counting numbers 1,2,3....t . Additioanally the series is non-stationary in the error variance . Feb 13 '18 at 13:03
• Thank you again for your answer. I don't understand how you estimated the residuals but I tried to use an AR(1) model and edited my original post. Unfortunately I sill have some big outliers that I don't want to use.
– PAS
Feb 15 '18 at 13:11

Upon detection that there might be quarterly effects , I re-entered the data to AUTOBOX specifying that it was quarterly data and obtained the following model (1,1,0)(1,0,0)4 . The first 44 observations were found to be statistically significantly different from the last 62 thus the data was segmented. A plot of the Actual/fit and Forecast is here . The acf of the model residuals is here an residual plot is here . Since the acf of the model residuals suggests a near-perfect fit I can only conclude that this data might have been artificially generated and that an important piece of information was left out i.e. the data was quarterly data.

• Thank you. The data is quarterly data. But I was pretty sure that there would be no quarterly effect based on the way the data were determined as well as the fact that FRED uses a quite similarly data series and I asked them why they don't seasonally adjust it. There answer was that there is no need to do this. About your hints in your first post: (1) I tried to incorporate some one time pulses in my original AR(1) regression but by doing this the residuals become nonstationatry again. (3) I tested different methods without much success.
– PAS
Feb 16 '18 at 8:54
• I really appreciate all the work you put into this analysis and I'm very grateful for it but I think there is a small misunderstanding. I know that one has to estimate the regression to get a good model and to use the residuals later on. But my goal is mainly to make the series stationary and to omit the big outliers between in the first part of the regression (the big decline between period 15 and).
– PAS
Feb 16 '18 at 9:00
• To your point that " 1) I tried to incorporate some one time pulses in my original AR(1) regression but by doing this the residuals become nonstationatry again. (3) I tested different methods without much success. – " ... your ARIMA model is mispecified as it is definitely not an AR(1) ... It is a first difference model with an AR(1) . Incorporating pulse indicators with an incorrect ARIMA model is unproductive Feb 16 '18 at 9:08
• there is a definite quarterly efffect BUT it is obfuscated by the parameter change/error variance error variance change suggesting either data segmentaion or Generalized Least Squares culminating in focusing on the last 62 data points or so. Feb 16 '18 at 9:13