9
$\begingroup$

I would like to explore the different ways one can detrend a time series without look ahead bias.

I wanted to use the Hodrick Prescott filter, which seems like a quite good frequency filter, but it is based on an optimization method, and I understand that it may give strange and volatile results at the border.

Wavelet smoothing on a rolling window would be another option, but again border effects can be huge (the data is copied by symmetry which is horrible for the precision of the technic at the edge).

Any idea or comments?

PS: The subject has already been discussed here, I know. But I would like to dig a bit more on a more precise question.

Improve this question
$\endgroup$
5
  • $\begingroup$ @RockScience I've added two tags. Could you confirm they're correct? $\endgroup$
    – chl
    Commented May 4, 2011 at 8:38
  • 1
    $\begingroup$ @RockScience, what particular problem requires de-trending without look ahead bias? I personally worked with different de-trending schemes analyzing (not forecasting) the business cycles. So the common thing is to pay little attention to the borders (i.e. ignore the points at the begining and end of period). The same bias may be lowered for instance looking for an appropriate model that drives the trend (either deterministic or stochastic) at the sample without edges and simply forecasting values. Building such robust model with unbiased predictions would be an art anyway. $\endgroup$
    – Dmitrij Celov
    Commented May 4, 2011 at 10:49
  • $\begingroup$ (+1) for the questions, if one knows the solution it would be nice to hear. Regarding HP, we may go for optimal filter (optimization in $\lambda$), but the bad edge behavior remains. $\endgroup$
    – Dmitrij Celov
    Commented May 4, 2011 at 10:52
  • $\begingroup$ @Dmitrij Celov: You are lucky! I need to estimate the current trend. I need also to apply the same process in the past to evaluate the methodology. I need to do a backtest of this method, and that's why I am interested ONLY in the last datapoints. $\endgroup$
    – RockScience
    Commented May 4, 2011 at 10:59
  • 1
    $\begingroup$ Depending on the length of your data set, you could try to look for look-ahead value of bias first (say you have 100 points take 50 for evaluation of trend by say HP on the whole sample and only on the first 50 points, then move by one the window from 2 to 51 and so on, look at the accuracy of your edges). May be in your case it is not so huge as you imagine. $\endgroup$
    – Dmitrij Celov
    Commented May 4, 2011 at 11:05

3 Answers 3

Reset to default
3
$\begingroup$

There is no way to get rid of the end effects.

Like any interpolation technique, the HP method depends on data before and after the current location to provide a filtered point/line for that location. As you approach either end of the data series and drop below the required number of future (or past) points, you either don't provide the filtered line or the characteristics of the filtered line must change.

It is dangerous to blindly extend the line and assume that it has the same properties at the ends of the series as it does in the middle. The bottom line is, the HP filter has no predicting power.

Improve this answer
$\endgroup$
0
$\begingroup$

De-trending requires a pre-specification of of how many values do you require before declaring that a new trend has started. Given this specification , say n values then one has to be concerned with distinguishing between Level Shifts ( i.e. intercept changes ) and time trend changes. If you assume that there are no Level Shifts then simply search for different points in time and select those points which have been find to be statistically significant. For example if you have the series 1,2,3,4,5,7,9,11 ...this would suggest two points in time where the trend "changed" , period 1 and period 5. Alternatively if you have a series like 0,0,0,0,0,1,2,3,4,5,,,,,, there is only 1 point in time where a significant trend is evidenced i.e. period 5 . Outliers and ARIMA structure in a time series can lead to distortions in the identification of trend-point changes and may need to be incorporated prior to trend-point detection. A recent paper on tree-ring data http://www.autobox.com/pdfs/forestdisturbance.pdf discusses this issue.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I am not sure I fully understand the difference between "Level Shifts ( i.e. intercept changes)" and "time trend changes". Can you provide with an example for each? $\endgroup$
    – RockScience
    Commented May 4, 2011 at 9:37
  • $\begingroup$ @RockScience . Level Shift 1,2,3,4,5,8,9,10,11,12 .... ls at t=6 Another example of a ls change 1,1,1,1,1,2,2,2,2,2,... ls at t=6, Time trend change 1,2,3,4,5,7,9,11,13,15,17....tt change at t=5 $\endgroup$
    – IrishStat
    Commented May 4, 2011 at 9:52
  • $\begingroup$ trends not necessarily are deterministic, so such knowledge could be of no use in real time-series applications. Even in filtering you deal with unknown data generating process and some believes about what the frequencies should be passed through by the ideal filter, so you have huge trade-offs in this case. In deterministic systems though you could go for structural changes detection. $\endgroup$
    – Dmitrij Celov
    Commented May 4, 2011 at 10:57
  • $\begingroup$ @Dmitri: one possible model to deal with trend is y(t)=theta0 + y(t-1) + a(t) . This easily generalizes to y(t)=[theta0 + P(T1)] + y(t-1) + a(t) where the steady state ramp changes at period T1. In contrast y(t)=3*x1 + 2*x2 + a(t) where x1=1,1,2,3,4,...t and x2=0,0,0,0,0,1,2,3,4,...t-5 reflect points in time where the trend changed ( i.e. period 1 and period 6 ). Careful analytics will suggest without any presumed knowledge which of these MODELS is most appropriate for a particular data set. The ARMA component must also be empirically found to complete the model along with any Level Shifts. $\endgroup$
    – IrishStat
    Commented May 4, 2011 at 11:29
0
$\begingroup$

One possibility would be to forecast/backcast both endpoints.

Many seasonal adjustment methods like X12-ARIMA and TRAMO-SEATS do that.

If you apply centered moving averages to the data, then you must somehow have more observations than series has. Some future and past values are needed.

Regards,

-A

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.