Time Series detrending with multiple polynomials

Question

I'm currently working on energy demand forecasting using daily load data. I'm using data since 1990 and, in order to use ARIMAX models, I detrended the data (using a first-order polynomial) and then I removed seasonality. As we can see on this image ,

in the last two years there is (unfortunately) a different trend with respect the previous 15 years. I was thinking to use two different first-order polynomials to perform detrend: one fitted on the data 1990-2008 and another one for 2008-2010. What is the best way to do this? I'm not sure if I must use a spline or not. Can you suggest me the best-practice method to perform this kind of detrending?

Thank you, P.S. I'm a computer scientist and I'm using R (and MATLAB).

UPDATE

This is the loess smothing with span 0.9

With span 0.5

and finally with span 0.1

I will try to perform detrending with span 0.5.

** UPDATE 2 ** This is the comparison with the time-series with the original detrend (top, a single first-order polynomial) and loess with span 0.5 (bottom). There is an important difference in the last years.

Answer 1

You could use smoothing with moving average(yearly,12 if the data points are monthly) as a trend estimate. Spline is also a good choice.

Answer 2

Here's an old post on electrical loads:

Predicting daily electricity load - fitting time series

The underlying long-term (> 5 years) trend is due to economic activity. As you can see in the graph labeled "US - Total Electricity Demand (Millions of Megawatt-hours)", the blue line is related to U.S. Gross Domestic Product. Keep in mind that this GDP relationship probably isn't linear. Here's a link to some GDP data (below is a graph from 1990 to now):

http://research.stlouisfed.org/fred2/graph/?id=GDP

The seasonal/repeating terms in the load data are winter, spring, summer, fall, holidays, weekdays/weekends, and time-of-day. For example, a Saturday's (weekend) demand may be lower than a nearby Thursday's demand (for the same temperature/weather), but that weekend/weekday relationship is probably much different in winter versus summer.

The bottom line is, each area has its own peculiarities and requires a lot of tweaking. If you're at a state level or below, finding a related "GDP" or economic activity data series may be difficult. From your graph, a 30MW load implies a single industrial plant or a group of small plants. Your "economic activity data" may be the production level of the plant(s).

Answer 3

Check out "Complimentary Ensemble Empirical Mode Decomposition" (CEEMD). It's rather compute-intensive, but provides a nicely adaptive approach to characterizing trends across multiple time scales. The original EMD method was published by Huang et al in Proc Royal Soc A back in 1998, but subsequently it seems Huang has been publishing his work in his own journal Advances in Adaptive Data Analysis. If you go to his research website you can find (at the bottom) links to the full text of every article published in AADA.

Here is some R code and data demonstrating running CEEMD in parallel in R.