# Time Series detrending with multiple polynomials

I'm currently working on energy demand forecasting using daily load data. I'm using data since 1990 and, in order to use ARIMAX models, I detrended the data (using a first-order polynomial) and then I removed seasonality. As we can see on this image ,

in the last two years there is (unfortunately) a different trend with respect the previous 15 years. I was thinking to use two different first-order polynomials to perform detrend: one fitted on the data 1990-2008 and another one for 2008-2010. What is the best way to do this? I'm not sure if I must use a spline or not. Can you suggest me the best-practice method to perform this kind of detrending?

Thank you, P.S. I'm a computer scientist and I'm using R (and MATLAB).

UPDATE

This is the loess smothing with span 0.9

With span 0.5

and finally with span 0.1

I will try to perform detrending with span 0.5.

** UPDATE 2 ** This is the comparison with the time-series with the original detrend (top, a single first-order polynomial) and loess with span 0.5 (bottom). There is an important difference in the last years.

-
I agree that 0.5 looks smooth and yet shows a reasonable trend. It feels un-statistical to talk about "looks", but I don't know enough to go beyond that. What does the detrended data (i.e. residual after subtracting your loess trend) look like? –  Wayne Dec 30 '11 at 18:29
I compared the detrended data obtained both with loess and with a first-order polynomial. The biggest difference shows obviously in the final part, and in fact all the analysis about energy demand is more consistent with our initial hypothesis. I will post a couple of additional figures later. –  Matteo De Felice Jan 3 '12 at 8:15

You could use smoothing with moving average(yearly,12 if the data points are monthly) as a trend estimate. Spline is also a good choice.

-
Agreed. If you want to try a spline, perhaps loess with span=0.9 or something like that would be worth it. I really don't think the two-polynomial approach would work: among other things it doesn't look (eyeballing it) like 2008 is the turning point. –  Wayne Dec 29 '11 at 19:10

Here's an old post on electrical loads:

Predicting daily electricity load - fitting time series

The underlying long-term (> 5 years) trend is due to economic activity. As you can see in the graph labeled "US - Total Electricity Demand (Millions of Megawatt-hours)", the blue line is related to U.S. Gross Domestic Product. Keep in mind that this GDP relationship probably isn't linear. Here's a link to some GDP data (below is a graph from 1990 to now):

http://research.stlouisfed.org/fred2/graph/?id=GDP

The seasonal/repeating terms in the load data are winter, spring, summer, fall, holidays, weekdays/weekends, and time-of-day. For example, a Saturday's (weekend) demand may be lower than a nearby Thursday's demand (for the same temperature/weather), but that weekend/weekday relationship is probably much different in winter versus summer.

The bottom line is, each area has its own peculiarities and requires a lot of tweaking. If you're at a state level or below, finding a related "GDP" or economic activity data series may be difficult. From your graph, a 30MW load implies a single industrial plant or a group of small plants. Your "economic activity data" may be the production level of the plant(s).

-

Check out "Complimentary Ensemble Empirical Mode Decomposition" (CEEMD). It's rather compute-intensive, but provides a nicely adaptive approach to characterizing trends across multiple time scales. The original EMD method was published by Huang et al in Proc Royal Soc A back in 1998, but subsequently it seems Huang has been publishing his work in his own journal Advances in Adaptive Data Analysis. If you go to his research website you can find (at the bottom) links to the full text of every article published in AADA.

Here is some R code and data demonstrating running CEEMD in parallel in R.

-
Any chance of matlab implementation of CEEMD? –  user18353 Jan 5 '13 at 0:05
Nope, sorry; I'm pretty strongly opposed to the use of non-FOSS scientific computing. –  Mike Lawrence Feb 14 '13 at 1:15