Incremental learning for LOESS time series model

I am currently working on some time series data, I know I can use LOESS/ARIMA model.

The data is written to a vector whose length is 1000, which is a queue, updating every 15 minutes,

Thus the old data will pop out while the new data push in the vector.

I can rerun the whole model on a scheduler, e.g. retrain the model every 15 minutes, that is, Use the whole 1000 value to train the LOESS model, However it is very inefficient, as every time only one value is insert while another 999 vlaues still same as last time.

So how can I achieve better performance?

Many thanks

• What is a "LOESS/ARIMA model"? Do you meant a nonlinear time trend with ARIMA errors where the nonlinear trend is estimated using LOESS? – Rob Hyndman Dec 20 '10 at 22:33
• sorry I mean LOESS or ARIMA model. e.g I use LOESS to find the the residuals as: residuals(loess(x ~ time)). because the data x is vector with 1000 values, which updates every 15 minutes. How can I efficiently get the residuals, but not rerun the whole datset as input everytime? as only 1 value update everytime, the other 999 values is still same as last time. – zhang Dec 21 '10 at 13:16
• one possible method, maybe use the first 1000 values to predict the next 1000 values(although LOESS only support predict 4 values), then calculate the residual as the difference between actual value and corresponding predict value.Then retrain the model every 1000 values However, this is not the original LOESS model I want at all :-( – zhang Dec 22 '10 at 16:59

Let me re-formulate this into something more familiar to me. The ARIMA is an analog to PID approximation. I is integral. MA is P. the AR can be expressed as difference equations which are the D term. LOESS is an analog to least squares fitting (high-tech big brother really).

So if I wanted to improve a second order model (PID) what could be done?

• First, I could use a Kalman Filter to update the model with a single piece of new information.
• I could also look at something called "gradient boosted trees". Using an analog of them, I would make a second ARIMA model whose inputs are both the raw inputs fed to the first, augmented with the errors of the first.
• I would consider looking at the PDF of the errors for multiple modes. If I could cluster the errors then I might want to split models, or use a Mixture model to separate the inputs into sub-models. The submodels might be better at handling the local phenomenology better than a single large-scale model.

One of the questions that I have failed to ask is "what does performance mean?". If we do not have a clearly stated measure of goodness then there is no way to tell if a candidate method "improves". It seems like you want better modeling, shorter compute time, and more efficient use of information. Having ephemeris about the actual data can also inform this. If you are modeling wind, then you can know where to look for augmenting models, or find transformations for your data that are useful.

This is a different question depending on whether you are using a loess or an ARIMA model. I will answer just the loess question for now, as I suspect there are little efficiencies possible in the ARIMA case other than perhaps having a good set of starting values.

A loess model works by fitting a weighted regression to different subsets of the data. Only a proportion of the data is used for each fit. So each time you refit the model having dropped off one data point at one end and added another at the opposite end, you technically only need to fit the local regressions that use the first and last point. All the local regressions in-between will be the same. Exactly how many of these un-impacted local regressions there are will depend on your smoothing parameter in the loess.

You could hack whatever package you are using to fit your model so that it can take most of the local regressions from a previous fit, and only fit those that are needed at the beginning and end of the data.

However, it would seem to me this was only worth doing if the cost in extra programming time was materially less than the cost in computer time of just fitting the model from scratch each 15 minutes. With only 1000 data points surely it's not such a big thing to fit the model from scratch each time.