6
$\begingroup$

I have various sets of irregular interval time series data to which I want to apply some sort of smoothing algorithm to produce a good fit.

I have attempted various methods which all were unsatisfactory.

  1. Loess - Too much of a tendency to overshoot/overreact to outliers
  2. Moving Average - The lag is unacceptable

Example Dataset:

http://i.stack.imgur.com/sFkTe.png

I have read about the "Improved Holt Method for Irregular Time Series", but the paper was too difficult for me to understand and implement in C#.

Can someone point me to a good method / algorithm which produces good smoothing?

The method must be able to calculate the smoothed point at time $t$, without requiring $t+1$, etc., data. It also must be capable of dealing with multiple $y$ values for a given $x$ time.

$\endgroup$
  • 1
    $\begingroup$ With loess, you can adjust the amount of smoothing, did you try different amounts? $\endgroup$ – Peter Flom Apr 20 '13 at 14:12
  • $\begingroup$ yes, but the overshooting problem persists if a sudden dip or change in trend occurs. Too large a smoothing factor or window creates lag similar to MA. $\endgroup$ – Ying Apr 20 '13 at 14:16
  • $\begingroup$ Seems like what you want is to use any typical smoothing but just disable it for large steps. If you do not do that it looks like you will always a get lag. $\endgroup$ – Bitwise Apr 20 '13 at 15:04
5
$\begingroup$

The simplest algorithm is the median filter. You can find an C++ implementation in the R package robfilter. That implementation also include an 'online' version that only uses past data and implements some algorithmic short-cuts.

Of course you will still have to set the "width" argument yourself, but this is the counter part of looking for a simple algorithm (this package also contains more sophisticated smoothing algorithms).

The median-filter is essentially a rolling window median, so it inherits the good behaviour of the median in terms of insensitivity to outliers and non-parametric interpret-ability.

So, considering the dataset you posted, the median filter would yield:

median filter

and the code:

a1<-read.table("sodat.txt",header=TRUE)
library("robfilter")
d1<-med.filter(a1[,2],width=10,online=TRUE)
plot(d1)
$\endgroup$
1
$\begingroup$

I think what you want is a median filter: "Median filtering is very widely used in digital image processing because, under certain conditions, it preserves edges while removing noise".

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.