5
$\begingroup$

Assume I have data originating from a model $$ y_i=f(t_i)+e(t_i) $$ with $f\in C_2(\mathbb{R})$.

The only thing I know about the errors is that they roughly happen to be of to different sources:

  • Some more or less random noise of varying magnitude over time. The noise might just oscillate between different values or may look rather random
  • Massive outliers. Those usually are correlated over a few time steps, mostly even lying on a straight line (e.g. $e(t_i)=at_i+b$ for $i=k,k+1,k+2$). In the presence of outliers, the noise usually can be ignored or is even non-existant.

As long as there are no outliers involved, I am quite successful fitting B-Splines through the data. However, this approach fails if the outliers have not been removed.

All fitting has to be done highly automated, thus manual analysis is no option.

One Idea I have for detecting the outliers is to estimate a local measure of dispersion and declare all datapoints as outliers that are much further away form a local estimate on position than that measure.

To identify my estimate on position, I have been very successful using a local median as robust estimate. A straightforward extension would be to use a local MAD as measure for dispersion.

However, now my theoretical knowledge is stretched to its limits. I do not really have the background to understand what I am doing from an analytical and mathematical point of view.

Thus, my question is twofold:

  • Do you have a better idea on how to identify outliers? (Just drop me a thought, I will gladly do the research)
  • Can you recommend literature giving me theoretical background on rubust dispresion estimation? Even better for models like the one I have sketched.

P.S.: While I had quite a few lectures on statistics at university, my background is based on numerical analysis, so please be gentle.

Improve this question
$\endgroup$
4
  • $\begingroup$ There is a whole book [outliers in statistical data amazon.com/Outliers-Statistical-Series-Probability-Statistics/… which is old, but has a lot of useful information. It should provide a starting point for research. An idea off the top of my head - since you are using splines anyway - you could see how adjusting the span of the spline affects things. A very smooth spline won't be affected by outliers unless they are very common (in which case, they aren't outliers). $\endgroup$
    – Peter Flom
    Nov 11, 2012 at 11:38
  • $\begingroup$ The problem is the strong correlation of the outliers. Of course you can make the spline as rigid as necessary to ignore outliers. However, at the same time you reduce it's ability to match features you want to preserve. I also tried iterative methods (IRLS), but they also suffer of the strong correlation. $\endgroup$
    – Thilo
    Nov 11, 2012 at 11:49
  • $\begingroup$ Ah, and thanks for the book recommendation. Looks promising at a first glance. $\endgroup$
    – Thilo
    Nov 11, 2012 at 11:50
  • $\begingroup$ Have you already looked at robust GAM fitting procedures? $\endgroup$
    – user603
    Nov 11, 2012 at 17:29

1 Answer 1

Reset to default
1
$\begingroup$

In order to identify outliers, one needs to have an equation that adequately describes the signal. When faced with analyzing time series data fitting splines initially seems interesting but ultimately fails. The problem is that one has to determine the number of splines ( pure deterministic structure) and the length of each spline.This approach essentially is a fitting approach rather than a modelling (data-based) approach. The nature of the signal and the nature of the errors needs to be identified. Possible models might possibly include both a deterministic structure for the signal and/or autoregressive structure for the signal. Deterministic structure can include level shifts, local time trends (splines), seasonal(periodic) pulses and one-time pulses and sometimes in "physical/biological time series" sines,cosines and such but never in powers of time like quadratics/cubics etc.. Autoregressive structure can include the explicit incorporation of previous values. Errors from this "model" can often be characterized with an ARIMA structure thus further enhancing the signal. The final model should render a set of residuals that have a constant mean and variance.

Your characterization implies that there might be identifiable "patches of residuals" , possibly having a "trend". I would suggest that you post your, perhaps scaled to hide the real data, and have some of the readers of your post to actually render a workable solution. You can then systematize their solution and incorporate it into your application.

You might want to review a similar thread as it deals with analysing time series data. How do I calculate projected figures for the next year based on past performance?

Improve this answer
$\endgroup$
4
  • $\begingroup$ This is not really on topic, but can you explain how exactly you discriminate btw fitting and modeling? $\endgroup$
    – bayerj
    Nov 11, 2012 at 20:10
  • $\begingroup$ Thanks for your input. Honestly, I do not really know where to start. For once, the pitfalls of splines you describe are pretty much gone when dealing with smoothing splines - just add more degrees of freedom than you need and the smoothing part of the spline cares for the rest (of course, you have to provide a smoothing parameter first, but there are methods to overcome this as well). But apart from that, the reason I use splines is their compact description and efficient possibility to formulate further algorithms on that data. $\endgroup$
    – Thilo
    Nov 11, 2012 at 20:35
  • $\begingroup$ Probably, I really try to fit the data to a given model then to create the model itself - but that again is required by the application. So, there are a bunch of reasons it is no option to go for another model. Furthermore, there is no way to assume a constant variance of the residuals - modeling that variance shifts away would certainly overfit my data. However, I will try to add more information and sample data tomorrow, as this might help to understand the problem better. $\endgroup$
    – Thilo
    Nov 11, 2012 at 20:41
  • $\begingroup$ Model Identification, Estimation and re-modelling via diagnostic tests for sufficiency and necessity characterizes modelling as compared to assuming structure(like a smoothing parameter)..This is easily done via a F Test comparing local variances.Tsay, R.S. (1986). "Time Series Model Specification in the Presence of Outliers," Journal of the American Statistical Society, Vol. 81, pp. 132-141. or "Outliers, Level Shifts, and Variance Changes in Time Series." Ruey S. Tsay; Journal of Forecasting, 1988, 7(1), pp. 1. $\endgroup$
    – IrishStat
    Nov 11, 2012 at 22:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.