Assume I have data originating from a model $$ y_i=f(t_i)+e(t_i) $$ with $f\in C_2(\mathbb{R})$.
The only thing I know about the errors is that they roughly happen to be of to different sources:
- Some more or less random noise of varying magnitude over time. The noise might just oscillate between different values or may look rather random
- Massive outliers. Those usually are correlated over a few time steps, mostly even lying on a straight line (e.g. $e(t_i)=at_i+b$ for $i=k,k+1,k+2$). In the presence of outliers, the noise usually can be ignored or is even non-existant.
As long as there are no outliers involved, I am quite successful fitting B-Splines through the data. However, this approach fails if the outliers have not been removed.
All fitting has to be done highly automated, thus manual analysis is no option.
One Idea I have for detecting the outliers is to estimate a local measure of dispersion and declare all datapoints as outliers that are much further away form a local estimate on position than that measure.
To identify my estimate on position, I have been very successful using a local median as robust estimate. A straightforward extension would be to use a local MAD as measure for dispersion.
However, now my theoretical knowledge is stretched to its limits. I do not really have the background to understand what I am doing from an analytical and mathematical point of view.
Thus, my question is twofold:
- Do you have a better idea on how to identify outliers? (Just drop me a thought, I will gladly do the research)
- Can you recommend literature giving me theoretical background on rubust dispresion estimation? Even better for models like the one I have sketched.
P.S.: While I had quite a few lectures on statistics at university, my background is based on numerical analysis, so please be gentle.