Detecting outliers in a time-series

Question

I'm trying to exclude the outliers using 2-sigma rule and I have a time series. So I use a moving average for this.

Let's say I have this:

W1 38 315
W2 48 002
W3 47 487
W4 50 977
W5 39 604
W6 46 058
W7 45 718
W8 22 408

and I want to exclude outliers. I calculate MVA for 8 values and moving stan.dev.

The first question is when I calculate MVA for week8. Should I include it or not? The second question is when I detect W4 as an outlier, should I delete it from range and detect another outliers with recalculated MVA and st.dev. (without 50 977 value) or not?

Answer 1

I'm not sure what your first question is asking. Also, I'm not sure what the values in that data frame represent (column headers would help).

As for your second question, recursively detecting and removing outliers is dangerous, so I would say the answer is "no".

Answer 2

Outliers should never be excluded. They are a part of your data set for a reason—namely something unusual happened. You may want to use the data to detect if a transcription error occured. If that is the case, the error is corrected and the data re-examined. By ignoring data when things are not what is expected or undesireable, you miss the whole point of examining the data.

An "outlier" is a point which is 1.5 times the Inner Quartile Range (IQR) from the First Quartile to the Minimum value or from the Third Quartile to the Maximum.

If you are trying to detect which points would not be caused by common-cause variation, then you should use an $I-MR$ or an $x-MR$ chart, which plots $\bar{x}\pm3\hat{s}$ as well as a moving range with its control limits. The limits on the individual $x$ chart are based on the average moving range, $\overline{MR}$ and are readily accessible. The linked page also includes the WECO rules for determining if a process is out of statistical control on a Shewhart chart.

A greater discussion and resource can be found on the NIST website.

If these methods for identifying special cause variation have helped you to identify and eliminate the special causes, then you can start from the improvement going forward to calculate new parameters and continue to monitor for special causes. You should never go back in time and eliminate the data of a problem—especially if it is a problem you never solved: it is all a part of the variation in the process that needs to be dealt with.

Answer 3

Outliers should always be identified and dealt with by including either a 0/1 dummy predictor variable (a form of data cleansing) or by introducing an additional (perhaps newly identified !) causal variable explaining the anomaly yielding an improved equation. The whole idea of modeling is to characterize "normal observations" and not be blind-sided by the anomalous one(s) leading to a robust model. Tests of significance and sufficiency are then conducted correctly. Not adjusting ( a flawed approach in my opinion ) causes the error variance to be over-estimated yielding both false rejections of hypothesis tests and more importantly a false "Alice in Wonderland" conclusion about the sufficiency of the model.

After creating a reasonable model, confidence limits should then be developed by re-sampling the reverse-adjusted error process providing possibly fat tails and asymmetric limits. The reason for this is that the empirically identified outliers enable robust parameter estimation but can severely under-estimate the going-forward uncertainty (error variance) around predictions.