Objective: Working with time series of historical price data that is liable to have outliers and wish to apply a defined procedure to identify the outliers.

Procedure: Take time series and apply Tukey median based smoothers, so this splits data into two parts a smooth part and a residue part. Take the residue part and calculate an outer threshold (fence) that is like the Tukey Outlier rule based on IQR, but instead use the IDR (Interdecile range) so upper outlier is greater than 90 percentile + M * IDR and lower outlier is less than 10 percentile - M * IDR. I am doing this using different resolutions of smoothers and wish to apply one that is very localised so use a simple median of 3.

Problem: For localised smoother using median of 3 a lot of data points the smooth are the same as the original data and therefore the residue is very frequently exactly zero. In fact the amount of zero residues is approximately 50% of the data points. So using IQR rule it would not work well as IQR is often zero or near zero with so many residuals being zero.

Choice/Question: Given that there are a lot of zero residuals I am asking if it is valid to exclude the zero residuals and just calculate the order statistics and the subsequent outlier threshold to be just on the data that excludes the zeros.

What would be Optimal: Rules make sense to other people when explain them and if slightly different smoother is applied e.g. median of 5 then multipliers to create thresholds can be set to be similar/the same.

  • 2
    $\begingroup$ Can you explain more about why you want to exclude zeros? Can you also give some sense of what "best" means in this context? (What is it you want to optimize?) $\endgroup$ – Glen_b Jan 30 '17 at 9:39
  • $\begingroup$ The existence of zero residuals implies this is a special kind of model or special kind of data, so in addition to answering @Glen_b's questions, could you please write something about the model and the data, too? $\endgroup$ – whuber Jan 30 '17 at 16:03
  • $\begingroup$ Thanks for comments. I have changed question to make it hopefully clearer what the problem i have is and what i am trying to achieve . $\endgroup$ – James65 Jan 30 '17 at 20:22
  • 1
    $\begingroup$ This explanation provides good motivation for appreciating why Tukey did not use the median-of-three smoother (aka "3"): he combined it with other smoothers, ultimately recommending "3RSSH". That last ("Hanning") step helps remove most of those zero residuals. $\endgroup$ – whuber Jan 30 '17 at 20:45
  • 1
    $\begingroup$ @whuber when using 3RSSH is not a problem. In using a variety of smoothers at different resolutions (a bit like Mandelebrot looking at coastline at different resolutions). A median of "3R" is a very local resolution smoother, in the unpublished paper Tukey wrote in 1986 "Thinking about non-linear smoothers" on page 48 recommends a bouquet/menu of smoothers and "3R" is in the list for "light smoothing". Having lots of zeros as residuals comes with this, From a similar post "How to detect outliers when many residuals are zero" this maybe a an issue that comes up in other contexts. $\endgroup$ – James65 Jan 31 '17 at 20:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.