I'm trying to find a way of correcting outliers once I find/detect them in time series data. Some methods, like nnetar in R, give some errors for time series with big/large outliers. I already managed to correct the missing values, but outliers are still damaging my forecasts...

  • what do you mean by "correcting"? – rbatt Sep 12 '13 at 17:36
  • Substituting or replacing... Or should it be just subtracting...? – Mila Sep 13 '13 at 15:51
  • 3
    Are the outliers "real"? By this I mean are outliers caused by the inability of the model to predict extreme events or shocks? Or are they "errors" such as data entry problems? – probabilityislogic Mar 18 '14 at 8:59

There is now a facility in the forecast package for R for identifying and replacying outliers. (It also handles the missing values.) As you are apparently already using the forecast package, this might be a convenient solution for you. For example:

fit <- nnetar(tsclean(x))

The tsclean() function will fit a robust trend using loess (for non-seasonal series), or robust trend and seasonal components using STL (for seasonal series). The residuals are computed and the following bounds are computed:

\begin{align} U &= q_{0.9} + 2(q_{0.9}-q_{0.1}) \\ L &= q_{0.1} - 2(q_{0.9}-q_{0.1}) \end{align} where $q_{0.1}$ and $q_{0.9}$ are the 10th and 90th percentiles of the residuals respectively.

Outliers are identified as points with residuals larger than $U$ or smaller than $L$.

For non-seasonal time series, outliers are replaced by linear interpolation. For seasonal time series, the seasonal component from the STL fit is removed and the seasonally adjusted series is linearly interpolated to replace the outliers, before re-seasonalizing the result.

When you identify an ARIMA model you should be simultaneously identifying Pulses/Level Shifts/Seasonal Pulses and/or Local Time Trends. You can get some reading material on Intervention Detection procedures at http://www.ebay.com/ctg/Time-Series-Analysis-Univariate-and-Multivariate-Methods-David-P-Reilly-and-William-Wei-1999-/85697 and http://www.unc.edu/~jbhill/tsay.pdf . You may have to pursue commercial software like SAS/SPSS/AUTOBOX to get any useful results as the free software I have seen is wanting. In passing I have contributed major technical improvements in this area to AUTOBOX.

EDIT:

An even better approach is to identify the outliers using the rigorous ARIMA method plus Intervention Detection procedures leading to robust ARIMA parameters and a good forecast. Now consider developing simulated forecasts incorporating re-sampled residuals free of pulse effects. In this way you get the best of both worlds viz a good model and more realistic uncertainty statements for the forecasts which don't assume that the estimated model parameters are the population values.

  • I will check the material, thanks a lot! – Mila Sep 15 '13 at 11:46

I agree with @Aksakal. Instead of removing the outliers, a better approach would be to use some kind of statistical procedure to deal with the outliers. I suggest you winsorise your data. If implemented properly, winsorisation can be relatively robust to outliers. On this page: http://www.r-bloggers.com/winsorization/, you will find R-codes to implement winsorisation. If you consider winsorising your data, you will need to think carefully about the tails of the distribution. Are the outliers expected to be extremely low, or are they expected to be extremely high, or maybe both. This will affect whether you winsorise at e.g. the 5% or 10% and/or 95% or 99% level.

  • 1
    Winsorization, for the purpose of estimating an overall mean, might make some sense in the absence of any trend or seasonal components in the data, but otherwise threatens to destroy important information. – whuber Jul 26 '15 at 15:57

In the forecasting context, removing outliers is very dangerous. For instance, you're forecasting sales of a grocery shop. Let's say there was a gas explosion in the neighboring building, which caused you to close the shop for a few days. This was the only time the shop was closed in 10 years. So, you get the time series, detect the outlier, remove it and forecast. You silently assumed that nothing like this will happen in future. In practical sense, you compressed your observed variance, and the coefficient variances shrank. So, if you show the confidence bands for your forecast they'll be narrower than they would have been if you did not remove the outlier.

Of course, you could keep the outlier, and proceed as usual, but this is not a good approach either. The reason is that this outlier will skew the coefficients.

I think a better approach in this case is allow for an error distribution with fat tails, maybe a stable distribution. In this case your outlier will not skew the coefficients too much. They'll be close to the coefficients with an outlier removed. However, the outlier will show up in the error distribution, the error variance. Essentially, you'll end up with wider forecast confidence bands.

The confidence bands convey a very important piece of information. If you are forecasting that the sales would be 1,000,000\$ this month, but there's 5% chance that they'll be$10,000, this impacts your decisions on spending, cash management etc.

To perform forecasting using (or not) a model with outliers removed depends on the probability of outliers occurring in the future and the expected distribution of its effect, if it indeed occurs. Is the trainning data sufficient for iluminating this?. A Bayesian approach should help...

  • 1
    This is incorrect. your forecast may be inaccurate if you don't adjust for outliers, especially if your outliers are at the end of the series. – forecaster Sep 5 '15 at 14:45

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.