How to correct outliers once detected for time series data forecasting?

Question

I'm trying to find a way of correcting outliers once I find/detect them in time series data. Some methods, like nnetar in R, give some errors for time series with big/large outliers. I already managed to correct the missing values, but outliers are still damaging my forecasts...

Answer 1

There is now a facility in the forecast package for R for identifying and replacying outliers. (It also handles the missing values.) As you are apparently already using the forecast package, this might be a convenient solution for you. For example:

fit <- nnetar(tsclean(x))

The tsclean() function will fit a robust trend using loess (for non-seasonal series), or robust trend and seasonal components using STL (for seasonal series). The residuals are computed and the following bounds are computed:

\begin{align} U &= q_{0.9} + 2(q_{0.9}-q_{0.1}) \\ L &= q_{0.1} - 2(q_{0.9}-q_{0.1}) \end{align} where $q_{0.1}$ and $q_{0.9}$ are the 10th and 90th percentiles of the residuals respectively.

Outliers are identified as points with residuals larger than $U$ or smaller than $L$.

For non-seasonal time series, outliers are replaced by linear interpolation. For seasonal time series, the seasonal component from the STL fit is removed and the seasonally adjusted series is linearly interpolated to replace the outliers, before re-seasonalizing the result.

Answer 2

When you identify an ARIMA model you should be simultaneously identifying Pulses/Level Shifts/Seasonal Pulses and/or Local Time Trends. You can get some reading material on Intervention Detection procedures. I recommend "Time Series Analysis: Univariate and Multivariate Methods" by David P. Reilly and William W. S. Wei.

You may have to pursue commercial software like SAS/SPSS/AUTOBOX to get any useful results as the free software I have seen is wanting. In passing, I have contributed major technical improvements in this area to AUTOBOX.

EDIT:

An even better approach is to identify the outliers using the rigorous ARIMA method plus Intervention Detection procedures leading to robust ARIMA parameters and a good forecast. Now consider developing simulated forecasts incorporating re-sampled residuals free of pulse effects. In this way, you get the best of both worlds viz a good model and more realistic uncertainty statements for the forecasts which don't assume that the estimated model parameters are the population values.

Answer 3

I agree with @Aksakal.

Instead of removing the outliers, a better approach would be to use some kind of statistical procedure to deal with the outliers.

I suggest you winsorise your data. If implemented properly, winsorisation can be relatively robust to outliers.

On this page: http://www.r-bloggers.com/winsorization/, you will find R-codes to implement winsorisation.

If you consider winsorising your data, you will need to think carefully about the tails of the distribution. Are the outliers expected to be extremely low, or are they expected to be extremely high, or maybe both. This will affect whether you winsorise at e.g. the 5% or 10% and/or the 95% or 99% level.

Answer 4

In the forecasting context, removing outliers is very dangerous. For instance, you're forecasting sales of a grocery shop. Let's say there was a gas explosion in the neighboring building, which caused you to close the shop for a few days. This was the only time the shop was closed in 10 years. So, you get the time series, detect the outlier, remove it, and forecast. You silently assumed that nothing like this will happen in the future. In a practical sense, you compressed your observed variance, and the coefficient variances shrank. So, if you show the confidence bands for your forecast they'll be narrower than they would have been if you did not remove the outlier.

Of course, you could keep the outlier, and proceed as usual, but this is not a good approach either. The reason is that this outlier will skew the coefficients.

I think a better approach, in this case, is to allow for an error distribution with fat tails, maybe a stable distribution. In this case, your outlier will not skew the coefficients too much. They'll be close to the coefficients with an outlier removed. However, the outlier will show up in the error distribution, the error variance. Essentially, you'll end up with wider forecast confidence bands.

The confidence bands convey a very important piece of information. If you are forecasting that the sales would be \$1,000,000 this month, but there's a 5% chance that they'll be $10,000, this impacts your decisions on spending, cash management, etc...

Answer 5

To perform forecasting using a model with outliers removed depends on the probability of outliers occurring in the future and the expected distribution of its effect if it indeed occurs. Is the training data sufficient for illuminating this?. A Bayesian approach should help...