# When forecasting, is it better to remove the outliers or just to transform them?

I am forecasting the number of logins. I have a dataset with the number of logins for each hour. First, I use LOF (local outlier factor) to find the outliers and then I remove them. Second, I use Kalman filter to forecast.

But as you can see in the plot, the prediction (blue) is displaced in time from the logins (red) and also from the future confidence interval (green).

So, would it be better to transform the outliers (maybe make them "normal") rather than completely removing them? Because right now I have some gaps in time. For example (the first monday), I have logins , for each hour, from 00h to 15h and the from 18h until the next day. So there is a gap of 2 hours.

Thank you

If the data are wrong, remove them regardless of whether or not they are outliers. If a data point is right but it is an outlier and you remove it, your analysis will be biased. This might even be the reason why your prediction desynchronizes with the trend. Transforming outliers is just a less extreme way of deleting them that is also prone to bias.

The key point of the Kalman Filter is that it should be able to use the response history to inform the system state, but it doesn't seem that you are doing that. That means that at time $$t$$ you can use all of $$t-1$$ to generate predictions for $$t$$ and onward. Later, at time $$t+k$$ you can use all of $$t+k-1$$ to generate predictions for $$t+k$$ and onward.

So at the dotted line, are you showing your system forecasts for that time onward unconditional of the later response, and then superimposing the observed system state afterward? In that case, it's not surprising to see the forecast lose fidelity, that's the inevitable outcome: you just have to be clear about the target range for forecasts... nobody can predict the entire future in a statistical model.

The weights for the lagged response inputs are usually quite high, the best prediction of a system state is often the state it has most recently been in. When you run a forecast way out in the future, it is prone to desynchronizing. One way you can enforce a strictly seasonal adherence is to use the actual season as covariate inputs.

• Can you talk a bit more about the 3rd paragraph? What I have after the green dotted line is my test set (inside the orange envelope. Around 12 days) and the forecast (green envelope. Around 1.5 days). As there is a displacement between logins and prediction; the metrics (RMSE and R2) give bad results) – Aizzaac Jun 20 at 16:14
• @Aizzaac I think you've confirmed my suspicion. You should also make darn sure you haven't mapped the wrong time... common foible for R users who forget to supply the "time" as the "X" when superimposing curves. Your prediction is desynchronized with the trend... so what? It provides locally good prediction but bad forecasts to the point that 2 weeks out, the trend is offset a half a period. Maybe that isn't so bad, if you can monitor trend and react daily, you only need forecasts going out one day. It's a matter of refining your scientific question. – AdamO Jun 20 at 16:34

I would replace packets of contiguous missing values with hourly averages around the missing values. If the values are not missing but are anomalous either manually adjust them or estimate what they should have been via Intervention Detection which is essentially a forward prediction/fitted value for an anomaly.

Outliers represent effects/variables that are omitted from your model and if possible need to be identified and accounted for by adding additional/logical predictor series or worst case dummy indicators.