In a question I asked recently, I was told that it was a big "no-no" to extrapolate with loess. But, in Nate Silver's most recent article on FiveThirtyEight.com he discussed using loess for making election predictions.

He was discussing the specifics of aggressive versus conservative forecasts with loess but I am curious as to the validity of making future predictions with loess?

I am also interested in this discussion and what other alternatives there are that might have similar benefits to loess.

  • $\begingroup$ If your x-variable is time, it would be dangerous to use loess to predict into the future (which would be outside the range of the data). But that doesn't mean you can't use loess to make predictions more generally. $\endgroup$ – Glen_b -Reinstate Monica Jul 28 '16 at 17:46
  • $\begingroup$ @Glen_b out of curiosity what would something I could "more generally" predict? $\endgroup$ – a.powell Jul 28 '16 at 17:50
  • 7
    $\begingroup$ Imagine a nonlinear relationship between the the proportion of people inclined to vote for party A and the unemployment rate (along with other predictors -- effects for the individual states for example). Further imagine there's new unemployment figures just become available; within the range of values experienced in the training set, but not necessarily a value represented in that set (e.g. past unemployment is between 5 and 12% and we now have a figure of 8.3%, forecast to be steady). Then we could use loess to predict the proportion voting A, without going outside 5-12% unemployment. $\endgroup$ – Glen_b -Reinstate Monica Jul 28 '16 at 18:07
  • 1
    $\begingroup$ @Glen_b Thank you. That is a wonderful illustration of how it can be used for forecasts. $\endgroup$ – a.powell Jul 28 '16 at 18:18

The problem with lowess or loess is that it uses a polynomial interpolation. It is well known in forecasting that polynomials have erratic behavior in the tails. When interpolating, piecewise 3rd degree polynomials provide excellent and flexible modeling of trends whereas extrapolating beyond the range of observed data, they explode. Had you observed later data in the time series, you definitely would have needed to include another breakpoint in the splines to obtain good fit.

Forecasting models, though, are well explored elsewhere in the literature. Filtering process like the Kalman filter and the particle filter provide excellent forecasts. Basically, a good forecast model will be anything based on Markov chains where time is not treated as a parameter in the model, but previous model state(s) are used to inform forecasts.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.