Skip to main content
2 of 2
added more detail
EdM
  • 101.5k
  • 11
  • 102
  • 303

There might be a specific duplicate of this question on the site, but until that's identified here's a brief answer.

This approach is not statistically sound, for a couple of reasons at least. It's also not clear that it's any simpler (let alone better) than LOWESS, if all that you want is a smooth summary of historical data.

First, it uses high-degree polynomials. Forcing a single high-degree polynomial to fit over an entire range of data typically won't work well unless there is a strong theoretical reason for the polynomial. There are much better choices, like regression splines or generalized additive models, to handle nonlinear associations between a predictor variable (evidently time, here) and an outcome variable within linear regression. That's true even if you just want to describe historical trends rather than make predictions.

Second, it uses the result of trial regressions to determine the ultimate form of the regression. That leads to the multiple problems involved in automated model selection, including severe overfitting that makes results difficult or impossible to generalize beyond the data set at hand. There are ways to penalize regression coefficients to minimize overfitting, but your proposed approach didn't specify penalization. I suppose that's less of an issue if all you want to do is describe the historical data, but I suspect that your attempt at high-degree polynomial fitting might have itself led to the limited "predictive power" that you found. You might have better predictive power if you use methods like those covered in Forecasting: Principles and Practice.

EdM
  • 101.5k
  • 11
  • 102
  • 303