5
$\begingroup$

I'm building a ridge regression model in scikit-learn and trying to find the optimal degree polynomial to use. The data I'm working with is a fairly predictable time series of hourly traffic volumes, and I'm predicting said volumes from the date, hour, and day of the week. R-squared values increase for both my train and test sets as I generate higher degree polynomial features, but suddenly drop from .91 to -1.4 when I go from degree 8 to 9, signifying that the 9th-order model is worse than the 0-order model.

Any idea why this happens?

Improve this question
$\endgroup$
5
  • 8
    $\begingroup$ Quick guess: I don't use these packages, but there can be numerical issues with polynomial curve fitting. As $n$ gets bigger, $x^n$ can get absurdly large and you'll may get an ill-conditioned design matrix. Something to try (if the package doesn't do it for you... which it may) is to center the data at zero first (i.e. subtract off the mean of $x$) and do your curve fitting on that. $\endgroup$
    – Matthew Gunn
    Commented Jul 27, 2016 at 15:56
  • 1
    $\begingroup$ That's a nice idea. Yeah I've noticed a lot of the coefficients the high order model generates are miniscule (1e-8 or smaller). So I might have found the breaking point for the optimization function, maybe the learning rate is too large, or it can't handle all the dimensions. $\endgroup$
    – Kevin Zhang
    Commented Jul 27, 2016 at 16:33
  • 3
    $\begingroup$ Regarding the possibility that the high-order polynomial is ill-conditioned, I wrote this answer However, I think it's more likely that a high-order polynomial is ill-advised because, well, it's generally a bad model. Especially for time-series data, there are better ways to account for time-varying trends like this, such as time-series analysis. $\endgroup$
    – Sycorax ♦
    Commented Jul 28, 2016 at 3:08
  • 1
    $\begingroup$ @GeneralAbrial On first glance, which time series analysis tool would you look into? I read a bit into exponential smoothing and that seems to fit the bill. $\endgroup$
    – Kevin Zhang
    Commented Jul 28, 2016 at 15:48
  • 1
    $\begingroup$ @KevinZhang The data you have sounds like it has trends that occur within days, within weeks, and within years. Gelman tackles a data set similar to this in Bayesian Data Analysis 3rd ed using Gaussian process regressions. But I'm not an expert on time-series analysis. It's a vast field. $\endgroup$
    – Sycorax ♦
    Commented Jul 28, 2016 at 17:10

3 Answers 3

Reset to default
3
$\begingroup$

Fitting polynomials to time series data (without prior theory) is in my opinion never a good idea and is definitely anachronistic (i.e. an outdated approach). See Does the p-value in the incremental F-test determine how many trials I expect to get correct? for @w.huber's wise reflection. Perhaps you have latent structure reflecting level shifts/multiple time trends or memory structure using lags of one or more series or even anomalous data points (pulses) or even non-constant error variance. Remedying these issues can often lead to reasonable models as @Sycorax and https://stats.stackexchange.com/users/11887/kjetil-b-halvorsen pointed out.

Improve this answer
$\endgroup$
1
$\begingroup$

I think this is happening due to overfitting. A large polynomial order model, sometimes tends to perform worse than the original model.

This is due to the fact that the large order model doesn't generalise well on data that it hasn't seen.

An overfit model always reduces the training error. It is natural for the training error to drop as you keep increasing the degree of your model. Infact, at some point, your training error will reach 0.

This is the relation between model complexity and training and test errors

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ Hi! Welcome to Cross Validated! Could I push you to elaborate a little bit more in your answer? I think you might be on to something, though it does seem odd for the training error to drop if the issue is overfitting. $\endgroup$
    – Matt Krause
    Commented Jul 28, 2016 at 3:45
  • 1
    $\begingroup$ My train and test set errors are both huge when I increase the order polynomial, so this doesn't seem to be about overfitting. $\endgroup$
    – Kevin Zhang
    Commented Jul 28, 2016 at 14:10
  • $\begingroup$ @KevinZhang Can you give me a link to the dataset? $\endgroup$
    – Near
    Commented Jul 28, 2016 at 15:47
  • $\begingroup$ Sorry I can't, it's proprietary data $\endgroup$
    – Kevin Zhang
    Commented Jul 28, 2016 at 15:49
  • 3
    $\begingroup$ High-order polynomial very seldom are good models, polynomials are way to stiff. Try splines! search this site for splines. $\endgroup$
    – kjetil b halvorsen ♦
    Commented Jun 8, 2017 at 9:35
0
$\begingroup$

The overfitting could be the reason if you are referring to the drop in accuracy on the testing or validation data. However, even the training accuracy will drop after increasing the polynomial degree beyond a certain threshold. That's why we always need to use the simplest model that fits the data, and avoid "suspicious coincidences" as is referred to in the domain of concept learning. This is called the Bayesian Occam's razor principle, which I have answered similar question about here.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.