0
$\begingroup$

For fun, I was trying to make a predictor for how long it would take for George R. R. Martin's The Winds of Winter to be released. My "best" model is the one that had the lowest combined AIC and BIC score (summed together). I tried polynomials of degree 0 to something like 50. The best one of this sort was this, which had a degree of 3 or 4:

enter image description here

Where the y-axis is the days since A Game of Thrones was released and the x-axis is the number of books released in the A Song of Ice and Fire series. Despite this having the best combined AIC and BIC score, it is a poor predictor, predicting in a way that doesn't make any temporal sense. However, this had the best combined AIC and BIC score (the lowest one). This pointed out to me that my notion that "the best predictor is the one with the lowest combined AIC and BIC score" is flawed. Where have I gone wrong in my thinking, and what kind of scoring criteria would be more appropriate?

$\endgroup$
3
  • $\begingroup$ In order to define the likelihood for the AIC you had to assign uncertainties to the Y values. Which values did you use? If those were too small, your criterion will naturally prefer the better fit to the data points $\endgroup$
    – J. Delaney
    Feb 20, 2022 at 20:20
  • $\begingroup$ I'm not aware of this, and have assigned no such uncertainties to my Y values. I just used statsmodels's AIC and BIC score as they're given as methods once the model is fit. There ought not to be any uncertainty with the y values, as the past release dates are certain. $\endgroup$
    – sangstar
    Feb 20, 2022 at 22:54
  • $\begingroup$ Then what you are doing is quite meaningless. the library you are using probably has some default value (maybe 1) that in your case is very small compared to the scale of the y data. Try for example to scale y by 1/1000 and see what happens. (Note that the uncertainty does not affect the fit result, but it does affect the likelihood, so you can't ignore it if you want to use AIC) $\endgroup$
    – J. Delaney
    Feb 21, 2022 at 0:04

1 Answer 1

1
$\begingroup$

Unpenalized polynomial fits to data are notoriously troublesome, particularly near or beyond the limits of the data. For example, does the shape of your relationship at high x-axis values make sense? See this page, for discussion of some better approaches.

Also, it seems that you probably would want to model this in the opposite direction, with days since "Games of Thrones" as the predictor on the x axis and the number of books in the series as the outcome on the y axis. The same principle of avoiding unpenalized polynomial fits would still apply.

$\endgroup$
2
  • $\begingroup$ So, essentially polynomial fits higher than 3 or 4 unregularized tend to suffer from troublesome behavior at the limits? And regularized cubic splines may be a more robust alternative? $\endgroup$
    – sangstar
    Feb 20, 2022 at 22:57
  • $\begingroup$ @sangstar even such low degrees of a polynomial can get you into trouble. Unless there’s a theoretical reason for a particular polynomial, penalization or natural (restricted cubic) splines are safest. $\endgroup$
    – EdM
    Feb 21, 2022 at 0:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.