# How to Choose Polynomial Degree for Regression Model when Error Keeps Reducing As Degree Increase

I have a relatively small dataset with less than 100 samples, with one predictor and one outcome variable, both numerical. I generated models using lm and glm functions.

For linear and polynomial (2 degree) models, the $$R^2$$ value seems to be high already (close to $$0.95$$). When I try higher degrees, the $$R^2$$ keeps increasing with RMSE, AIC and BIC decreasing.

I tried to use bootstrapping to avoid overfitting, but as the degree gets higher (such as $$5$$), the error still keeps reducing, I also tried to see if there is a significant difference between the RMSE/AIC/BIC/$$R^2$$ of models with different degrees, but it seems they all have significant differences, therefore it would seem to be better to use higher degree models. However, as this is a small dataset, I don't believe the model should be that complex.

Is there a way to set a threshold (like when exceeds, stop trying higher dimension models), or some significance test that would help here to determine when to stop generating higher degree models?

• one option would be to use the technique this website is named after :) Commented Jun 29, 2023 at 1:40
• Higher order polynomials generally cause highly unstable predicts towards the extremes of the data. Unless you have good justification, I would stick to a linear or second order fit. Commented Jun 29, 2023 at 1:57

$$R^2$$ is non-decreasing with model complexity; if you keep adding higher-order polynomial terms, it can only increase (not decrease). So it is not a useful tool for model selection. On the other hand, AIC and BIC shouldn't continue to decrease (because they are penalized for the number of parameters in the model). I suggest you double-check your calculation of those measures.