How to Choose Polynomial Degree for Regression Model when Error Keeps Reducing As Degree Increase

Question

I have a relatively small dataset with less than 100 samples, with one predictor and one outcome variable, both numerical. I generated models using lm and glm functions.

For linear and polynomial (2 degree) models, the $R^2$ value seems to be high already (close to $0.95$). When I try higher degrees, the $R^2$ keeps increasing with RMSE, AIC and BIC decreasing.

I tried to use bootstrapping to avoid overfitting, but as the degree gets higher (such as $5$), the error still keeps reducing, I also tried to see if there is a significant difference between the RMSE/AIC/BIC/$R^2$ of models with different degrees, but it seems they all have significant differences, therefore it would seem to be better to use higher degree models. However, as this is a small dataset, I don't believe the model should be that complex.

Is there a way to set a threshold (like when exceeds, stop trying higher dimension models), or some significance test that would help here to determine when to stop generating higher degree models?

Answer 1

$R^2$ is non-decreasing with model complexity; if you keep adding higher-order polynomial terms, it can only increase (not decrease). So it is not a useful tool for model selection. On the other hand, AIC and BIC shouldn't continue to decrease (because they are penalized for the number of parameters in the model). I suggest you double-check your calculation of those measures.

The other point to keep in mind is that you can't conduct formal inference using the same dataset that you've used to select the model. E.g., if you choose the model with lowest AIC, you shouldn't then report p-values associated with the polynomial terms.