I have read some answers saying to make the maximum degree equals n-1 for testing. However, I wonder if it is also the case for a polynomial trend model with season variables included. I am using monthly data to analyse and if so then the max degree will be thousands. Does this suffer overfitting issue? If n-1 is wrong, which should I use for the maximum?
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Your proposal does indeed suffer from overfitting. You will be simultaneously fitting month to month variation and year to year variation but it sounds like you want to focus in on month to month variation.
The maximum degree should be the maximum number of differences you want to observe in the data. The N-1 rule works for a single repetition of an experiment but does represent the hard upper boundary, not the generally desirable one as it will fit a new curvature between every point and not actually reflect a 'trend'. When you fit a polynomial you generally do so to find a simpler underlying description of reality, and the lower N is the the simpler that description.
Since you are interested in point to point trend over repeating periods, i.e. month to month on yearly cycles. This means you have 12 time periods you want to trend over, with X years of repetitions.
You n-1 would then be 11 as you have 12 calendar months with X repetitions
$\begingroup$
$\endgroup$
You definitely don't want to be fitting polynomials to time series data . See Why is my high degree polynomial regression model suddenly unfit for the data? for a discussion of this.
You might also want to read Does the p-value in the incremental F-test determine how many trials I expect to get correct? and follow huber's remarks on fitting polynomials.
