How to deal and interpret local minima in a [time series] cross-validation error plot? I am working with some time series data and I developed a linear regression model to make future predictions. The model has the following form: 
$$
y\left( t \right) =\sum _{ i=1 }^{ M }{ { \alpha  }_{ i }{ f }_{ i }\left( x\left( t \right)  \right)  } 
$$
where ${ f }_{ i }\left( x\left( t \right)  \right) $ is a nonlinear function and ${ \alpha  }_{ i }$ are the parameters learned during training.
The data has 120 observations from  which 108 are used for training. In order to determine the appropriate number of terms $M$, a time series cross-validation was used on the 12 remaining observations, as described by Prof. Hyndman on his website. I applied a slightly modified version so I describe it below:


*

*Start with one model term.

*Estimate the parameter coefficients of the model using the first 108 observations, forecast the remaining 12 and compute the errors.

*Move the first observation of the test set to the training set and repeat step 2, i.e. train on 109 observations, forecast on the remaining 11 observations, and compute the errors.

*Step 3 is repeated for the remaining observations in the test set, i.e. until train on 119 observations, forecast the last observation available, and compute the last error.

*All the errors are combined into a Mean Absolute Error value. This represents the CV error for the model.

*Add a new model term, and repeat steps 2-5.

*The procedure is repeated until a specified number of terms is reached.


Such technique produces the following plot:

From the plot, the final model selected has 47 terms because this produced the lowest cross-validation error. However, there are several local minima, the most notorious at 2, 17 and 32 terms. 
My questions are:


*

*Is it common for a cross-validation plot to have several minima? Shouldn't it be smoother with a single optimum?

*Why use all 47 terms when it seems that the inclusion of some of them increased the cross-validation error?

*Why not use the first or the second local minimum as the best number of terms?


Any help is greatly appreciated!
 A: *

*Is it common for a cross-validation plot to have several minima? Shouldn't it be smoother with a single optimum?


Yes, this is very common. Adding an additional term improves accuracy proportional to how much information it adds, and decreases accuracy because the model has become more complex and harder to fit. Plus, there is noise due to the fact that your training set is finite. So you shouldn't expect this to be super smooth. I often smooth CV plots before trying to interpret them or find minima.


*Why use all 47 terms when it seems that the inclusion of some of them increased the cross-validation error?


It's not clear from the plot if any of the terms actually increased the CV error in a significant way, since there is some level of statistical noise. But you're right, maybe we shouldn't use all 47 of the terms - maybe it would be better to skip some of the functions in the middle. Selecting the exact set of features that gives the best performance is NP-hard, but you could try a greedy approach. First try each function by itself, and pick the one that gives the lowest error. Then try adding in every remaining function one at a time, and pick the one that decreases the error the most. Repeat, and stop once none of the remaining functions decreases the error. This isn't guaranteed to work (for example, maybe some functions are only useful as a group and not individually) but could give you a better set than just taking the functions in order.


*Why not use the first or the second local minimum as the best number of terms?


These have worse CV error than 47, so they are unlikely to be better models. However they might be informative - the functions on the downward slopes of these minima might be the best features to use, while the ones on the upswing are just adding complexity and hurting you. You could try taking the set of all functions of the 70 that decreased CV error when they were added, which could be better than your 47.
