I am working with some time series data and I developed a linear regression model to make future predictions. The model has the following form:

$$ y\left( t \right) =\sum _{ i=1 }^{ M }{ { \alpha }_{ i }{ f }_{ i }\left( x\left( t \right) \right) } $$

where ${ f }_{ i }\left( x\left( t \right) \right) $ is a nonlinear function and ${ \alpha }_{ i }$ are the parameters learned during training.

The data has 120 observations from which 108 are used for training. In order to determine the appropriate number of terms $M$, a time series cross-validation was used on the 12 remaining observations, as described by Prof. Hyndman on his website. I applied a slightly modified version so I describe it below:

  1. Start with one model term.
  2. Estimate the parameter coefficients of the model using the first 108 observations, forecast the remaining 12 and compute the errors.
  3. Move the first observation of the test set to the training set and repeat step 2, i.e. train on 109 observations, forecast on the remaining 11 observations, and compute the errors.
  4. Step 3 is repeated for the remaining observations in the test set, i.e. until train on 119 observations, forecast the last observation available, and compute the last error.
  5. All the errors are combined into a Mean Absolute Error value. This represents the CV error for the model.
  6. Add a new model term, and repeat steps 2-5.
  7. The procedure is repeated until a specified number of terms is reached.

Such technique produces the following plot:

Cross-validation error plot

From the plot, the final model selected has 47 terms because this produced the lowest cross-validation error. However, there are several local minima, the most notorious at 2, 17 and 32 terms.

My questions are:

  1. Is it common for a cross-validation plot to have several minima? Shouldn't it be smoother with a single optimum?
  2. Why use all 47 terms when it seems that the inclusion of some of them increased the cross-validation error?
  3. Why not use the first or the second local minimum as the best number of terms?

Any help is greatly appreciated!

  • $\begingroup$ "cross-validation was used on the 12 remaining observations" -- this is not clear, did you use 10-fold cross-validation (ten times leaving out 12 observations our of 120) or simply computed the error on the fixed test set consisting of 12 observations? $\endgroup$
    – amoeba
    Dec 15, 2015 at 16:21
  • $\begingroup$ @amoeba I computed the error on the fixed test set of 12 observations, but using the approach described here $\endgroup$ Dec 15, 2015 at 16:25
  • $\begingroup$ Then do 10-fold cross-validation instead. This will essentially give you 10 error curves instead of only 1 and you can average them. This will decrease the noise and make it smoother. $\endgroup$
    – amoeba
    Dec 15, 2015 at 16:27
  • $\begingroup$ @amoeba I considered using a k-fold CV, but it seems it is not the best approach to do given that this is a time-series problem. That is the reason why I used the method I shared in the link. It seems that this approach is usually used in order to save the dependencies within the data. This was briefly discussed on another post here which basically refers to the original source by Hyndman. $\endgroup$ Dec 15, 2015 at 16:49
  • 2
    $\begingroup$ I see. But the procedure described in the thread you gave the link to, uses multiple test sets too: the only difference from k-fold is that the training data should be only preceding the test chunk. If you adopt this approach then you can obtain several error curves too. My main point is that one of the reasons you might be getting local minima in your error curve is due to noise, so it would make sense to try to reduce it by additional estimates. $\endgroup$
    – amoeba
    Dec 15, 2015 at 17:17

1 Answer 1

  1. Is it common for a cross-validation plot to have several minima? Shouldn't it be smoother with a single optimum?

Yes, this is very common. Adding an additional term improves accuracy proportional to how much information it adds, and decreases accuracy because the model has become more complex and harder to fit. Plus, there is noise due to the fact that your training set is finite. So you shouldn't expect this to be super smooth. I often smooth CV plots before trying to interpret them or find minima.

  1. Why use all 47 terms when it seems that the inclusion of some of them increased the cross-validation error?

It's not clear from the plot if any of the terms actually increased the CV error in a significant way, since there is some level of statistical noise. But you're right, maybe we shouldn't use all 47 of the terms - maybe it would be better to skip some of the functions in the middle. Selecting the exact set of features that gives the best performance is NP-hard, but you could try a greedy approach. First try each function by itself, and pick the one that gives the lowest error. Then try adding in every remaining function one at a time, and pick the one that decreases the error the most. Repeat, and stop once none of the remaining functions decreases the error. This isn't guaranteed to work (for example, maybe some functions are only useful as a group and not individually) but could give you a better set than just taking the functions in order.

  1. Why not use the first or the second local minimum as the best number of terms?

These have worse CV error than 47, so they are unlikely to be better models. However they might be informative - the functions on the downward slopes of these minima might be the best features to use, while the ones on the upswing are just adding complexity and hurting you. You could try taking the set of all functions of the 70 that decreased CV error when they were added, which could be better than your 47.

  • $\begingroup$ Thank you very much for your reply, Chris. I just have a quick question, can you clarify the approach you follow to smooth CV plots? $\endgroup$ Dec 18, 2015 at 15:29
  • 1
    $\begingroup$ I've used a Savitzky-Golay in MATLAB, but any reasonable filter would work. Often my goal is to make sure the minimum I select is robust to small changes in a hyperparameter (a little different than your situation here) but it would also help for eyeballing things. $\endgroup$
    – Chris
    Dec 21, 2015 at 1:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.