0
$\begingroup$

Linear regression has model hyperparameters such as number of predictors. For example in a autoregressive time series model AR(p), p is the number of predictors. To find which value of p to find we have 2 approaches:

  • adjusted metrics such as Bayesian Information Criterion (BIC) and others. These metrics depends on an error (such as RSS) but only on the number of parameters, i.e. the metric is penalized for high number of parameters to avoid overfitting.
  • validation/cross-validation on a train/test split data.

Which one is preferable and when?

For example I am reading a book where they use BIC to find p in AR(p) and not crossvalidation, the application is forex prediction. I was wondering why.

$\endgroup$
0
$\begingroup$

In general, having a testing set (with cross-validation) that matches the exact condition in which your regression will be applied in the future is the best way to go.

Now BIC or AIC are useful as they try to select model more general by avoiding using parameters with small influences but they are less robust. You will find that they are quite arbitrary. It is hard to give proper value of equivalence between having n more parameters but increasing by x% the RSS. They can be used when the data you have is too small to do a proper test set and cross-validation. You can also use them, if a trade-off between performance and number of parameters is well defined in your problem, for instance you could have computation condition or memory condition and you are willing to trade performance for that.

$\endgroup$
2
  • $\begingroup$ Is there a book where BIC, AIC, etc... are well explained? $\endgroup$ – randomwalker Mar 27 at 15:10
  • $\begingroup$ I don't know any book but there a lot of articles. This one is particularly useful in your case ieeexplore.ieee.org/document/8498082. if you browse wikipedia pages for AIC and BIC you might also find more answer to your questions $\endgroup$ – Romain Mar 28 at 9:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.