As I am trying to automate the process of evaluating models for my prediction problem, I would like to verify the following concerning the process of creating a model consisting of multiple predictor variables for predicting a response variable.

Assume that polynomial regression is promising for solving our prediction problem.

The goal of feature selection is to select which predictor variables are useful and should be included in the model under construction. After having defined which factors are important, then we have to define what is the maximum polynomial degree for each of the "useful" predictor variables.

Does it mean that we have two ways for tuning the complexity of the model: (a) the number of features, and (b) the polynomial degree of each selected feature?

Do you think that the above process is sequential? Is it possible to separate the first step from the second?


Automating model selection is going, inevitably I think, to lead to problems.

In this case, no, the two steps are not separable. Suppose a variable is only important when it is taken as a quadratic. Or suppose adding the quadratic term changes the linear term substantially. Or suppose that adding a linear term for variable B makes variable A more or less important.

If you really do want to do something semi-automated, I think you could start with a model that includes splines of all the variables on a training set, then use those splines to guess at a polynomial term, then test that on a testing set and then (probably) validate the final model on a validation set. However, if your goal is pure prediction, splines may be better than polynomials - one of the problems with splines (in my view) is that they are somewhat harder to explain than polynomials - they don't yield the sort of output that a lot of people are used to.

  • 1
    $\begingroup$ Splines can be written as an equation that is moderately easy to look at. A similar approach to what Peter wrote is to fit a model in which every continuous variable is linear, then one in which every continuous variable has $k$ knots in the spline function, with $k$ varying over a sensible range (for restricted cubic splines, $k=3,4,5,6$ usually does the job). Then pick the model using AIC. $\endgroup$ – Frank Harrell Jul 24 '13 at 11:40
  • $\begingroup$ They can? Is there a good example in your book? (I read it, but it was a while ago). $\endgroup$ – Peter Flom - Reinstate Monica Jul 24 '13 at 11:49
  • 1
    $\begingroup$ Yes there are several examples. In the R rms package running latex(fit) or Function(fit) will create equations in simplest form. $\endgroup$ – Frank Harrell Jul 24 '13 at 11:50
  • $\begingroup$ Thank you both for your replies. Peter, could you please point me out a good (practical) reference for looking at the things you proposed? $\endgroup$ – Dion_E Jul 24 '13 at 12:10
  • $\begingroup$ Look at @FrankHarrell 's book Regression Modeling Strategies. $\endgroup$ – Peter Flom - Reinstate Monica Jul 24 '13 at 12:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.