When trying to predict data using linear regression or classify with logistic regression, with a polynomial, I know how to find the best degree of a polynomial to fits given data when the regularization coefficient is fixed. I also know how to find the best regularization coefficient when the degree of the polynomial is fixed.

What I want to know is how to find the best model when none of these parameters are known.

  • Should I find the best degree without regularization first, then the regularization parameter ?
  • Should I, for every degree, train with every possible regularization parameter value (assuming it belongs to an ensemble of discrete values), and then pick the combination degree/regularization that had the best results on the validation set ?

Or is there a better solution to find these hyperparameters ?

  • 1
    $\begingroup$ Is the context linear models with polynomial basis functions? $\endgroup$ – Matthew Drury Jan 29 '16 at 23:01
  • $\begingroup$ Yes, i updated the question to mention that $\endgroup$ – Carl Levasseur Jan 29 '16 at 23:49

Since u are using regularization for feature selection I guess I would find the best regularization parameter for each order and then select the model with the smallest testing error. However, I think if u use a high order polynomial first and choose the best regularization parameter for that specific order you can get some insight.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.