6
$\begingroup$

I am studying Andrew Ng's Machine Learning lecture notes. I understand either we can manually choose the number of parameters, or we can use regularization to make it correctly fit.

enter image description here

I was wondering are there any basic rules for choosing the right number of parameters? Can anyone give any explanation please?

Improve this question
$\endgroup$
3
  • $\begingroup$ Actually I want to know how to choose the correct no of theta's. $\endgroup$
    – Tropa
    Commented May 9, 2014 at 11:15
  • 1
    $\begingroup$ Note that regularisation isn't a "fix all", especially the typically use L1 and L2 penalties. These suffer in "sparse" scenarios, where say 5-10 variables out of 100 are non-zero. $\endgroup$
    – probabilityislogic
    Commented May 9, 2014 at 12:45
  • $\begingroup$ One option to choose between few models: minimise the cross-validated error in your (training) data set. $\endgroup$
    – snoram
    Commented Mar 27, 2015 at 14:18

1 Answer 1

Reset to default
5
$\begingroup$

Distortion of statistical properties can occur when you "fit to the data", so I think of this more in terms of specifying the number of parameters that I can afford to estimate and that I want to devote to the portion of the model that pertains to that one predictor. I use regression splines, place knots where $X$ is dense, and specify the number of knots (or the number of parameters and back calculate the number of knots) by asking (1) what does the sample size and distribution of $Y$ support and (2) what is the signal:noise ratio in this dataset. When $n \uparrow$ or signal:noise ratio $\uparrow$ I can use more knots. There is no set formula for the number of parameters that should be fitted, although in a minority of situations you can use cross-validation or AIC to determine this. As you mentioned, shrinkage is a great alternative, because you can start out with many parameters then shrink the coefficients down to what cross-validation or effective AIC dictate.

Improve this answer
$\endgroup$
2
  • 6
    $\begingroup$ I'd add that what you do should be science too. In the original example, there could easily be no grounds for thinking that price and size must be linearly related, some grounds for thinking that they should be monotonically related and no grounds for expecting an arbitrarily complicated curve. Regardless of who you are, you should know more about the science underlying the data than any software does. If you are the statistician and you have non-statistical collaborators, it can be a case of what makes sense to them. In short, it's naive to expect formal rules to be the complete story. $\endgroup$
    – Nick Cox
    Commented May 9, 2014 at 11:52
  • 1
    $\begingroup$ Exceptionally perceptive. I should have mentioned that subject matter knowledge needs to drive the process. $\endgroup$
    – Frank Harrell
    Commented May 9, 2014 at 14:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.