# How to choose the right number of parameters in Logistic Regression?

I am studying Andrew Ng's Machine Learning lecture notes. I understand either we can manually choose the number of parameters, or we can use regularization to make it correctly fit.

I was wondering are there any basic rules for choosing the right number of parameters? Can anyone give any explanation please?

• Actually I want to know how to choose the correct no of theta's. Commented May 9, 2014 at 11:15
• Note that regularisation isn't a "fix all", especially the typically use L1 and L2 penalties. These suffer in "sparse" scenarios, where say 5-10 variables out of 100 are non-zero. Commented May 9, 2014 at 12:45
• One option to choose between few models: minimise the cross-validated error in your (training) data set. Commented Mar 27, 2015 at 14:18

Distortion of statistical properties can occur when you "fit to the data", so I think of this more in terms of specifying the number of parameters that I can afford to estimate and that I want to devote to the portion of the model that pertains to that one predictor. I use regression splines, place knots where $X$ is dense, and specify the number of knots (or the number of parameters and back calculate the number of knots) by asking (1) what does the sample size and distribution of $Y$ support and (2) what is the signal:noise ratio in this dataset. When $n \uparrow$ or signal:noise ratio $\uparrow$ I can use more knots. There is no set formula for the number of parameters that should be fitted, although in a minority of situations you can use cross-validation or AIC to determine this. As you mentioned, shrinkage is a great alternative, because you can start out with many parameters then shrink the coefficients down to what cross-validation or effective AIC dictate.