Occam's razor (when is it appropriate to add another free parameter?)

Question

So if I fit data to a function you can almost always decrease $\chi_{\nu}^2$ by adding more free parameters. However, this becomes ridiculous if you are fitting a 100-order polynomial to a straight line data set. There is obviously some point where you stop adding parameters!

Is there any quantitative means to find this point other than intuition? I've heard about the F-test but not sure on how to apply it!

Thanks

Answer 1

The biggest problem with adding more free parameters is that you can "overfit" your dataset. Essentially -- you begin to fit the noise in your data rather than the reliable trends that are actually there.

A common way to combat this is to use cross-validation. Briefly -- you hold out some portion of your data (a validation set) and train the model on the remaining portion (the training set). If your model fits the training set well, but does not generalize to the validation set, then you have too many free parameters and your model is overfit.

http://en.wikipedia.org/wiki/Cross-validation_(statistics)

https://www.coursera.org/course/ml

Other related concepts and resources:

http://en.wikipedia.org/wiki/Akaike_information_criterion

http://en.wikipedia.org/wiki/Regularization_(mathematics)

http://www.youtube.com/watch?v=I-VfYXzC5ro

http://ai.stanford.edu/~ang/papers/cv-final.pdf

Answer 2

If you're only concerned about linear regression, the nested F test is a great too. This is the process:

1. Come up with a linear model and estimate it. This is the "restricted" model."
2. Decide which other predictors you'd like to add and estimate the model with them added. This is the "full" model.
3. Compute the RSS for each model. (RSS = sum of squares of residuals)
4. Find RSS(restricted) - RSS(full) / # of predictors you added in step 2.
5. Find RSS(full) / (N - total # of predictors + 1)
6. Your F statistic is step 5 divided by step 6. Your null hypothesis is that the full model contributes nothing to prediction, against an alternative of contributing something.
7. If F > F(# of added predictors, N - total # predictors + 1), reject your null. Thereby you are justified in adding those parameters.

The issue with this test is that you need it only operates pairwise. That is, it won't tell you which parameters to include, which parameters to include separately or as groups, or generally which comparisons to make. So you have to have a somewhat reasonable set of additional parameters in mind before you jump into nesting models. There's also the Ramsey test for misspecification, which can tell you whether your linear model is appropriate.