# Occam's razor (when is it appropriate to add another free parameter?)

So if I fit data to a function you can almost always decrease $\chi_{\nu}^2$ by adding more free parameters. However, this becomes ridiculous if you are fitting a 100-order polynomial to a straight line data set. There is obviously some point where you stop adding parameters!

Is there any quantitative means to find this point other than intuition? I've heard about the F-test but not sure on how to apply it!

Thanks

The biggest problem with adding more free parameters is that you can "overfit" your dataset. Essentially -- you begin to fit the noise in your data rather than the reliable trends that are actually there.

A common way to combat this is to use cross-validation. Briefly -- you hold out some portion of your data (a validation set) and train the model on the remaining portion (the training set). If your model fits the training set well, but does not generalize to the validation set, then you have too many free parameters and your model is overfit.

http://en.wikipedia.org/wiki/Cross-validation_(statistics)

https://www.coursera.org/course/ml

Other related concepts and resources:

http://en.wikipedia.org/wiki/Akaike_information_criterion

http://en.wikipedia.org/wiki/Regularization_(mathematics)

http://ai.stanford.edu/~ang/papers/cv-final.pdf

If you're only concerned about linear regression, the nested F test is a great too. This is the process:

1. Come up with a linear model and estimate it. This is the "restricted" model."
2. Decide which other predictors you'd like to add and estimate the model with them added. This is the "full" model.
3. Compute the RSS for each model. (RSS = sum of squares of residuals)