Choosing a regression model How can one objectively (read "algorithmically") select an appropriate model for doing a simple linear least-squares regression with two variables?
For example, say the data seem to show a quadratic trend, and a parabola is generated which fits the data quite well. How do we justify making this the regression? Or how do we eliminate the possibility of there existing a better model?
What I'm really worried about is this: we could just keep adding polynomial terms until we had a perfect fit for the data set (an interpolation of the points), with no error whatsoever. But this would be useless as far as predicting or extrapolating, because there would be no reason to think that the "model" was actually appropriate. So how does one balance the needs of accuracy and intuitive appeal?
(Also, please alert me if this has been asked before, I assumed it would have been but did not find anything.)
 A: It's likely to be the case that you are not going to be able to find a polynomial that constitutes a correct description of the relationship no matter how much data you have.
This problem may extend to almost any class of models.
However, usually we're interested in getting a good description that suffices for some purpose (a model), rather than discovering the (possibly over-complicated) actual process driving the observations.
Indeed, even where the true process is from some hypothesized class of potential models, it may be counterproductive to discover the true model (which might be of high order, for example, but the high order terms might be very very small). It may be that a simpler (i.e. wrong) model is much better for our purposes.
For example, imagine we were trying to predict the next few values in a somewhat noisy series. Any model we fit has some error in the parameter estimates, and that error will be magnified by the forecasting. It doesn't take much to have a low-order model (which is necessarily biased) with much better mean square prediction error (say) performance than the 'true' model order.
One common tool for evaluating model performance is at out-of-sample prediction (not necessarily over time). Cross-validation is one common way to choose models or compare model performance.
Rob Hyndman wrote a nice little introduction here.
A: You can look at AIC, BIC or any of the other similar measures. 
You could use your eyes and sense of the field.
Or you could avoid some of the problem by using splines. 
A: I would say very often people align themselves with one of three different approaches:


*

*frequentists, which make use of tests such as the F-test

*bayesians, which make use of bayesian inference

*information theory guys, which use the BIC and AIC, just like other examples cited above.


Frequentist analysis is probably both the most straightforward and the most criticized for its shortcomings. Information theory on the other hand, underwent a boom recently, drawing the attention of more and more people as time goes by. 
I think you should try to understand a bit and draw some ideas from each of the three approaches. 
If you have no idea about what the data should contain, then the frequentist approach is a good way to start; on the other hand If you have some information on the underlying model, take a look at bayesian inference. And I would always keep the number of free parameters low, and that's what AIC and BIC try to balancing information with parameters.
A: I would use restricted cubic splines which allow you to better approximate the curve. As an added refinement, may use AICc (or BIC) to chose the number of knots.
