Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

How can one objectively (read "algorithmically") select an appropriate model for doing a simple linear least-squares regression with two variables?

For example, say the data seem to show a quadratic trend, and a parabola is generated which fits the data quite well. How do we justify making this the regression? Or how do we eliminate the possibility of there existing a better model?

What I'm really worried about is this: we could just keep adding polynomial terms until we had a perfect fit for the data set (an interpolation of the points), with no error whatsoever. But this would be useless as far as predicting or extrapolating, because there would be no reason to think that the "model" was actually appropriate. So how does one balance the needs of accuracy and intuitive appeal?

(Also, please alert me if this has been asked before, I assumed it would have been but did not find anything.)

share|improve this question
1  
The question itself really begs the question that you need to ask yourself which is why am I building this regression model? What will you use the model for? What do you hope to learn from the model? These are huge questions that will surely guide the steps you take in building the model. –  jsk May 19 at 1:55
    
I think the figure in the Wikipedia page for overfitting speaks by itself. –  nico Jul 1 at 9:30
add comment

4 Answers 4

up vote 2 down vote accepted

You can look at AIC, BIC or any of the other similar measures.

You could use your eyes and sense of the field.

Or you could avoid some of the problem by using splines.

share|improve this answer
    
Thanks. I'm just a beginner in statistics, may I ask if the AIC,BIC measures are "objective" (as in derived from something), or are they basically "made-up" from statistical experience? –  NotNotLogical May 18 at 20:56
    
AIC, BIC, etc are derived from two important qualities of the model under consideration: the number of parameters (k) and the likelihood of the model (L). Consider the $AIC \equiv 2k-2\ln(L)$ - here we see a tradeoff between the number of parameters and the likelihood. –  abaumann May 18 at 21:42
add comment

It's likely to be the case that you are not going to be able to find a polynomial that constitutes a correct description of the relationship no matter how much data you have.

This problem may extend to almost any class of models.

However, usually we're interested in getting a good description that suffices for some purpose (a model), rather than discovering the (possibly over-complicated) actual process driving the observations.

Indeed, even where the true process is from some hypothesized class of potential models, it may be counterproductive to discover the true model (which might be of high order, for example, but the high order terms might be very very small). It may be that a simpler (i.e. wrong) model is much better for our purposes.

For example, imagine we were trying to predict the next few values in a somewhat noisy series. Any model we fit has some error in the parameter estimates, and that error will be magnified by the forecasting. It doesn't take much to have a low-order model (which is necessarily biased) with much better mean square prediction error (say) performance than the 'true' model order.

One common tool for evaluating model performance is at out-of-sample prediction (not necessarily over time). Cross-validation is one common way to choose models or compare model performance.

Rob Hyndman wrote a nice little introduction here.

share|improve this answer
add comment

I would say very often people align themselves with one of three different approaches:

  • frequentists, which make use of tests such as the F-test
  • bayesians, which make use of bayesian inference
  • information theory guys, which use the BIC and AIC, just like other examples cited above.

Frequentist analysis is probably both the most straightforward and the most criticized for its shortcomings. Information theory on the other hand, underwent a boom recently, drawing the attention of more and more people as time goes by. I think you should try to understand a bit and draw some ideas from each of the three approaches. If you have no idea about what the data should contain, then the frequentist approach is a good way to start; on the other hand If you have some information on the underlying model, take a look at bayesian inference. And I would always keep the number of free parameters low, and that's what AIC and BIC try to balancing information with parameters.

share|improve this answer
add comment

I would use restricted cubic splines which allow you to better approximate the curve. As an added refinement, may use AICc (or BIC) to chose the number of knots.

share|improve this answer
    
It really depends on the specific problem and data set, sometimes a straight line is the best difficult to say anything without seeing the data. –  nico Jul 1 at 9:28
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.