8
$\begingroup$

How can one objectively (read "algorithmically") select an appropriate model for doing a simple linear least-squares regression with two variables?

For example, say the data seem to show a quadratic trend, and a parabola is generated which fits the data quite well. How do we justify making this the regression? Or how do we eliminate the possibility of there existing a better model?

What I'm really worried about is this: we could just keep adding polynomial terms until we had a perfect fit for the data set (an interpolation of the points), with no error whatsoever. But this would be useless as far as predicting or extrapolating, because there would be no reason to think that the "model" was actually appropriate. So how does one balance the needs of accuracy and intuitive appeal?

(Also, please alert me if this has been asked before, I assumed it would have been but did not find anything.)

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ The question itself really begs the question that you need to ask yourself which is why am I building this regression model? What will you use the model for? What do you hope to learn from the model? These are huge questions that will surely guide the steps you take in building the model. $\endgroup$
    – jsk
    May 19, 2014 at 1:55
  • $\begingroup$ I think the figure in the Wikipedia page for overfitting speaks by itself. $\endgroup$
    – nico
    Jul 1, 2014 at 9:30

4 Answers 4

Reset to default
1
$\begingroup$

You can look at AIC, BIC or any of the other similar measures.

You could use your eyes and sense of the field.

Or you could avoid some of the problem by using splines.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks. I'm just a beginner in statistics, may I ask if the AIC,BIC measures are "objective" (as in derived from something), or are they basically "made-up" from statistical experience? $\endgroup$
    – user45240
    May 18, 2014 at 20:56
  • $\begingroup$ AIC, BIC, etc are derived from two important qualities of the model under consideration: the number of parameters (k) and the likelihood of the model (L). Consider the $AIC \equiv 2k-2\ln(L)$ - here we see a tradeoff between the number of parameters and the likelihood. $\endgroup$
    – abaumann
    May 18, 2014 at 21:42
9
$\begingroup$

It's likely to be the case that you are not going to be able to find a polynomial that constitutes a correct description of the relationship no matter how much data you have.

This problem may extend to almost any class of models.

However, usually we're interested in getting a good description that suffices for some purpose (a model), rather than discovering the (possibly over-complicated) actual process driving the observations.

Indeed, even where the true process is from some hypothesized class of potential models, it may be counterproductive to discover the true model (which might be of high order, for example, but the high order terms might be very very small). It may be that a simpler (i.e. wrong) model is much better for our purposes.

For example, imagine we were trying to predict the next few values in a somewhat noisy series. Any model we fit has some error in the parameter estimates, and that error will be magnified by the forecasting. It doesn't take much to have a low-order model (which is necessarily biased) with much better mean square prediction error (say) performance than the 'true' model order.

One common tool for evaluating model performance is at out-of-sample prediction (not necessarily over time). Cross-validation is one common way to choose models or compare model performance.

Rob Hyndman wrote a nice little introduction here.

Improve this answer
$\endgroup$
0
1
$\begingroup$

I would say very often people align themselves with one of three different approaches:

  • frequentists, which make use of tests such as the F-test
  • bayesians, which make use of bayesian inference
  • information theory guys, which use the BIC and AIC, just like other examples cited above.

Frequentist analysis is probably both the most straightforward and the most criticized for its shortcomings. Information theory on the other hand, underwent a boom recently, drawing the attention of more and more people as time goes by. I think you should try to understand a bit and draw some ideas from each of the three approaches. If you have no idea about what the data should contain, then the frequentist approach is a good way to start; on the other hand If you have some information on the underlying model, take a look at bayesian inference. And I would always keep the number of free parameters low, and that's what AIC and BIC try to balancing information with parameters.

Improve this answer
$\endgroup$
0
$\begingroup$

I would use restricted cubic splines which allow you to better approximate the curve. As an added refinement, may use AICc (or BIC) to chose the number of knots.

Improve this answer
$\endgroup$
1
  • $\begingroup$ It really depends on the specific problem and data set, sometimes a straight line is the best difficult to say anything without seeing the data. $\endgroup$
    – nico
    Jul 1, 2014 at 9:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.