10
$\begingroup$

How should one decide between using a linear regression model or non-linear regression model?

My goal is to predict Y.

In case of simple $x$ and $y$ dataset I could easily decide which regression model should be used by plotting a scatter plot.

In case of multi-variant like $x_1,x_2,...x_n$ and $y$. How can I decide which regression model has to be used? That is, How will I decide about going with simple linear model or non linear models such as quadric, cubic etc.

Is there any technique or statistical approach or graphical plots to infer and decide which regression model has to be used?

$\endgroup$
  • $\begingroup$ "Non-linear model" is a pretty broad category. Did you have one in mind? What are your analysis goals? $\endgroup$ – shadowtalker Feb 6 '15 at 14:33
  • $\begingroup$ This depends on your goals. Are you building a prediction/forecasting model? $\endgroup$ – Aksakal Feb 6 '15 at 15:58
  • $\begingroup$ Prediction is my goal. $\endgroup$ – shakthydoss Feb 6 '15 at 16:02
  • 1
    $\begingroup$ If you're after something like the "plot the data" approach but for multiple predictors, there are added variable plots which can be of some value. But if your goal is prediction, the problem is you're choosing what to git based on seeing the data, so it will look much better on the data you have than on other data (and there are multiple other issues that come with such an approach to model selection) -- to properly evaluate out of sample predictive ability you need to assess things on a holdout sample/consider something like cross validation. $\endgroup$ – Glen_b Feb 6 '15 at 17:16
  • 1
    $\begingroup$ You might find useful a related discussion that I've started some time ago. $\endgroup$ – Aleksandr Blekh Feb 11 '15 at 7:09
9
$\begingroup$

This is a realm of statistics called model selection. A lot of research is done in this area and there's no definitive and easy answer.

Let's assume you have $X_1, X_2$, and $X_3$ and you want to know if you should include an $X_3^2$ term in the model. In a situation like this your more parsimonious model is nested in your more complex model. In other words, the variables $X_1, X_2$, and $X_3$ (parsimonious model) are a subset of the variables $X_1, X_2, X_3$, and $X_3^2$ (complex model). In model building you have (at least) one of the following two main goals:

  1. Explain the data: you are trying to understand how some set of variables affect your response variable or you are interested in how $X_1$ effects $Y$ while controlling for the effects of $X_2,...X_p$
  2. Predict $Y$: you want to accurately predict $Y$, without caring about what or how many variables are in your model

If your goal is number 1, then I recommend the Likelihood Ratio Test (LRT). LRT is used when you have nested models and you want to know "are the data significantly more likely to come from the complex model than the parsimonous model?". This will give you insight into which model better explains the relationship between your data.

If your goal is number 2, then I recommend some sort of cross-validation (CV) technique ($k$-fold CV, leave-one-out CV, test-training CV) depending on the size of your data. In summary, these methods build a model on a subset of your data and predict the results on the remaining data. Pick the model that does the best job predicting on the remaining data.

$\endgroup$
  • $\begingroup$ Please, Could you make/explain the difference between goals (1) and (2) more pronounced? Currently there is not much difference. $\endgroup$ – ttnphns Feb 11 '15 at 6:57
  • $\begingroup$ @ttnphns I added a brief description of the two goals. $\endgroup$ – TrynnaDoStat Feb 17 '15 at 16:51
  • $\begingroup$ @TrynnaDoStat Just confused here by the statement Pick the model that does the best job predicting. By best model you mean to choose between the linear(parsimonious) model and complex model....right? Because what I know is k-fold, leave-one-out CV are used to check the model performance on unseen data. They are not used for model selection. I am confused here. $\endgroup$ – tushaR Sep 1 '17 at 4:52
1
+25
$\begingroup$

When I google for "linearn or non-linear model for regression" I get some links which lead to this book: http://www.graphpad.com/manuals/prism4/RegressionBook.pdf This book is not interesting, and I don't trust it in 100% (for some reasons).

I found also this article: http://hunch.net/?p=524 with title: Nearly all natural problems require nonlinearity

I also found similar question with pretty good explanation: https://stackoverflow.com/questions/1148513/difference-between-a-linear-problem-and-a-non-linear-problem-essence-of-dot-pro

Based on my experience, when you don't know which model use, use both and try another features.

$\endgroup$
0
$\begingroup$

As you state, linear models are typically simpler than non-linear models, meaning they run faster (building and predicting), are easier to interpret and explain, and usually straight-forward in error measurements. So the goal is to find out if the assumptions of a linear regression hold with your data (if you fail to support linear, then just go with non-linear). Usually you would repeat your single-variable plot with all variables individually, holding all other variables constant.

Perhaps more importantly, though, you want to know if you can apply some sort of transformation, variable interaction, or dummy variable to move your data to linear space. If you are able to validate the assumptions, or if you know your data well enough to apply well-motivated or otherwise intelligently informed transformations or modifications, then you want to proceed with that transform and use linear regression. Once you have the residuals, you can plot them versus predicted values or independent variables to further decide if you need to move on to non-linear methods.

There is an excellent breakdown of the assumptions of linear regression here at Duke. The four main assumptions are listed, and each one is broken down into the effects on the model, how to diagnose it in the data, and potential ways to "fix" (i.e. transform or add to) the data to make the assumption hold. Here is a small excerpt from the top summarizing the four assumptions addressed, but you should go there and read the breakdowns.

There are four principal assumptions which justify the use of linear regression models for purposes of inference or prediction:

(i) linearity and additivity of the relationship between dependent and independent variables:

(a) The expected value of dependent variable is a straight-line function of each independent variable, holding the others fixed.

(b) The slope of that line does not depend on the values of the other variables.

(c) The effects of different independent variables on the expected value of the dependent variable are additive.

(ii) statistical independence of the errors (in particular, no correlation between >consecutive errors in the case of time series data)

(iii) homoscedasticity (constant variance) of the errors

(a) versus time (in the case of time series data)

(b) versus the predictions

(c) versus any independent variable

(iv) normality of the error distribution.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.