# how to compare linear and nonlinear regression models in goodness of fit?

Is nonlinear regression (always?) better than linear regression? How can I decide which model to use? I have following alternative models in mind:

$G$ is DV, $x$ and $y$ are IVs

1. $G_i = (b_1(x_i^{b2} - y_i^{b2}) + y_i^{b2})^{1/b2}$ (2 parameters)

2. $G_i = (0.5(x_i^{b2} - y_i^{b2}) + y_i^{b2})^{1/b2}$ (1 parameter)

3. $G_i = b_1(x_i - y_i) + y_i$ (1 parameter)

If my goal is to find the best fit, is calculating AIC, BIC, and adjusted r-squared a good way to select the model between linear and nonlinear regressions? And should I still compare the residual plots? Is there any better way to select the best model given these 3 models?

I should mention that my data size is small.

Thanks!

• I assume that $G_i$ is your target variable, and $x_i, y_i$ are your right hand side variables? This is nonstandard notation, hence the check. Aug 2, 2017 at 19:39
• @jbowman yes, that's right. Thanks for pointing it out, I edited the description. Aug 2, 2017 at 19:52
• Am I misreading, or are 2 and 3 just special cases of 1? Aug 2, 2017 at 20:52
• @DanHicks Yes, 2 and 3 are special cases of 1, by assuming b1 or b2 is constant. Do you think I can still compare them in these way? Thanks Aug 2, 2017 at 20:56

Obviously, nonlinear regression will not always be better than linear regression, because sometimes relationships are linear.

Models with more parameters will produce higher R2 values unless the additional predictors are perfectly correlated with previous ones. Taken to the extreme, adding parameters will lead to meaningless models that fit your data perfectly but perform terribly at out-of-sample prediction and in cross-validation. AIC, BIC, and adjusted R2 are metrics used to penalise the additional model parameters to achieve a balance between explanatory/predictive power and model complexity. The specific penalties differ, and the most appropriate one is debated; the need for some penalty is universally agreed upon. Since you have a small dataset, these metrics will tend to favour simpler models. With more data, it is possible that more complex models will be favoured.

Examining residual plots is useful to see whether any particular model fit is appropriate. For example, patterns in the residuals can sometimes suggest that a different model is necessary. They can sometimes justify choosing a more complex model even when the metrics favour a simpler one.

• Thanks for the answer! In terms of complexity of models, AIC prefers more complex model, right? My AIC, BIC, and adjusted r-squared are really close in alternative models (i.e., adjusted r-squared 0.989 vs 0.987 vs.0.983; AIC 1418.3 vs. 1416.4 vs. 1418.2), how should I interpret this similarity? Are they indicating that it is meaningless to have these alternative models because they do not significantly differ? Is there any way to test the significance of the difference? Thanks! Aug 2, 2017 at 19:50
• BIC tends to penalise parameters more than AIC, yes. I don't think using adjusted R2 for model comparison is advisable, but it's not something I've looked into. There are commonly used thresholds that correspond to testing for significant differences; for delta AIC, I've seen values of 2, 7 and 10 used in various places.
– mkt
Aug 2, 2017 at 19:56
• If the AIC/BIC/whatever values are similar for all models, it indicates that they are similar in quality. There's not too much you can do beyond collect more data at that point.
– mkt
Aug 2, 2017 at 19:58

What I would do here is compute scatterplots of G against the predicted G for the 3 different models. I might also compute Tukey mean difference plots. And I'd look at plots of the residuals from the three models.

Then I'd make my decision based on what those plots told me. Models can go wrong in various ways; not all are equally important in every situation. E.g. perhaps, for you, an underestimate of G is much worse than an overestimate (or vice versa). Perhaps you can put up with a lot of small errors if you don't get any big ones. Or maybe something else.

• Thanks! By scatterplots of G against predicted G, do you mean that I need to do a cross validation? Or residual plots are good? Thanks again! Aug 3, 2017 at 0:01
• No, not cross validation. Each model will predict G. You also have actual G. You can do scatterplots of the predicted values vs. actual. Aug 3, 2017 at 0:05